cleanup projects list

Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
This commit is contained in:
Nico Schottelius 2009-07-30 16:29:20 +02:00
parent 326c553b09
commit 5d75edf0e2
4 changed files with 419 additions and 316 deletions

View File

@ -5,18 +5,20 @@ The following is an incomplete list of projects I'm working on:
* [carbon / corpus](http://git.schottelius.org/?p=cLinux/carbon.git): Package management
* [cconf / chamelon](http://unix.schottelius.org/cconf/): autoconf replacements
* [ccollaborate](http://unix.schottelius.org/ccollaborate/): web application framework
* [cinit](http://unix.schottelius.org/cinit/): a fast, small and simple init with support for profiles.
* [cinit](http://unix.schottelius.org/cinit/):
a fast, small and simple init with support for profiles.
* cLinux: communities Linux distribution
* cnp: create new project (assistant for new FOSS projects)
* [[ccollect|software/ccollect]]: (pseudo) incremental backup with different exclude lists using hardlinks and rsync
* [corein](http://git.schottelius.org/?p=cLinux/corein): Complete remote, automatic installation and configuration
* [[ccollect|software/ccollect]]:
(pseudo) incremental backup with different exclude lists using hardlinks and rsync
* [corein](http://git.schottelius.org/?p=cLinux/corein):
Complete remote, automatic installation and configuration
* cvoktrainer: vocabulary trainer
* cvpn: manage vpns
* crassh: Encrypted, easy-to-use user authentification for Unix systems
* decr-f: packet description format
* [dmp3find](http://unix.schottelius.org/dmp3find/): find doubled mp3s
* easyC / shcl
* ecml / saml
* ecml: Encrypted mailinglist
* EOF / ceof / ceofhack
* fsdb: filesystem database
* [gpm](http://unix.schottelius.org/gpm): general purpose mouse
@ -27,7 +29,9 @@ The following is an incomplete list of projects I'm working on:
* nsfortunes
* rlmanager
* [schwanz3](http://home.schottelius.org/~nico/schwanz3/)
* smtp_logger
* shcl: Small helper c library
* [smtp_logger](http://git.schottelius.org/?p=smtp_logger):
Logs smtp sessions
* ssa2: server side admin
* tcpserver_blocker
* unix_installer
@ -37,4 +41,7 @@ The following is an incomplete list of projects I'm working on:
A lot of those projects are just ideas, most of them have some code
or documentation and some are even implemented and working.
Most of the projects, which have code or documentation, can be found on
[git.schottelius.org](http://git.schottelius.org/).
[[!tag net schottelius unix]]

File diff suppressed because one or more lines are too long

View File

@ -3,8 +3,7 @@
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta name="generator" content="AsciiDoc 8.4.4" />
<title>ccollect - Installing, Configuring and Using</title>
<meta name="generator" content="AsciiDoc 8.2.7" />
<style type="text/css">
/* Debug borders */
p, li, dt, dd, div, pre, h1, h2, h3, h4, h5, h6 {
@ -111,13 +110,11 @@ div#footer-badges {
padding-bottom: 0.5em;
}
div#preamble {
margin-top: 1.5em;
margin-bottom: 1.5em;
}
div#preamble,
div.tableblock, div.imageblock, div.exampleblock, div.verseblock,
div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
div.admonitionblock {
margin-right: 10%;
margin-top: 1.5em;
margin-bottom: 1.5em;
}
@ -159,6 +156,9 @@ div.sidebarblock > div.content {
padding: 0.5em;
}
div.listingblock {
margin-right: 0%;
}
div.listingblock > div.content {
border: 1px solid silver;
background: #f4f4f4;
@ -167,7 +167,6 @@ div.listingblock > div.content {
div.quoteblock {
padding-left: 2.0em;
margin-right: 10%;
}
div.quoteblock > div.attribution {
padding-top: 0.5em;
@ -176,7 +175,6 @@ div.quoteblock > div.attribution {
div.verseblock {
padding-left: 2.0em;
margin-right: 10%;
}
div.verseblock > div.content {
white-space: pre;
@ -220,7 +218,6 @@ dt {
margin-top: 0.5em;
margin-bottom: 0;
font-style: normal;
color: navy;
}
dd > *:first-child {
margin-top: 0.1em;
@ -229,28 +226,12 @@ dd > *:first-child {
ul, ol {
list-style-position: outside;
}
ol.arabic {
div.olist > ol {
list-style-type: decimal;
}
ol.loweralpha {
div.olist2 > ol {
list-style-type: lower-alpha;
}
ol.upperalpha {
list-style-type: upper-alpha;
}
ol.lowerroman {
list-style-type: lower-roman;
}
ol.upperroman {
list-style-type: upper-roman;
}
div.compact ul, div.compact ol,
div.compact p, div.compact p,
div.compact div, div.compact div {
margin-top: 0.1em;
margin-bottom: 0.1em;
}
div.tableblock > table {
border: 3px solid #527bbd;
@ -262,53 +243,22 @@ thead {
tfoot {
font-weight: bold;
}
td > div.verse {
white-space: pre;
}
p.table {
margin-top: 0;
}
/* Because the table frame attribute is overriden by CSS in most browsers. */
div.tableblock > table[frame="void"] {
border-style: none;
}
div.tableblock > table[frame="hsides"] {
border-left-style: none;
border-right-style: none;
}
div.tableblock > table[frame="vsides"] {
border-top-style: none;
border-bottom-style: none;
}
div.hdlist {
div.hlist {
margin-top: 0.8em;
margin-bottom: 0.8em;
}
div.hdlist tr {
div.hlist td {
padding-bottom: 15px;
}
dt.hdlist1.strong, td.hdlist1.strong {
font-weight: bold;
}
td.hdlist1 {
td.hlist1 {
vertical-align: top;
font-style: normal;
padding-right: 0.8em;
color: navy;
}
td.hdlist2 {
td.hlist2 {
vertical-align: top;
}
div.hdlist.compact tr {
margin: 0;
padding-bottom: 0;
}
.comment {
background: yellow;
}
@media print {
div#footer-badges { display: none; }
@ -380,7 +330,16 @@ div.exampleblock-content {
/* IE6 sets dynamically generated links as visited. */
div#toc a:visited { color: blue; }
/* Because IE6 child selector is broken. */
div.olist2 ol {
list-style-type: lower-alpha;
}
div.olist2 div.olist ol {
list-style-type: decimal;
}
</style>
<title>ccollect - Installing, Configuring and Using</title>
</head>
<body>
<div id="header">
@ -392,22 +351,22 @@ for ccollect 0.7.1, Initial Version from 2006-01-13
</div>
<div id="preamble">
<div class="sectionbody">
<div class="paragraph"><p>(pseudo) incremental backup
<div class="para"><p>(pseudo) incremental backup
with different exclude lists
using hardlinks and <tt>rsync</tt></p></div>
</div>
</div>
<h2 id="_introduction">1. Introduction</h2>
<div class="sectionbody">
<div class="paragraph"><p><tt>ccollect</tt> is a backup utility written in the sh-scripting language.
<div class="para"><p><tt>ccollect</tt> is a backup utility written in the sh-scripting language.
It does not depend on a specific shell, only <tt>/bin/sh</tt> needs to be
bourne shell compatible (like <em>dash</em>, <em>ksh</em>, <em>zsh</em>, <em>bash</em>, &#8230;).</p></div>
<h3 id="_supported_and_tested_operating_systems_and_architectures">1.1. Supported and tested operating systems and architectures</h3><div style="clear:left"></div>
<div class="paragraph"><p><tt>ccollect</tt> was successfully tested on the following platforms:</p></div>
<div class="ulist"><ul>
<div class="para"><p><tt>ccollect</tt> was successfully tested on the following platforms:</p></div>
<div class="ilist"><ul>
<li>
<p>
GNU/Linux on amd64/hppa/i386/ppc
GNU/Linux on amd64/hppa/i386/ppc/ARM
</p>
</li>
<li>
@ -431,35 +390,58 @@ OpenBSD on amd64
</p>
</li>
</ul></div>
<div class="paragraph"><p>It <strong>should</strong> run on any Unix that supports <tt>rsync</tt> and has a POSIX-compatible
<div class="para"><p>It <strong>should</strong> run on any Unix that supports <tt>rsync</tt> and has a POSIX-compatible
bourne shell. If your platform is not listed above and you have it successfully
running, please drop me a mail.</p></div>
<h3 id="_why_you_could_only_backup_from_remote_hosts_not_to_them">1.2. Why you COULD only backup from remote hosts, not to them</h3><div style="clear:left"></div>
<div class="paragraph"><p>While considering the design of ccollect, I thought about enabling
<div class="para"><p>While considering the design of ccollect, I thought about enabling
backup to <strong>remote</strong> hosts. Though this sounds like a nice feature
(<em>Backup my notebook to the server now.</em>), in my opinion it is a
bad idea to backup to a remote host.</p></div>
<div class="paragraph"><p>But as more and more people requested this feature, it was implemented,
<div class="para"><p>But as more and more people requested this feature, it was implemented,
so you have the choice whether you want to use it or not.</p></div>
<h4 id="_reason">1.2.1. Reason</h4>
<div class="paragraph"><p>If you want to backup <strong>TO</strong> a remote host, you have to loosen security on it.</p></div>
<div class="paragraph"><p>Imagine the following situation: You backup your farm of webservers <strong>TO</strong>
<div class="para"><p>If you want to backup <strong>TO</strong> a remote host, you have to loosen security on it.</p></div>
<div class="para"><p>Imagine the following situation: You backup your farm of webservers <strong>TO</strong>
a backup host somewhere else.
Now one of your webservers which has access to your backup host gets
compromised.</p></div>
<div class="paragraph"><p>Your backup server will be compromised, too.</p></div>
<div class="paragraph"><p>And the attacker will have access to all data on the other webservers.</p></div>
<div class="para"><p>Your backup server will be compromised, too.</p></div>
<div class="para"><p>And the attacker will have access to all data on the other webservers.</p></div>
<h4 id="_doing_it_securely">1.2.2. Doing it securely</h4>
<div class="paragraph"><p>Think of it the other way round: The backup server (now behind a
<div class="para"><p>Think of it the other way round: The backup server (now behind a
firewall, not accessable from outside) connects to the
webservers and pulls the data <strong>from</strong> them. If someone gets access to one
of the webservers, this person will perhaps not even see your machine. If
the attacker sees connections from a host to the compromised
machine, she will not be able to log in on the backup machine.
All other backups are still secure.</p></div>
<h3 id="_incompatibilities">1.3. Incompatibilities</h3><div style="clear:left"></div>
<h4 id="_versions_0_6_and_0_7">1.3.1. Versions 0.6 and 0.7</h4>
<div class="ulist"><div class="title">The format of <tt>destination</tt> changed:</div><ul>
<h3 id="_incompatibilities_and_changes">1.3. Incompatibilities and changes</h3><div style="clear:left"></div>
<h4 id="_versions_0_7_and_0_8">1.3.1. Versions 0.7 and 0.8</h4>
<div class="olist"><ol>
<li>
<p>
The argument order changed:
</p>
<div class="ilist"><ul>
<li>
<p>
Old: "&lt;interval name&gt; [args] &lt;sources to backup&gt;"
</p>
</li>
<li>
<p>
New: "[args] &lt;interval name&gt; &lt;sources to backup&gt;"
</p>
</li>
</ul></div>
</li>
</ol></div>
<div class="para"><p>If you did not use arguments (most people do not), nothing will
change for you.</p></div>
<div class="para"><p>.</p></div>
<h4 id="_versions_0_6_and_0_7">1.3.2. Versions 0.6 and 0.7</h4>
<div class="ilist"><div class="title">The format of <tt>destination</tt> changed:</div><ul>
<li>
<p>
Before 0.7 it was a (link to a) directory
@ -471,16 +453,16 @@ As of 0.7 it is a textfile containing the destination
</p>
</li>
</ul></div>
<div class="paragraph"><p>You can update your configuration using <tt>tools/config-pre-0.7-to-0.7.sh</tt>.</p></div>
<div class="ulist"><div class="title">Added <em>remote_host</em></div><ul>
<div class="para"><p>You can update your configuration using <tt>tools/config-pre-0.7-to-0.7.sh</tt>.</p></div>
<div class="ilist"><div class="title">Added <em>remote_host</em></div><ul>
<li>
<p>
As of 0.7 it is possible to backup <strong>to</strong> hosts (see section remote_host below).
</p>
</li>
</ul></div>
<h4 id="_versions_0_5_and_0_6">1.3.2. Versions 0.5 and 0.6</h4>
<div class="ulist"><div class="title">The format of <tt>rsync_options</tt> changed:</div><ul>
<h4 id="_versions_0_5_and_0_6">1.3.3. Versions 0.5 and 0.6</h4>
<div class="ilist"><div class="title">The format of <tt>rsync_options</tt> changed:</div><ul>
<li>
<p>
Before 0.6 it was whitespace delimeted
@ -492,8 +474,8 @@ As of 0.6 it is newline seperated (so you can pass whitespaces to <tt>rsync</tt>
</p>
</li>
</ul></div>
<div class="paragraph"><p>You can update your configuration using <tt>tools/config-pre-0.6-to-0.6.sh</tt>.</p></div>
<div class="ulist"><div class="title">The name of the backup directories changed:</div><ul>
<div class="para"><p>You can update your configuration using <tt>tools/config-pre-0.6-to-0.6.sh</tt>.</p></div>
<div class="ilist"><div class="title">The name of the backup directories changed:</div><ul>
<li>
<p>
Before 0.6: "date +%Y-%m-%d-%H%M"
@ -505,11 +487,11 @@ As of 0.6: "date +%Y%m%d-%H%M" (better readable, date is closer together)
</p>
</li>
</ul></div>
<div class="paragraph"><p>For the second change there is no updated needed, as XXXX- is always before
<div class="para"><p>For the second change there is no updated needed, as XXXX- is always before
XXXXX (- comes before digit).</p></div>
<h4 id="_versions_0_4_and_0_5">1.3.3. Versions 0.4 and 0.5</h4>
<div class="paragraph"><p>Not a real incompatibilty, but seems to fit in this section:</p></div>
<div class="ulist"><div class="title">0.5 does <strong>NOT</strong> require</div><ul>
<h4 id="_versions_0_4_and_0_5">1.3.4. Versions 0.4 and 0.5</h4>
<div class="para"><p>Not a real incompatibilty, but seems to fit in this section:</p></div>
<div class="ilist"><div class="title">0.5 does <strong>NOT</strong> require</div><ul>
<li>
<p>
PaX
@ -521,11 +503,11 @@ bc
</p>
</li>
</ul></div>
<div class="paragraph"><p>anymore!</p></div>
<h4 id="_versions_lt_0_4_and_0_4">1.3.4. Versions &lt; 0.4 and 0.4</h4>
<div class="paragraph"><p>Since <tt>ccollect</tt> 0.4 there are several incompatibilities with earlier
<div class="para"><p>anymore!</p></div>
<h4 id="_versions_lt_0_4_and_0_4">1.3.5. Versions &lt; 0.4 and 0.4</h4>
<div class="para"><p>Since <tt>ccollect</tt> 0.4 there are several incompatibilities with earlier
versions:</p></div>
<div class="ulist"><div class="title">List of incompatibilities</div><ul>
<div class="ilist"><div class="title">List of incompatibilities</div><ul>
<li>
<p>
<tt>pax</tt> (Posix) is now required, <tt>cp -al</tt> (GNU specific) is removed
@ -552,7 +534,7 @@ ccollect now reports when postexec returns non-zero
</p>
</li>
</ul></div>
<div class="paragraph"><p>You can convert your old configuration directory using
<div class="para"><p>You can convert your old configuration directory using
<tt>config-pre-0.4-to-0.4.sh</tt>, which can be found in the <strong>tools/</strong>
subdirectory:</p></div>
<div class="listingblock">
@ -562,7 +544,7 @@ subdirectory:</p></div>
</div>
<h2 id="_quick_start">2. Quick start</h2>
<div class="sectionbody">
<div class="paragraph"><p>For those who do not want to read the whole long document:</p></div>
<div class="para"><p>For those who do not want to read the whole long document:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt># get latest ccollect tarball from http://www.nico.schottelius.org/software/ccollect/
@ -624,15 +606,15 @@ du -s ~/DASI /bin
# report success
echo "Please report success using ./tools/report_success.sh"</tt></pre>
</div></div>
<div class="paragraph"><p>Cutting and pasting the complete section above to your shell will result in
<div class="para"><p>Cutting and pasting the complete section above to your shell will result in
the download of ccollect, the creation of a sample configuration and the
execution of some backups.</p></div>
</div>
<h2 id="_requirements">3. Requirements</h2>
<div class="sectionbody">
<h3 id="_installing_ccollect">3.1. Installing ccollect</h3><div style="clear:left"></div>
<div class="paragraph"><p>For the installation you need at least</p></div>
<div class="ulist"><ul>
<div class="para"><p>For the installation you need at least</p></div>
<div class="ilist"><ul>
<li>
<p>
the latest ccollect package (<a href="http://www.nico.schottelius.org/software/ccollect/">http://www.nico.schottelius.org/software/ccollect/</a>)
@ -655,7 +637,7 @@ for rebuilding the generated documentation: additionally <tt>asciidoc</tt>
</li>
</ul></div>
<h3 id="_using_ccollect">3.2. Using ccollect</h3><div style="clear:left"></div>
<div class="ulist"><div class="title">Running ccollect requires the following tools to be installed:</div><ul>
<div class="ilist"><div class="title">Running ccollect requires the following tools to be installed:</div><ul>
<li>
<p>
<tt>date</tt>
@ -675,11 +657,11 @@ for rebuilding the generated documentation: additionally <tt>asciidoc</tt>
</div>
<h2 id="_installing">4. Installing</h2>
<div class="sectionbody">
<div class="paragraph"><p>Either type <em>make install</em> or simply copy it to a directory in your
<div class="para"><p>Either type <em>make install</em> or simply copy it to a directory in your
$PATH and execute <em>chmod <strong>0755</strong> /path/to/ccollect.sh</em>. If you would
like to use the new management scripts (available since 0.6), copy
the following scripts to a directory in $PATH:</p></div>
<div class="ulist"><ul>
<div class="ilist"><ul>
<li>
<p>
<tt>tools/ccollect_add_source.sh</tt>
@ -706,16 +688,16 @@ the following scripts to a directory in $PATH:</p></div>
</p>
</li>
</ul></div>
<div class="paragraph"><p>After having installed and used ccollect, report success using
<div class="para"><p>After having installed and used ccollect, report success using
<em>./tools/report_success.sh</em>.</p></div>
</div>
<h2 id="_configuring">5. Configuring</h2>
<div class="sectionbody">
<div class="paragraph"><p>For configuration aid have a look at the above mentioned tools, which can assist
<div class="para"><p>For configuration aid have a look at the above mentioned tools, which can assist
you quite well. When you are successfully using <tt>ccollect</tt>, report success using
<tt>tools/report_success.sh</tt>.</p></div>
<h3 id="_runtime_options">5.1. Runtime options</h3><div style="clear:left"></div>
<div class="paragraph"><p><tt>ccollect</tt> looks for its configuration in <em>/etc/ccollect</em> or, if set, in
<div class="para"><p><tt>ccollect</tt> looks for its configuration in <em>/etc/ccollect</em> or, if set, in
the directory specified by the variable <em>$CCOLLECT_CONF</em>:</p></div>
<div class="listingblock">
<div class="content">
@ -725,26 +707,26 @@ $ CCOLLECT_CONF=/your/config/dir ccollect.sh ...
# csh
$ ( setenv CCOLLECT_CONF /your/config/dir ; ccollect.sh ... )</tt></pre>
</div></div>
<div class="paragraph"><p>When you start <tt>ccollect</tt>, you have to specify in which interval
<div class="para"><p>When you start <tt>ccollect</tt>, you have to specify in which interval
to backup (daily, weekly, yearly; you can specify the names yourself, see below)
and which sources to backup (or -a to backup all sources).</p></div>
<div class="paragraph"><p>The interval specifies how many backups are kept.</p></div>
<div class="paragraph"><p>There are also some self-explanatory parameters you can pass to ccollect,
simply use <tt>ccollect.sh --help</tt> for info.</p></div>
<div class="para"><p>The interval specifies how many backups are kept.</p></div>
<div class="para"><p>There are also some self-explanatory parameters you can pass to ccollect,
simply use <tt>ccollect.sh &#8212;help</tt> for info.</p></div>
<h3 id="_general_configuration">5.2. General configuration</h3><div style="clear:left"></div>
<div class="paragraph"><p>The general configuration can be found in $CCOLLECT_CONF/defaults or
<div class="para"><p>The general configuration can be found in $CCOLLECT_CONF/defaults or
/etc/ccollect/defaults. All options specified there are generally valid for
all source definitions, although the values can be overwritten in the source
configuration.</p></div>
<div class="paragraph"><p>All configuration entries are plain-text files (use UTF-8 for non-ascii characters).</p></div>
<div class="para"><p>All configuration entries are plain-text files (use UTF-8 for non-ascii characters).</p></div>
<h4 id="_interval_definition">5.2.1. Interval definition</h4>
<div class="paragraph"><p>The interval definition can be found in
<div class="para"><p>The interval definition can be found in
<em>$CCOLLECT_CONF/defaults/intervals/</em> or <em>/etc/ccollect/defaults/intervals</em>.
Each file in this directory specifies an interval. The name of the file is
the same as the name of the interval: <tt>intervals/'&lt;interval name&gt;'</tt>.</p></div>
<div class="paragraph"><p>The content of this file should be a single line containing a number.
the same as the name of the interval: <tt>intervals/<em>&lt;interval name&gt;</em></tt>.</p></div>
<div class="para"><p>The content of this file should be a single line containing a number.
This number defines how many versions of this interval are kept.</p></div>
<div class="paragraph"><p>Example:</p></div>
<div class="para"><p>Example:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt> [10:23] zaphodbeeblebrox:ccollect-0.2% ls -l conf/defaults/intervals/
@ -757,13 +739,13 @@ This number defines how many versions of this interval are kept.</p></div>
12
4</tt></pre>
</div></div>
<div class="paragraph"><p>This means to keep 28 daily backups, 12 monthly backups and 4 weekly.</p></div>
<div class="para"><p>This means to keep 28 daily backups, 12 monthly backups and 4 weekly.</p></div>
<h4 id="_general_pre_and_post_execution">5.2.2. General pre- and post-execution</h4>
<div class="paragraph"><p>If you add <em>$CCOLLECT_CONF/defaults/<tt>pre_exec</tt></em> or
<div class="para"><p>If you add <em>$CCOLLECT_CONF/defaults/<tt>pre_exec</tt></em> or
<em>/etc/ccollect/defaults/<tt>pre_exec</tt></em> (same with <tt>post_exec</tt>), <tt>ccollect</tt>
will start <tt>pre_exec</tt> before the whole backup process and
<tt>post_exec</tt> after backup of all sources is done.</p></div>
<div class="paragraph"><p>The following example describes how to report free disk space in
<div class="para"><p>The following example describes how to report free disk space in
human readable format before and after the whole backup process:</p></div>
<div class="listingblock">
<div class="content">
@ -775,15 +757,15 @@ human readable format before and after the whole backup process:</p></div>
[13:01] hydrogenium:~# ln -s /etc/ccollect/defaults/pre_exec /etc/ccollect/defaults/post_exec</tt></pre>
</div></div>
<h4 id="_general_delete_incomplete">5.2.3. General delete_incomplete</h4>
<div class="paragraph"><p>If you add <em>$CCOLLECT_CONF/defaults/<tt>delete_incomplete</tt></em>, this
<div class="para"><p>If you add <em>$CCOLLECT_CONF/defaults/<tt>delete_incomplete</tt></em>, this
option applies for all sources. See below for a longer
explanation.</p></div>
<h3 id="_source_configuration">5.3. Source configuration</h3><div style="clear:left"></div>
<div class="paragraph"><p>Each source configuration exists in <em>$CCOLLECT_CONF/sources/$name</em> or
<div class="para"><p>Each source configuration exists in <em>$CCOLLECT_CONF/sources/$name</em> or
<em>/etc/ccollect/sources/$name</em>.</p></div>
<div class="paragraph"><p>The name you choose for the subdirectory describes the source.</p></div>
<div class="paragraph"><p>Each source contains at least the following files:</p></div>
<div class="ulist"><ul>
<div class="para"><p>The name you choose for the subdirectory describes the source.</p></div>
<div class="para"><p>Each source contains at least the following files:</p></div>
<div class="ilist"><ul>
<li>
<p>
<tt>source</tt> (a text file containing the <tt>rsync</tt> compatible path to backup)
@ -795,8 +777,8 @@ explanation.</p></div>
</p>
</li>
</ul></div>
<div class="paragraph"><p>Additionally a source may have the following files:</p></div>
<div class="ulist"><ul>
<div class="para"><p>Additionally a source may have the following files:</p></div>
<div class="ilist"><ul>
<li>
<p>
<tt>verbose</tt> whether to be verbose (passes -v to <tt>rsync</tt>)
@ -842,8 +824,23 @@ explanation.</p></div>
<tt>remote_host</tt> host to backup to
</p>
</li>
<li>
<p>
<tt>rsync_failure_codes</tt> list of rsync exit codes that indicate complete failure
</p>
</li>
<li>
<p>
<tt>mtime</tt> Sort backup directories based on their modification time
</p>
</li>
<li>
<p>
<tt>quiet_if_down</tt> Suppress error messages if source is not connectable
</p>
</li>
</ul></div>
<div class="paragraph"><p>Example:</p></div>
<div class="para"><p>Example:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt> [10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2
@ -865,56 +862,56 @@ explanation.</p></div>
/home/nico/vpn</tt></pre>
</div></div>
<h4 id="_detailed_description_of_source">5.3.1. Detailed description of "source"</h4>
<div class="paragraph"><p><tt>source</tt> describes a <tt>rsync</tt> compatible source (one line only).</p></div>
<div class="paragraph"><p>For instance <em>backup_user@foreign_host:/home/server/video</em>.
<div class="para"><p><tt>source</tt> describes a <tt>rsync</tt> compatible source (one line only).</p></div>
<div class="para"><p>For instance <em>backup_user@foreign_host:/home/server/video</em>.
To use the <tt>rsync</tt> protocol without the <tt>ssh</tt>-tunnel, use
<em>rsync::USER@HOST/SRC</em>. For more information have a look at the manpage
of <tt>rsync</tt>(1).</p></div>
<h4 id="_detailed_description_of_destination">5.3.2. Detailed description of "destination"</h4>
<div class="paragraph"><p><tt>destination</tt> must be a text file containing the destination directory.
<div class="para"><p><tt>destination</tt> must be a text file containing the destination directory.
<tt>destination</tt> <strong>USED</strong> to be a link to the destination directory in
earlier versions, so do not be confused if you see such examples.</p></div>
<div class="paragraph"><p>Example:</p></div>
<div class="para"><p>Example:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt> [11:36] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/destination
/home/nico/backupdir</tt></pre>
</div></div>
<h4 id="_detailed_description_of_remote_host">5.3.3. Detailed description of "remote_host"</h4>
<div class="paragraph"><p><tt>remote_host</tt> must be a text file containing the destination host.
<div class="para"><p><tt>remote_host</tt> must be a text file containing the destination host.
If this file is existing, you are backing up your data <strong>TO</strong> this host
and <strong>not</strong> to you local host.</p></div>
<div class="paragraph"><p><strong>Warning</strong>: You need to have <tt>ssh</tt> access to the remote host. <tt>rsync</tt> and
<div class="para"><p><strong>Warning</strong>: You need to have <tt>ssh</tt> access to the remote host. <tt>rsync</tt> and
<tt>ccollect</tt> will connect to that host via <tt>ssh</tt>. <tt>ccollect</tt> needs the shell
access, because it needs to find out how many backups exist on the remote
host and to be able to delete them.</p></div>
<div class="paragraph"><p>Example:</p></div>
<div class="para"><p>Example:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt> [10:17] denkbrett:ccollect-0.7.0% cat conf/sources/remote1/remote_host
home.schottelius.org</tt></pre>
</div></div>
<div class="paragraph"><p>It may contain all the ssh-specific values like <em>myuser@yourhost.ch</em>.</p></div>
<div class="para"><p>It may contain all the ssh-specific values like <em>myuser@yourhost.ch</em>.</p></div>
<h4 id="_detailed_description_of_verbose">5.3.4. Detailed description of "verbose"</h4>
<div class="paragraph"><p><tt>verbose</tt> tells <tt>ccollect</tt> that the log should contain verbose messages.</p></div>
<div class="paragraph"><p>If this file exists in the source specification <strong>-v</strong> will be passed to <tt>rsync</tt>.</p></div>
<div class="paragraph"><p>`
<div class="para"><p><tt>verbose</tt> tells <tt>ccollect</tt> that the log should contain verbose messages.</p></div>
<div class="para"><p>If this file exists in the source specification <strong>-v</strong> will be passed to <tt>rsync</tt>.</p></div>
<div class="para"><p>`
Example:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt> [11:35] zaphodbeeblebrox:ccollect-0.2% touch conf/sources/testsource1/verbose</tt></pre>
</div></div>
<h4 id="_detailed_description_of_very_verbose">5.3.5. Detailed description of "very_verbose"</h4>
<div class="paragraph"><p><tt>very_verbose</tt> tells <tt>ccollect</tt> that it should log very verbosely.</p></div>
<div class="paragraph"><p>If this file exists in the source specification <strong>-v</strong> will be passed to
<div class="para"><p><tt>very_verbose</tt> tells <tt>ccollect</tt> that it should log very verbosely.</p></div>
<div class="para"><p>If this file exists in the source specification <strong>-v</strong> will be passed to
<tt>rsync</tt>, <tt>rm</tt> and <tt>mkdir</tt>.</p></div>
<div class="paragraph"><p>Example:</p></div>
<div class="para"><p>Example:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt> [23:67] nohost:~% touch conf/sources/testsource1/very_verbose</tt></pre>
</div></div>
<h4 id="_detailed_description_of_summary">5.3.6. Detailed description of "summary"</h4>
<div class="paragraph"><p>If you create the file <tt>summary</tt> in the source definition,
<div class="para"><p>If you create the file <tt>summary</tt> in the source definition,
<tt>ccollect</tt> will present you a nice summary at the end.</p></div>
<div class="listingblock">
<div class="content">
@ -944,12 +941,12 @@ backup:~# ccollect.sh werktags root
[root] Successfully finished backup.
==&gt; Finished ccollect.sh &lt;==</tt></pre>
</div></div>
<div class="paragraph"><p>You could also combine it with <tt>verbose</tt> or <tt>very_verbose</tt>, but these
<div class="para"><p>You could also combine it with <tt>verbose</tt> or <tt>very_verbose</tt>, but these
already print some statistics (though not all / the same as presented by
<tt>summary</tt>).</p></div>
<h4 id="_detailed_description_of_exclude">5.3.7. Detailed description of "exclude"</h4>
<div class="paragraph"><p><tt>exclude</tt> specifies a list of paths to exclude. The entries are seperated by a newline (\n).</p></div>
<div class="paragraph"><p>Example:</p></div>
<div class="para"><p><tt>exclude</tt> specifies a list of paths to exclude. The entries are seperated by a newline (\n).</p></div>
<div class="para"><p>Example:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt> [11:35] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
@ -959,10 +956,10 @@ already print some statistics (though not all / the same as presented by
something with spaces is not a problem</tt></pre>
</div></div>
<h4 id="_detailed_description_of_intervals">5.3.8. Detailed description of "intervals/"</h4>
<div class="paragraph"><p>When you create the subdirectory <tt>intervals/</tt> in your source configuration
<div class="para"><p>When you create the subdirectory <tt>intervals/</tt> in your source configuration
directory, you can specify individiual intervals for this specific source.
Each file in this directory describes an interval.</p></div>
<div class="paragraph"><p>Example:</p></div>
<div class="para"><p>Example:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt> [11:37] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervals/
@ -974,24 +971,24 @@ Each file in this directory describes an interval.</p></div>
20</tt></pre>
</div></div>
<h4 id="_detailled_description_of_rsync_options">5.3.9. Detailled description of "rsync_options"</h4>
<div class="paragraph"><p>When you create the file <tt>rsync_options</tt> in your source configuration,
<div class="para"><p>When you create the file <tt>rsync_options</tt> in your source configuration,
all the parameters in this file will be passed to rsync. This
way you can pass additional options to rsync. For instance you can tell rsync
to show progress ("--progress"), or which -password-file ("--password-file")
to show progress ("&#8212;progress"), or which -password-file ("&#8212;password-file")
to use for automatic backup over the rsync-protocol.</p></div>
<div class="paragraph"><p>Example:</p></div>
<div class="para"><p>Example:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt> [23:42] hydrogenium:ccollect-0.2% cat conf/sources/test_rsync/rsync_options
--password-file=/home/user/backup/protected_password_file</tt></pre>
</div></div>
<h4 id="_detailled_description_of_pre_exec_and_post_exec">5.3.10. Detailled description of "pre_exec" and "post_exec"</h4>
<div class="paragraph"><p>When you create <tt>pre_exec</tt> and / or <tt>post_exec</tt> in your source
<div class="para"><p>When you create <tt>pre_exec</tt> and / or <tt>post_exec</tt> in your source
configuration, <tt>ccollect</tt> will execute this command before and
respectively after doing the backup for <strong>this specific</strong> source.
If you want to have pre-/post-exec before and after <strong>all</strong>
backups, see above for general configuration.</p></div>
<div class="paragraph"><p>Example:</p></div>
<div class="para"><p>Example:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>[13:09] hydrogenium:ccollect-0.3% cat conf/sources/with_exec/pre_exec
@ -1006,15 +1003,38 @@ df -h
df -h</tt></pre>
</div></div>
<h4 id="_detailed_description_of_delete_incomplete">5.3.11. Detailed description of "delete_incomplete"</h4>
<div class="paragraph"><p>If you create the file <tt>delete_incomplete</tt> in a source specification directory,
<div class="para"><p>If you create the file <tt>delete_incomplete</tt> in a source specification directory,
<tt>ccollect</tt> will look for incomplete backups (when the whole <tt>ccollect</tt> process
was interrupted) and remove them. Without this file <tt>ccollect</tt> will only warn
the user.</p></div>
<h4 id="_detailed_description_of_rsync_failure_codes">5.3.12. Detailed description of "rsync_failure_codes"</h4>
<div class="para"><p>If you have the file <tt>rsync_failure_codes</tt> in your source configuration
directory, it should contain a newline-separated list of numbers representing
rsync exit codes. If rsync exits with any code in this list, a marker will
be left in the destination directory indicating failure of this backup. If
you have enabled delete_incomplete, then this backup will be deleted during
the next ccollect run on the same interval.</p></div>
<h4 id="_detailed_description_of_mtime">5.3.13. Detailed description of "mtime"</h4>
<div class="para"><p>By default, ccollect.sh chooses the most recent backup directory for cloning or
the oldest for deletion based on the directory's last change time (ctime).
With this option, the sorting is done based on modification time (mtime). With
this version of ccollect, the ctime and mtime of your backups will normally
be the same and this option has no effect. However, if you, for example, move
your backups to another hard disk using cp -a or rsync -a, you should use this
option because the ctimes are not preserved during such operations.</p></div>
<div class="para"><p>If you have any backups in your repository made with ccollect version 0.7.1 or
earlier, do not use this option.</p></div>
<h4 id="_detailed_description_of_quiet_if_down">5.3.14. Detailed description of "quiet_if_down"</h4>
<div class="para"><p>By default, ccollect.sh emits a series of error messages if a source is not
connectable. With this option enabled, ccollect still reports that the
source is not connectable but the associated error messages generated by
rsync or ssh are suppressed. You may want to use this option for sources,
like notebook PCs, that are often disconnected.</p></div>
</div>
<h2 id="_hints">6. Hints</h2>
<div class="sectionbody">
<h3 id="_smart_logging">6.1. Smart logging</h3><div style="clear:left"></div>
<div class="paragraph"><p>Since ccollect-0.6.1 you can use the ccollect-logwrapper.sh(1) for logging.
<div class="para"><p>Since ccollect-0.6.1 you can use the ccollect-logwrapper.sh(1) for logging.
You call it the same way you call ccollect.sh and it will create a
logfile containing the output of ccollect.sh. For more information look
at the manpage <em>ccollect-logwrapper</em>. The following is an example running
@ -1026,13 +1046,13 @@ ccollect-logwrapper.sh (11722): Starting with arguments: daily u0160.nshq.ch.net
ccollect-logwrapper.sh (11722): Finished.</tt></pre>
</div></div>
<h3 id="_using_a_different_ssh_port">6.2. Using a different ssh port</h3><div style="clear:left"></div>
<div class="paragraph"><p>Mostly easy is to use your ~/.ssh/config file:</p></div>
<div class="para"><p>Mostly easy is to use your ~/.ssh/config file:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>host mx2.schottelius.org
Port 2342</tt></pre>
</div></div>
<div class="paragraph"><p>If you only use that port for backup only and normally want to use another port,
<div class="para"><p>If you only use that port for backup only and normally want to use another port,
you can add <em>HostName</em> and "HostKeyAlias" (if you also have different
keys on the different ports):</p></div>
<div class="listingblock">
@ -1048,8 +1068,8 @@ Host bruehe
HostKeyAlias bruehe.schottelius.org</tt></pre>
</div></div>
<h3 id="_using_source_names_or_interval_in_pre_post_exec_scripts">6.3. Using source names or interval in pre_/post_exec scripts</h3><div style="clear:left"></div>
<div class="paragraph"><p>The pre-/post_exec scripts can access some internal variables from <tt>ccollect</tt>:</p></div>
<div class="ulist"><ul>
<div class="para"><p>The pre-/post_exec scripts can access some internal variables from <tt>ccollect</tt>:</p></div>
<div class="ilist"><ul>
<li>
<p>
INTERVAL: The interval specified on the command line
@ -1073,11 +1093,11 @@ name: the name of the currently being backuped source (not available for
</li>
</ul></div>
<h3 id="_using_rsync_protocol_without_ssh">6.4. Using rsync protocol without ssh</h3><div style="clear:left"></div>
<div class="paragraph"><p>When you have a computer with little computing power, it may be useful to use
<div class="para"><p>When you have a computer with little computing power, it may be useful to use
rsync without ssh, directly using the rsync protocol
(specify <em>user@host::share</em> in <tt>source</tt>). You may wish to use
<tt>rsync_options</tt> to specify a password file to use for automatic backup.</p></div>
<div class="paragraph"><p>Example:</p></div>
<div class="para"><p>Example:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>backup:~# cat /etc/ccollect/sources/sample.backup.host.org/source
@ -1089,10 +1109,10 @@ backup:~# cat /etc/ccollect/sources/sample.backup.host.org/rsync_options
backup:~# cat /etc/ccollect/sources/sample.backup.host.org/rsync_password
this_is_the_rsync_password</tt></pre>
</div></div>
<div class="paragraph"><p>This hint was reported by Daniel Aubry.</p></div>
<div class="para"><p>This hint was reported by Daniel Aubry.</p></div>
<h3 id="_not_excluding_top_level_directories">6.5. Not excluding top-level directories</h3><div style="clear:left"></div>
<div class="paragraph"><p>When you exclude "/proc" or "/mnt" from your backup, you may run into
trouble when you restore your backup. When you use "/proc/*" or "/mnt/\*"
<div class="para"><p>When you exclude "/proc" or "/mnt" from your backup, you may run into
trouble when you restore your backup. When you use "/proc/*" or "/mnt/*"
instead, <tt>ccollect</tt> will backup empty directories.</p></div>
<div class="admonitionblock">
<table><tr>
@ -1100,19 +1120,19 @@ instead, <tt>ccollect</tt> will backup empty directories.</p></div>
<div class="title">Note</div>
</td>
<td class="content">
<div class="paragraph"><p>When those directories contain hidden files
<div class="para"><p>When those directories contain hidden files
(those beginning with a dot (<strong>.</strong>)),
they will still be transferred!</p></div>
</td>
</tr></table>
</div>
<div class="paragraph"><p>This hint was reported by Marcus Wagner.</p></div>
<div class="para"><p>This hint was reported by Marcus Wagner.</p></div>
<h3 id="_re_using_already_created_rsync_backups">6.6. Re-using already created rsync-backups</h3><div style="clear:left"></div>
<div class="paragraph"><p>If you used <tt>rsync</tt> directly before you use <tt>ccollect</tt>, you can
<div class="para"><p>If you used <tt>rsync</tt> directly before you use <tt>ccollect</tt>, you can
use this old backup as initial backup for <tt>ccollect</tt>: You
simply move it into a directory below the destination directory
and name it "<em>interval</em>.0".</p></div>
<div class="paragraph"><p>Example:</p></div>
<div class="para"><p>Example:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>backup:/home/backup/web1# ls
@ -1127,14 +1147,14 @@ backup:/home/backup/web1# mv * daily.0 2&gt;/dev/null
backup:/home/backup/web1# ls
daily.0</tt></pre>
</div></div>
<div class="paragraph"><p>Now you can use /home/backup/web1 as the <tt>destination</tt> for the backup.</p></div>
<div class="para"><p>Now you can use /home/backup/web1 as the <tt>destination</tt> for the backup.</p></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">
<div class="paragraph"><p>It does not matter anymore how you name your directory, as <tt>ccollect</tt> uses
<div class="para"><p>It does not matter anymore how you name your directory, as <tt>ccollect</tt> uses
the -c option from <tt>ls</tt> to find out which directory to clone from.</p></div>
</td>
</tr></table>
@ -1145,7 +1165,7 @@ the -c option from <tt>ls</tt> to find out which directory to clone from.</p></d
<div class="title">Note</div>
</td>
<td class="content">
<div class="paragraph"><p>Older versions (pre 0.6, iirc) had a problem, if you named the first backup
<div class="para"><p>Older versions (pre 0.6, iirc) had a problem, if you named the first backup
something like "daily.initial". It was needed to use the "<strong>0</strong>" (or some
number that is lower than the current year) as extension. <tt>ccollect</tt>
used <tt>sort</tt> to find the latest backup. <tt>ccollect</tt> itself uses
@ -1159,9 +1179,9 @@ wasted resources, but your backup had beer complete anyway.</p></div>
</tr></table>
</div>
<h3 id="_using_pre_post_exec">6.7. Using pre_/post_exec</h3><div style="clear:left"></div>
<div class="paragraph"><p>Your pre_/post_exec script does not need to be a script, you can also
<div class="para"><p>Your pre_/post_exec script does not need to be a script, you can also
use a link to</p></div>
<div class="ulist"><ul>
<div class="ilist"><ul>
<li>
<p>
an existing program
@ -1173,40 +1193,40 @@ an already written script
</p>
</li>
</ul></div>
<div class="paragraph"><p>The only requirement is that it is executable.</p></div>
<div class="para"><p>The only requirement is that it is executable.</p></div>
<h3 id="_using_source_specific_interval_definitions">6.8. Using source specific interval definitions</h3><div style="clear:left"></div>
<div class="paragraph"><p>When you are backing up multiple hosts via cron each night, it may be
<div class="para"><p>When you are backing up multiple hosts via cron each night, it may be
a problem that host "big_server" may only have 4 daily backups, because
otherwise its backup device will be full. But for all other hosts
you want to keep 20 daily backups. In this case you would create
<tt>/etc/ccollect/default/intervals/daily</tt> containing "20" and
<tt>/etc/ccollect/sources/big_server/intervals/daily</tt> containing "4".</p></div>
<div class="paragraph"><p>Source specific intervals always overwrite the default values.
<div class="para"><p>Source specific intervals always overwrite the default values.
If you have to specify it individually for every host, because
of different requirements, you can even omit creating
<tt>/etc/ccollect/default/intervals/daily</tt>.</p></div>
<h3 id="_comparing_backups">6.9. Comparing backups</h3><div style="clear:left"></div>
<div class="paragraph"><p>If you want to see what changed between two backups, you can use
<div class="para"><p>If you want to see what changed between two backups, you can use
<tt>rsync</tt> directly:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>[12:00] u0255:ddba034.netstream.ch# rsync -n -a --delete --stats --progress daily.20080324-0313.17841/ daily.20080325-0313.31148/</tt></pre>
</div></div>
<div class="paragraph"><p>This results in a listing of changes. Because we pass -n to rsync no transfer
<div class="para"><p>This results in a listing of changes. Because we pass -n to rsync no transfer
is made (i.e. report only mode)"</p></div>
<div class="paragraph"><p>This hint was reported by Daniel Aubry.</p></div>
<div class="para"><p>This hint was reported by Daniel Aubry.</p></div>
<h3 id="_testing_for_host_reachabilty">6.10. Testing for host reachabilty</h3><div style="clear:left"></div>
<div class="paragraph"><p>If you want to test whether the host you try to backup is reachable, you can use
<div class="para"><p>If you want to test whether the host you try to backup is reachable, you can use
the following script as source specific pre-exec:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>#!/bin/sh
# ping -c1 -q `cat "/etc/ccollect/sources/$name/source" | cut -d"@" -f2 | cut -d":" -f1`</tt></pre>
</div></div>
<div class="paragraph"><p>This prevents the deletion of old backups, if the host is not reachable.</p></div>
<div class="paragraph"><p>This hint was reported by Daniel Aubry.</p></div>
<div class="para"><p>This prevents the deletion of old backups, if the host is not reachable.</p></div>
<div class="para"><p>This hint was reported by Daniel Aubry.</p></div>
<h3 id="_easy_check_for_errors">6.11. Easy check for errors</h3><div style="clear:left"></div>
<div class="paragraph"><p>If you want to see whether there have been any errors while doing the backup,
<div class="para"><p>If you want to see whether there have been any errors while doing the backup,
you can run <tt>ccollect</tt> together with <tt>ccollect_analyse_logs.sh</tt>:</p></div>
<div class="listingblock">
<div class="content">
@ -1216,11 +1236,11 @@ you can run <tt>ccollect</tt> together with <tt>ccollect_analyse_logs.sh</tt>:</
<h2 id="_f_a_q">7. F.A.Q.</h2>
<div class="sectionbody">
<h3 id="_what_happens_if_one_backup_is_broken_or_empty">7.1. What happens if one backup is broken or empty?</h3><div style="clear:left"></div>
<div class="paragraph"><p>Let us assume that one backup failed (connection broke or the source
hard disk had some failures). Therefore we&#8217;ve got one incomplete backup in our history.</p></div>
<div class="paragraph"><p><tt>ccollect</tt> will transfer the missing files the next time you use it.
<div class="para"><p>Let us assume that one backup failed (connection broke or the source
hard disk had some failures). Therefore we've got one incomplete backup in our history.</p></div>
<div class="para"><p><tt>ccollect</tt> will transfer the missing files the next time you use it.
This leads to</p></div>
<div class="ulist"><ul>
<div class="ilist"><ul>
<li>
<p>
more transferred files
@ -1232,24 +1252,24 @@ much greater disk space usage, as no hardlinks can be used
</p>
</li>
</ul></div>
<div class="paragraph"><p>If the whole <tt>ccollect</tt> process was interrupted, <tt>ccollect</tt> (since 0.6) can
<div class="para"><p>If the whole <tt>ccollect</tt> process was interrupted, <tt>ccollect</tt> (since 0.6) can
detect that and remove the incomplete backups, so you can clone from a complete
backup instead</p></div>
<h3 id="_when_backing_up_from_localhost_the_destination_is_also_included_is_this_a_bug">7.2. When backing up from localhost the destination is also included. Is this a bug?</h3><div style="clear:left"></div>
<div class="paragraph"><p>No. <tt>ccollect</tt> passes your source definition directly to <tt>rsync</tt>. It
<div class="para"><p>No. <tt>ccollect</tt> passes your source definition directly to <tt>rsync</tt>. It
does not try to analyze it. So it actually does not know if a source
comes from local harddisk or from a remote server. And it does not want
to. When you backup from the local harddisk (which is perhaps not
even a good idea when thinking of security), add the <tt>destination</tt>
to <em>source/exclude</em>. (Daniel Aubry reported this problem)</p></div>
<h3 id="_why_does_ccollect_say_permission_denied_with_my_pre_postexec_script">7.3. Why does ccollect say "Permission denied" with my pre-/postexec script?</h3><div style="clear:left"></div>
<div class="paragraph"><p>The most common error is that you have not given your script the correct
permissions. Try <tt>chmod 0755 /etc/ccollect/sources/'yoursource'/*_exec</tt>`.</p></div>
<div class="para"><p>The most common error is that you have not given your script the correct
permissions. Try <tt>chmod 0755 /etc/ccollect/sources/<em>yoursource</em>/*_exec</tt>`.</p></div>
<h3 id="_why_does_the_backup_job_fail_when_part_of_the_source_is_a_link">7.4. Why does the backup job fail when part of the source is a link?</h3><div style="clear:left"></div>
<div class="paragraph"><p>When a part of your path you specified in the source is a
<div class="para"><p>When a part of your path you specified in the source is a
(symbolic, hard links are not possible for directories) link,
the backup <strong>must</strong> fail.</p></div>
<div class="paragraph"><p>First of all, let us have a look at how it looks like:</p></div>
<div class="para"><p>First of all, let us have a look at how it looks like:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>==&gt; ccollect 0.4: Beginning backup using interval taeglich &lt;==
@ -1262,7 +1282,7 @@ the backup <strong>must</strong> fail.</p></div>
[testsource] rsync: stat "/etc/ccollect/sources/testsource/destination/taeglich.2006-04-29-0001.3874/home/user/nico/projekte/ccollect" failed: No such file or directory (2)
[...]</tt></pre>
</div></div>
<div class="paragraph"><p>So what is the problem? It is very obvious when you look deeper into it:</p></div>
<div class="para"><p>So what is the problem? It is very obvious when you look deeper into it:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>% cat /etc/ccollect/sources/testsource/source
@ -1272,17 +1292,17 @@ lrwxrwxrwx 1 nico nico 29 2005-12-02 23:28 /home/user/nico/projekte -&gt; oeffen
% ls -l /etc/ccollect/sources/testsource/destination/taeglich.2006-04-29-0001.3874/home/user/nico
lrwxrwxrwx 1 nico nico 29 2006-04-29 00:01 projekte -&gt; oeffentlich/computer/projekte</tt></pre>
</div></div>
<div class="paragraph"><p><tt>rsync</tt> creates the directory structure before it creates the symbolic link.
<div class="para"><p><tt>rsync</tt> creates the directory structure before it creates the symbolic link.
This link now links to something not reachable (dead link). It is
impossible to create subdirectories under the broken link.</p></div>
<div class="paragraph"><p>In conclusion you cannot use paths with a linked part.</p></div>
<div class="paragraph"><p>However, you can backup directories containing symbolic links
<div class="para"><p>In conclusion you cannot use paths with a linked part.</p></div>
<div class="para"><p>However, you can backup directories containing symbolic links
(in this case you could backup /home/user/nico, which contains
/home/user/nico/projekte and oeffentlich/computer/projekte).</p></div>
<h3 id="_how_can_i_prevent_missing_the_right_time_to_enter_my_password">7.5. How can I prevent missing the right time to enter my password?</h3><div style="clear:left"></div>
<div class="paragraph"><p>As <tt>ccollect</tt> first deletes the old backups, it may take some time
<div class="para"><p>As <tt>ccollect</tt> first deletes the old backups, it may take some time
until <tt>rsync</tt> requests the password for the <tt>ssh</tt> session from you.</p></div>
<div class="paragraph"><p>The easiest way not to miss that point is running <tt>ccollect</tt> in <tt>screen</tt>,
<div class="para"><p>The easiest way not to miss that point is running <tt>ccollect</tt> in <tt>screen</tt>,
which has the ability to monitor the output for activity. So as soon as
your screen beeps, after <tt>ccollect</tt> began to remove the last directory,
you can enter your password (have a look at screen(1), especially "C-a M"
@ -1316,7 +1336,7 @@ srwali01:/etc/ccollect/sources/local-root# ccollect.sh taeglich local-root
/=&gt; Beginning to backup "local-root" ...
|-&gt; 0 backup(s) already exist, keeping 28 backup(s).</tt></pre>
</div></div>
<div class="paragraph"><p>After that, I added some more sources:</p></div>
<div class="para"><p>After that, I added some more sources:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>srwali01:~# cd /etc/ccollect/sources
@ -1365,7 +1385,7 @@ srwali01:/etc/ccollect/sources# du -sh /mnt/hdbackup/wl6/*
147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:42.312
147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:45.588</tt></pre>
</div></div>
<div class="paragraph"><p>The backup of our main fileserver:</p></div>
<div class="para"><p>The backup of our main fileserver:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>backup:~# df -h /home/backup/srsyg01/
@ -1387,7 +1407,7 @@ backup:~# du -sh /home/backup/srsyg01/*
backup:~# du --version | head -n1
du (coreutils) 5.2.1</tt></pre>
</div></div>
<div class="paragraph"><p>Newer versions of du also detect the hardlinks, so we can even compare
<div class="para"><p>Newer versions of du also detect the hardlinks, so we can even compare
the sizes directly with du:</p></div>
<div class="listingblock">
<div class="content">
@ -1404,10 +1424,10 @@ du (GNU coreutils) 5.93
1.5G hydrogenium/durcheinander.2006-01-25-23:18.31328
200M hydrogenium/durcheinander.2006-01-26-00:11.3332</tt></pre>
</div></div>
<div class="paragraph"><p>In the second report (without -l) the sizes include the space the inodes of
<div class="para"><p>In the second report (without -l) the sizes include the space the inodes of
the hardlinks allocate.</p></div>
<h3 id="_a_collection_of_backups_on_the_backup_server">8.3. A collection of backups on the backup server</h3><div style="clear:left"></div>
<div class="paragraph"><p>All the data of my important hosts is backuped to eiche into
<div class="para"><p>All the data of my important hosts is backuped to eiche into
/mnt/schwarzesloch/backup:</p></div>
<div class="listingblock">
<div class="content">
@ -1432,7 +1452,7 @@ woechentlich.2006-02-06-08:22.15994 woechentlich.2006-03-02-23:00.17346
woechentlich.2006-02-06-19:40.16321 woechentlich.2006-03-09-23:00.29317
woechentlich.2006-02-12-11:51.2514 woechentlich.2006-03-16-23:00.4218</tt></pre>
</div></div>
<div class="paragraph"><p>And this incremental backup and the archive are copied to an external
<div class="para"><p>And this incremental backup and the archive are copied to an external
usb harddisk (attention: you <strong>should</strong> really use -H to backup the backup):</p></div>
<div class="listingblock">
<div class="content">
@ -1458,7 +1478,7 @@ rsync -av -H --delete /mnt/schwarzesloch/ "$DDIR/schwarzesloch/"
rsync -av -H --delete /mnt/archiv/ "$DDIR/archiv/"</tt></pre>
</div></div>
<h3 id="_processes_running_when_doing_ccollect_p">8.4. Processes running when doing ccollect -p</h3><div style="clear:left"></div>
<div class="paragraph"><p>Truncated output from <tt>ps axuwwwf</tt>:</p></div>
<div class="para"><p>Truncated output from <tt>ps axuwwwf</tt>:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt> S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily -p ddba034 ddba045 ddba046 ddba047 ddba049 ddna010 ddna011
@ -1500,13 +1520,13 @@ rsync -av -H --delete /mnt/archiv/ "$DDIR/archiv/"</tt></pre>
S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddna011
S+ 11:40 0:00 | | | \_ sed s:^:\[ddna011\] :</tt></pre>
</div></div>
<div class="paragraph"><p>As you can see, six processes are deleting old backups, while one backup
<div class="para"><p>As you can see, six processes are deleting old backups, while one backup
(ddba034) is already copying data.</p></div>
</div>
<div id="footer">
<div id="footer-text">
Version 0.7.1<br />
Last updated 2009-07-08 16:07:19 CEST
Last updated 2009-07-29 08:01:46 CEST
</div>
</div>
</body>

View File

@ -21,7 +21,7 @@ Supported and tested operating systems and architectures
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
`ccollect` was successfully tested on the following platforms:
- GNU/Linux on amd64/hppa/i386/ppc
- GNU/Linux on amd64/hppa/i386/ppc/ARM
- FreeBSD on amd64/i386
- Mac OS X 10.5
- NetBSD on alpha/amd64/i386/sparc/sparc64
@ -68,8 +68,22 @@ machine, she will not be able to log in on the backup machine.
All other backups are still secure.
Incompatibilities
~~~~~~~~~~~~~~~~~
Incompatibilities and changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Versions 0.7 and 0.8
^^^^^^^^^^^^^^^^^^^^^
. The argument order changed:
- Old: "<interval name> [args] <sources to backup>"
- New: "[args] <interval name> <sources to backup>"
If you did not use arguments (most people do not), nothing will
change for you.
.
Versions 0.6 and 0.7
@ -359,6 +373,9 @@ Additionally a source may have the following files:
- `delete_incomplete` delete incomplete backups
- `remote_host` host to backup to
- `rsync_failure_codes` list of rsync exit codes that indicate complete failure
- `mtime` Sort backup directories based on their modification time
- `quiet_if_down` Suppress error messages if source is not connectable
Example:
@ -574,6 +591,36 @@ If you create the file `delete_incomplete` in a source specification directory,
was interrupted) and remove them. Without this file `ccollect` will only warn
the user.
Detailed description of "rsync_failure_codes"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you have the file `rsync_failure_codes` in your source configuration
directory, it should contain a newline-separated list of numbers representing
rsync exit codes. If rsync exits with any code in this list, a marker will
be left in the destination directory indicating failure of this backup. If
you have enabled delete_incomplete, then this backup will be deleted during
the next ccollect run on the same interval.
Detailed description of "mtime"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
By default, ccollect.sh chooses the most recent backup directory for cloning or
the oldest for deletion based on the directory's last change time (ctime).
With this option, the sorting is done based on modification time (mtime). With
this version of ccollect, the ctime and mtime of your backups will normally
be the same and this option has no effect. However, if you, for example, move
your backups to another hard disk using cp -a or rsync -a, you should use this
option because the ctimes are not preserved during such operations.
If you have any backups in your repository made with ccollect version 0.7.1 or
earlier, do not use this option.
Detailed description of "quiet_if_down"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
By default, ccollect.sh emits a series of error messages if a source is not
connectable. With this option enabled, ccollect still reports that the
source is not connectable but the associated error messages generated by
rsync or ssh are suppressed. You may want to use this option for sources,
like notebook PCs, that are often disconnected.
Hints
-----