<html><head><metahttp-equiv="Content-Type"content="text/html; charset=ISO-8859-1"><title>ccollect - Installing, Configuring and Using</title><metaname="generator"content="DocBook XSL Stylesheets V1.76.1"></head><bodybgcolor="white"text="black"link="#0000FF"vlink="#840084"alink="#0000FF"><divlang="en"class="article"title="ccollect - Installing, Configuring and Using"><divclass="titlepage"><div><div><h2class="title"><aname="idp62409320"></a>ccollect - Installing, Configuring and Using</h2></div><div><divclass="author"><h3class="author"><spanclass="firstname">Nico</span><spanclass="surname">Schottelius</span></h3><codeclass="email"><<aclass="email"href="mailto:nico-ccollect__@__schottelius.org">nico-ccollect__@__schottelius.org</a>></code></div></div><div><divclass="revhistory"><tableborder="1"width="100%"summary="Revision history"><tr><thalign="left"valign="top"colspan="3"><b>Revision History</b></th></tr><tr><tdalign="left">Revision 2.2</td><tdalign="left">for ccollect 2.2, Initial Version from 2006-01-13</td><tdalign="left">NS</td></tr></table></div></div></div><hr></div><divclass="toc"><p><b>Table of Contents</b></p><dl><dt><spanclass="section"><ahref="#_introduction">Introduction</a></span></dt><dd><dl><dt><spanclass="section"><ahref="#_supported_and_tested_operating_systems_and_architectures">Supported and tested operating systems and architectures</a></span></dt><dt><spanclass="section"><ahref="#_why_you_could_only_backup_from_remote_hosts_not_to_them">Why you COULD only backup from remote hosts, not to them</a></span></dt><dt><spanclass="section"><ahref="#_incompatibilities_and_changes">Incompatibilities and changes</a></span></dt></dl></dd><dt><spanclass="section"><ahref="#_quick_start">Quick start</a></span></dt><dt><spanclass="section"><ahref="#_requirements">Requirements</a></span></dt><dd><dl><dt><spanclass="section"><ahref="#_installing_ccollect">Installing ccollect</a></span></dt><dt><spanclass="section"><ahref="#_using_ccollect">Using ccollect</a></span></dt></dl></dd><dt><spanclass="section"><ahref="#_installing">Installing</a></span></dt><dt><spanclass="section"><ahref="#_configuring">Configuring</a></span></dt><dd><dl><dt><spanclass="section"><ahref="#_runtime_options">Runtime options</a></span></dt><dt><spanclass="section"><ahref="#_general_configuration">General configuration</a></span></dt><dt><spanclass="section"><ahref="#_source_configuration">Source configuration</a></span></dt></dl></dd><dt><spanclass="section"><ahref="#_hints">Hints</a></span></dt><dd><dl><dt><spanclass="section"><ahref="#_smart_logging">Smart logging</a></span></dt><dt><spanclass="section"><ahref="#_using_a_different_ssh_port">Using a different ssh port</a></span></dt><dt><spanclass="section"><ahref="#_using_source_names_or_interval_in_pre__post_exec_scripts">Using source names or interval in pre_/post_exec scripts</a></span></dt><dt><spanclass="section"><ahref="#_using_rsync_protocol_without_ssh">Using rsync protocol without ssh</a></span></dt><dt><spanclass="section"><ahref="#_not_excluding_top_level_directories">Not excluding top-level directories</a></span></dt><dt><spanclass="section"><ahref="#_re_using_already_created_rsync_backups">Re-using already created rsync-backups</a></span></dt><dt><spanclass="section"><ahref="#_using_pre__post_exec">Using pre_/post_exec</a></span></dt><dt><spanclass="section"><ahref="#_using_source_specific_interval_definitions">Using source specific interval definitions</a></span></dt><dt><spanclass="section"><ahref="#_comparing_backups">Comparing backups</a></span></dt><dt><spanclass="section"><ahref="#_testing_for_host_reachabilty">Testing for host reachabilty</a></span></dt><dt><spanclass="section"><ahref="#_easy_check_for_errors">Easy check for errors</a></span></dt></dl></dd><dt><spanclass="section"><ahref="#_f_a_q">F.A.Q.</a></span></dt><dd><dl><dt><spanclass="section"><ahref="#_what_happens_if_one_backup_is_broken_or_empty">What happens if one backup is broken or empty?</a></span></dt><dt><spancl
using hardlinks and <codeclass="literal">rsync</code></p><divclass="section"title="Introduction"><divclass="titlepage"><div><div><h2class="title"style="clear: both"><aname="_introduction"></a>Introduction</h2></div></div></div><p><codeclass="literal">ccollect</code> is a backup utility written in the sh-scripting language.
bourne shell compatible (like <spanclass="emphasis"><em>dash</em></span>, <spanclass="emphasis"><em>ksh</em></span>, <spanclass="emphasis"><em>zsh</em></span>, <spanclass="emphasis"><em>bash</em></span>, …).</p><divclass="section"title="Supported and tested operating systems and architectures"><divclass="titlepage"><div><div><h3class="title"><aname="_supported_and_tested_operating_systems_and_architectures"></a>Supported and tested operating systems and architectures</h3></div></div></div><p><codeclass="literal">ccollect</code> was successfully tested on the following platforms:</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div><p>It <spanclass="strong"><strong>should</strong></span> run on any Unix that supports <codeclass="literal">rsync</code> and has a POSIX-compatible
bourne shell. If your platform is not listed above and you have it successfully
running, please drop me a mail.</p></div><divclass="section"title="Why you COULD only backup from remote hosts, not to them"><divclass="titlepage"><div><div><h3class="title"><aname="_why_you_could_only_backup_from_remote_hosts_not_to_them"></a>Why you COULD only backup from remote hosts, not to them</h3></div></div></div><p>While considering the design of ccollect, I thought about enabling
so you have the choice whether you want to use it or not.</p><divclass="section"title="Reason"><divclass="titlepage"><div><div><h4class="title"><aname="_reason"></a>Reason</h4></div></div></div><p>If you want to backup <spanclass="strong"><strong>TO</strong></span> a remote host, you have to loosen security on it.</p><p>Imagine the following situation: You backup your farm of webservers <spanclass="strong"><strong>TO</strong></span>
compromised.</p><p>Your backup server will be compromised, too.</p><p>And the attacker will have access to all data on the other webservers.</p></div><divclass="section"title="Doing it securely"><divclass="titlepage"><div><div><h4class="title"><aname="_doing_it_securely"></a>Doing it securely</h4></div></div></div><p>Think of it the other way round: The backup server (now behind a
All other backups are still secure.</p></div></div><divclass="section"title="Incompatibilities and changes"><divclass="titlepage"><div><div><h3class="title"><aname="_incompatibilities_and_changes"></a>Incompatibilities and changes</h3></div></div></div><divclass="section"title="Versions 0.9 and 1.0"><divclass="titlepage"><div><div><h4class="title"><aname="_versions_0_9_and_1_0"></a>Versions 0.9 and 1.0</h4></div></div></div><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
Added "Error: " prefix in _exit_err()
</li></ul></div></div><divclass="section"title="Versions 0.8 and 0.9"><divclass="titlepage"><div><div><h4class="title"><aname="_versions_0_8_and_0_9"></a>Versions 0.8 and 0.9</h4></div></div></div><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
Renamed script to ccollect (.sh is not needed)
</li><liclass="listitem">
Removed feature to backup to a host via ccollect, added new tool
(FIXME: insert name here) that takes care of this via tunnel
</li><liclass="listitem">
Perhaps creating subdirectory of source name (idea from Stefan Schl<68>rholz)
</li></ul></div></div><divclass="section"title="Versions 0.7 and 0.8"><divclass="titlepage"><div><div><h4class="title"><aname="_versions_0_7_and_0_8"></a>Versions 0.7 and 0.8</h4></div></div></div><divclass="itemizedlist"title="The argument order changed:"><pclass="title"><b>The argument order changed:</b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div><p>If you did not use arguments (most people do not), nothing will
change for you.</p><divclass="itemizedlist"title="Deletion of incomplete backups using the delete_incomplete option"><pclass="title"><b>Deletion of incomplete backups using the <spanclass="emphasis"><em>delete_incomplete</em></span> option</b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
Old: Only incomplete backups from the current interval have been removed
</li><liclass="listitem">
New: All incomplete backups are deleted
</li></ul></div><divclass="itemizedlist"title="Support for standard values"><pclass="title"><b>Support for standard values</b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
Old: no support
</li><liclass="listitem">
New: Options in $CCOLLECT_CONF/defaults are used as defaults (see below)
</li></ul></div></div><divclass="section"title="Versions 0.6 and 0.7"><divclass="titlepage"><div><div><h4class="title"><aname="_versions_0_6_and_0_7"></a>Versions 0.6 and 0.7</h4></div></div></div><divclass="itemizedlist"title="The format of destination changed:"><pclass="title"><b>The format of <codeclass="literal">destination</code> changed:</b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div><p>You can update your configuration using <codeclass="literal">tools/config-pre-0.7-to-0.7.sh</code>.</p><divclass="itemizedlist"title="Added remote_host"><pclass="title"><b>Added <spanclass="emphasis"><em>remote_host</em></span></b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div></div><divclass="section"title="Versions 0.5 and 0.6"><divclass="titlepage"><div><div><h4class="title"><aname="_versions_0_5_and_0_6"></a>Versions 0.5 and 0.6</h4></div></div></div><divclass="itemizedlist"title="The format of rsync_options changed:"><pclass="title"><b>The format of <codeclass="literal">rsync_options</code> changed:</b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div><p>You can update your configuration using <codeclass="literal">tools/config-pre-0.6-to-0.6.sh</code>.</p><divclass="itemizedlist"title="The name of the backup directories changed:"><pclass="title"><b>The name of the backup directories changed:</b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
XXXXX (- comes before digit).</p></div><divclass="section"title="Versions 0.4 and 0.5"><divclass="titlepage"><div><div><h4class="title"><aname="_versions_0_4_and_0_5"></a>Versions 0.4 and 0.5</h4></div></div></div><p>Not a real incompatibilty, but seems to fit in this section:</p><divclass="itemizedlist"title="0.5 does NOT require"><pclass="title"><b>0.5 does <spanclass="strong"><strong>NOT</strong></span> require</b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div><p>anymore!</p></div><divclass="section"title="Versions < 0.4 and 0.4"><divclass="titlepage"><div><div><h4class="title"><aname="_versions_lt_0_4_and_0_4"></a>Versions < 0.4 and 0.4</h4></div></div></div><p>Since <codeclass="literal">ccollect</code> 0.4 there are several incompatibilities with earlier
versions:</p><divclass="itemizedlist"title="List of incompatibilities"><pclass="title"><b>List of incompatibilities</b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
subdirectory:</p><preclass="screen">[10:05] hydrogenium:ccollect-0.4# ./tools/config-pre-0.4-to-0.4.sh /etc/ccollect</pre></div></div></div><divclass="section"title="Quick start"><divclass="titlepage"><div><div><h2class="title"style="clear: both"><aname="_quick_start"></a>Quick start</h2></div></div></div><p>For those who do not want to read the whole long document:</p><preclass="screen"># get latest ccollect tarball from http://www.nico.schottelius.org/software/ccollect/
execution of some backups.</p></div><divclass="section"title="Requirements"><divclass="titlepage"><div><div><h2class="title"style="clear: both"><aname="_requirements"></a>Requirements</h2></div></div></div><divclass="section"title="Installing ccollect"><divclass="titlepage"><div><div><h3class="title"><aname="_installing_ccollect"></a>Installing ccollect</h3></div></div></div><p>For the installation you need at least</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
the latest ccollect package (<aclass="ulink"href="http://www.nico.schottelius.org/software/ccollect/"target="_top">http://www.nico.schottelius.org/software/ccollect/</a>)
</li></ul></div></div><divclass="section"title="Using ccollect"><divclass="titlepage"><div><div><h3class="title"><aname="_using_ccollect"></a>Using ccollect</h3></div></div></div><divclass="itemizedlist"title="Running ccollect requires the following tools to be installed:"><pclass="title"><b>Running ccollect requires the following tools to be installed:</b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div></div></div><divclass="section"title="Installing"><divclass="titlepage"><div><div><h2class="title"style="clear: both"><aname="_installing"></a>Installing</h2></div></div></div><p>Either type <spanclass="emphasis"><em>make install</em></span> or simply copy it to a directory in your
$PATH and execute <spanclass="emphasis"><em>chmod <spanclass="strong"><strong>0755</strong></span> /path/to/ccollect.sh</em></span>. If you like
to use the new management scripts (available since 0.6), copy the
following scripts to a directory in $PATH:</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
<spanclass="emphasis"><em>./tools/report_success.sh</em></span>.</p></div><divclass="section"title="Configuring"><divclass="titlepage"><div><div><h2class="title"style="clear: both"><aname="_configuring"></a>Configuring</h2></div></div></div><p>For configuration aid have a look at the above mentioned tools, which can assist
you quite well. When you are successfully using <codeclass="literal">ccollect</code>, I would be happy if
you add a link to your website, stating "I backup with ccollect", which points
to the ccollect homepage. So more people now about ccollect, use it and
improve it. You can also report success using <codeclass="literal">tools/report_success.sh</code>.</p><divclass="section"title="Runtime options"><divclass="titlepage"><div><div><h3class="title"><aname="_runtime_options"></a>Runtime options</h3></div></div></div><p><codeclass="literal">ccollect</code> looks for its configuration in <spanclass="emphasis"><em>/etc/ccollect</em></span> or, if set, in
the directory specified by the variable <spanclass="emphasis"><em>$CCOLLECT_CONF</em></span>:</p><preclass="screen"># sh-compatible (dash, zsh, mksh, ksh, bash, ...)
$ CCOLLECT_CONF=/your/config/dir ccollect.sh ...
# csh
$ ( setenv CCOLLECT_CONF /your/config/dir ; ccollect.sh ... )</pre><p>When you start <codeclass="literal">ccollect</code>, you have to specify in which interval
to backup (daily, weekly, yearly; you can specify the names yourself, see below)
and which sources to backup (or -a to backup all sources).</p><p>The interval specifies how many backups are kept.</p><p>There are also some self-explanatory parameters you can pass to ccollect,
simply use <codeclass="literal">ccollect.sh --help</code> for info.</p></div><divclass="section"title="General configuration"><divclass="titlepage"><div><div><h3class="title"><aname="_general_configuration"></a>General configuration</h3></div></div></div><p>The general configuration can be found in $CCOLLECT_CONF/defaults or
configuration.</p><p>All configuration entries are plain-text files
(use UTF-8 for non-ascii characters).</p><divclass="section"title="Interval definition"><divclass="titlepage"><div><div><h4class="title"><aname="_interval_definition"></a>Interval definition</h4></div></div></div><p>The interval definition can be found in
the same as the name of the interval: <codeclass="literal">intervals/'<interval name>'</code>.</p><p>The content of this file should be a single line containing a number.
This number defines how many versions of this interval are kept.</p><p>Example:</p><preclass="screen"> [10:23] zaphodbeeblebrox:ccollect-0.2% ls -l conf/defaults/intervals/
4</pre><p>This means to keep 28 daily backups, 12 monthly backups and 4 weekly.</p></div><divclass="section"title="General pre- and post-execution"><divclass="titlepage"><div><div><h4class="title"><aname="_general_pre_and_post_execution"></a>General pre- and post-execution</h4></div></div></div><p>If you add <spanclass="emphasis"><em>$CCOLLECT_CONF/defaults/<codeclass="literal">pre_exec</code></em></span> or
<spanclass="emphasis"><em>/etc/ccollect/defaults/<codeclass="literal">pre_exec</code></em></span> (same with <codeclass="literal">post_exec</code>), <codeclass="literal">ccollect</code>
will start <codeclass="literal">pre_exec</code> before the whole backup process and
<codeclass="literal">post_exec</code> after backup of all sources is done.</p><p>If <codeclass="literal">pre_exec</code> exits with a non-zero return code, the whole backup
process will be aborted.</p><p>The <codeclass="literal">pre_exec</code> and <codeclass="literal">post_exec</code> script can access the following exported variables:</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
<spanclass="emphasis"><em>INTERVAL</em></span>: the interval selected (<codeclass="literal">daily</code>)
</li><liclass="listitem">
<spanclass="emphasis"><em>no_sources</em></span>: number of sources to backup (<codeclass="literal">2</code>)
</li><liclass="listitem">
<spanclass="emphasis"><em>source_$no</em></span>: name of the source, <spanclass="emphasis"><em>$no</em></span> starts at 0 (<codeclass="literal">$source_0</code>)
</li></ul></div><p>The following example describes how to report free disk space in
[13:01] hydrogenium:~# ln -s /etc/ccollect/defaults/pre_exec /etc/ccollect/defaults/post_exec</pre></div></div><divclass="section"title="Source configuration"><divclass="titlepage"><div><div><h3class="title"><aname="_source_configuration"></a>Source configuration</h3></div></div></div><p>Each source configuration exists in <spanclass="emphasis"><em>$CCOLLECT_CONF/sources/$name</em></span> or
<spanclass="emphasis"><em>/etc/ccollect/sources/$name</em></span>.</p><p>The name you choose for the subdirectory describes the source.</p><p>Each source contains at least the following files:</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div><p>Additionally a source may have the following files:</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
<codeclass="literal">pre_exec</code> program to execute before backing up <spanclass="strong"><strong>this</strong></span> source
</li><liclass="listitem">
<codeclass="literal">post_exec</code> program to execute after backing up <spanclass="strong"><strong>this</strong></span> source
<codeclass="literal">very_verbose</code> be very verbose (<codeclass="literal">mkdir -v</code>, <codeclass="literal">rm -v</code> and <codeclass="literal">rsync -vv</code>)
/home/nico/vpn</pre><divclass="section"title="Default options"><divclass="titlepage"><div><div><h4class="title"><aname="_default_options"></a>Default options</h4></div></div></div><p>If you add <spanclass="emphasis"><em>$CCOLLECT_CONF/defaults/<codeclass="literal">option_name</code></em></span>, the value will
be used in abscence of the option in a source. If you want to prevent
the default value to be used in a source, you can create the file
<spanclass="emphasis"><em>$CCOLLECT_CONF/sources/$name/<codeclass="literal">no_option_name</code></em></span> (i.e. prefix it with
[9:04] ikn2:ccollect% touch conf/sources/local/no_verbose</pre><p>This enables the verbose option for all sources, but disables it for the
source <spanclass="emphasis"><em>local</em></span>.</p><p>If an option is specified in the defaults folder and in the source,
the source specific version overrides the default one:</p><p>Example:</p><preclass="screen"> [9:05] ikn2:ccollect% echo "backup-host" > conf/defaults/remote_host
[9:05] ikn2:ccollect% echo "different-host" > conf/sources/local/remote_host</pre><p>You can use all source options as defaults, with the exception of</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
<codeclass="literal">source</code>
</li><liclass="listitem">
<codeclass="literal">destination</code>
</li><liclass="listitem">
<codeclass="literal">pre_exec</code>
</li><liclass="listitem">
<codeclass="literal">post_exec</code>
</li></ul></div></div><divclass="section"title='Detailed description of "source"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_source"></a>Detailed description of "source"</h4></div></div></div><p><codeclass="literal">source</code> describes a <codeclass="literal">rsync</code> compatible source (one line only).</p><p>For instance <spanclass="emphasis"><em>backup_user@foreign_host:/home/server/video</em></span>.
of <codeclass="literal">rsync</code>(1).</p></div><divclass="section"title='Detailed description of "destination"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_destination"></a>Detailed description of "destination"</h4></div></div></div><p><codeclass="literal">destination</code> must be a text file containing the destination directory.
<codeclass="literal">destination</code><spanclass="strong"><strong>USED</strong></span> to be a link to the destination directory in
earlier versions, so do not be confused if you see such examples.</p><p>Example:</p><preclass="screen"> [11:36] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/destination
/home/nico/backupdir</pre></div><divclass="section"title='Detailed description of "remote_host"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_remote_host"></a>Detailed description of "remote_host"</h4></div></div></div><p><codeclass="literal">remote_host</code> must be a text file containing the destination host.
If this file is existing, you are backing up your data <spanclass="strong"><strong>TO</strong></span> this host
and <spanclass="strong"><strong>not</strong></span> to you local host.</p><p><spanclass="strong"><strong>Warning</strong></span>: You need to have <codeclass="literal">ssh</code> access to the remote host. <codeclass="literal">rsync</code> and
<codeclass="literal">ccollect</code> will connect to that host via <codeclass="literal">ssh</code>. <codeclass="literal">ccollect</code> needs the shell
access, because it needs to find out how many backups exist on the remote
host and to be able to delete them.</p><p>Example:</p><preclass="screen"> [10:17] denkbrett:ccollect-0.7.0% cat conf/sources/remote1/remote_host
home.schottelius.org</pre><p>It may contain all the ssh-specific values like <spanclass="emphasis"><em>myuser@yourhost.ch</em></span>.</p></div><divclass="section"title='Detailed description of "verbose"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_verbose"></a>Detailed description of "verbose"</h4></div></div></div><p><codeclass="literal">verbose</code> tells <codeclass="literal">ccollect</code> that the log should contain verbose messages.</p><p>If this file exists in the source specification <spanclass="strong"><strong>-v</strong></span> will be passed to <codeclass="literal">rsync</code>.</p><p>Example:</p><preclass="screen"> [11:35] zaphodbeeblebrox:ccollect-0.2% touch conf/sources/testsource1/verbose</pre></div><divclass="section"title='Detailed description of "very_verbose"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_very_verbose"></a>Detailed description of "very_verbose"</h4></div></div></div><p><codeclass="literal">very_verbose</code> tells <codeclass="literal">ccollect</code> that it should log very verbosely.</p><p>If this file exists in the source specification <spanclass="strong"><strong>-v</strong></span> will be passed to
<codeclass="literal">rsync</code>, <codeclass="literal">rm</code> and <codeclass="literal">mkdir</code>.</p><p>Example:</p><preclass="screen"> [23:67] nohost:~% touch conf/sources/testsource1/very_verbose</pre></div><divclass="section"title='Detailed description of "summary"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_summary"></a>Detailed description of "summary"</h4></div></div></div><p>If you create the file <codeclass="literal">summary</code> in the source definition,
<codeclass="literal">ccollect</code> will present you a nice summary at the end.</p><preclass="screen">backup:~# touch /etc/ccollect/sources/root/summary
backup:~# ccollect.sh werktags root
==> ccollect.sh: Beginning backup using interval werktags <==
[root] Beginning to backup this source ...
[root] Currently 3 backup(s) exist, total keeping 50 backup(s).
[root] Beginning to backup, this may take some time...
[root] Hard linking...
[root] Transferring files...
[root]
[root] Number of files: 84183
[root] Number of files transferred: 32
[root] Total file size: 26234080536 bytes
[root] Total transferred file size: 9988252 bytes
[root] Literal data: 9988252 bytes
[root] Matched data: 0 bytes
[root] File list size: 3016771
[root] File list generation time: 1.786 seconds
[root] File list transfer time: 0.000 seconds
[root] Total bytes sent: 13009119
[root] Total bytes received: 2152
[root]
[root] sent 13009119 bytes received 2152 bytes 2891393.56 bytes/sec
[root] total size is 26234080536 speedup is 2016.26
[root] Successfully finished backup.
==> Finished ccollect.sh <==</pre><p>You could also combine it with <codeclass="literal">verbose</code> or <codeclass="literal">very_verbose</code>, but these
already print some statistics (though not all / the same as presented by
<codeclass="literal">summary</code>).</p></div><divclass="section"title='Detailed description of "exclude"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_exclude"></a>Detailed description of "exclude"</h4></div></div></div><p><codeclass="literal">exclude</code> specifies a list of paths to exclude. The entries are seperated by a newline (\n).</p><p>Example:</p><preclass="screen"> [11:35] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
something with spaces is not a problem</pre></div><divclass="section"title='Detailed description of "intervals/"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_intervals"></a>Detailed description of "intervals/"</h4></div></div></div><p>When you create the subdirectory <codeclass="literal">intervals/</code> in your source configuration
directory, you can specify individiual intervals for this specific source.
Each file in this directory describes an interval.</p><p>Example:</p><preclass="screen"> [11:37] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervals/
20</pre></div><divclass="section"title='Detailled description of "rsync_options"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailled_description_of_rsync_options"></a>Detailled description of "rsync_options"</h4></div></div></div><p>When you create the file <codeclass="literal">rsync_options</code> in your source configuration,
to use for automatic backup over the rsync-protocol.</p><p>Example:</p><preclass="screen"> [23:42] hydrogenium:ccollect-0.2% cat conf/sources/test_rsync/rsync_options
--password-file=/home/user/backup/protected_password_file</pre></div><divclass="section"title='Detailled description of "pre_exec" and "post_exec"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailled_description_of_pre_exec_and_post_exec"></a>Detailled description of "pre_exec" and "post_exec"</h4></div></div></div><p>When you create <codeclass="literal">pre_exec</code> and / or <codeclass="literal">post_exec</code> in your source
backups, see above for general configuration.</p><p>If <codeclass="literal">pre_exec</code> exits with a non-zero return code, the backup
process of <codeclass="literal">this source</code> will be aborted (i.e. backup skipped).</p><p>The <codeclass="literal">post_exec</code> script can access the following exported variables from
<spanclass="emphasis"><em>name</em></span>: name of the source that is being backed up
</li><liclass="listitem">
<spanclass="emphasis"><em>destination_name</em></span>: contains the base directory name (<codeclass="literal">daily.20091031-1013.24496</code>)
</li><liclass="listitem">
<spanclass="emphasis"><em>destination_dir</em></span>: full path (<codeclass="literal">/tmp/ccollect/daily.20091031-1013.24496</code>)
</li><liclass="listitem">
<spanclass="emphasis"><em>destination_full</em></span>: like <spanclass="emphasis"><em>destination_dir</em></span>, but prepended with the remote_host, if set (<codeclass="literal">host:/tmp/ccollect/daily.20091031-1013.24496</code> or <codeclass="literal">/tmp/ccollect/daily.20091031-1013.24496</code>)
df -h</pre></div><divclass="section"title='Detailed description of "delete_incomplete"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_delete_incomplete"></a>Detailed description of "delete_incomplete"</h4></div></div></div><p>If you create the file <codeclass="literal">delete_incomplete</code> in a source specification directory,
the user.</p></div><divclass="section"title='Detailed description of "rsync_failure_codes"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_rsync_failure_codes"></a>Detailed description of "rsync_failure_codes"</h4></div></div></div><p>If you have the file <codeclass="literal">rsync_failure_codes</code> in your source configuration
the next ccollect run on the same interval.</p></div><divclass="section"title='Detailed description of "mtime"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_mtime"></a>Detailed description of "mtime"</h4></div></div></div><p>By default, ccollect.sh chooses the most recent backup directory for cloning or
the oldest for deletion based on the directory’s last change time (ctime).
this version of ccollect, the ctime and mtime of your backups will normally
be the same and this option has no effect. However, if you, for example, move
your backups to another hard disk using cp -a or rsync -a, you should use this
option because the ctimes are not preserved during such operations.</p><p>If you have any backups in your repository made with ccollect version 0.7.1 or
earlier, do not use this option.</p></div><divclass="section"title='Detailed description of "quiet_if_down"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_quiet_if_down"></a>Detailed description of "quiet_if_down"</h4></div></div></div><p>By default, ccollect.sh emits a series of error messages if a source is not
like notebook PCs, that are often disconnected.</p></div></div></div><divclass="section"title="Hints"><divclass="titlepage"><div><div><h2class="title"style="clear: both"><aname="_hints"></a>Hints</h2></div></div></div><divclass="section"title="Smart logging"><divclass="titlepage"><div><div><h3class="title"><aname="_smart_logging"></a>Smart logging</h3></div></div></div><p>Since ccollect-0.6.1 you can use the ccollect-logwrapper.sh(1) for logging.
ccollect-logwrapper.sh (11722): Finished.</pre></div><divclass="section"title="Using a different ssh port"><divclass="titlepage"><div><div><h3class="title"><aname="_using_a_different_ssh_port"></a>Using a different ssh port</h3></div></div></div><p>Mostly easy is to use your ~/.ssh/config file:</p><preclass="screen">host mx2.schottelius.org
HostKeyAlias bruehe.schottelius.org</pre></div><divclass="section"title="Using source names or interval in pre_/post_exec scripts"><divclass="titlepage"><div><div><h3class="title"><aname="_using_source_names_or_interval_in_pre__post_exec_scripts"></a>Using source names or interval in pre_/post_exec scripts</h3></div></div></div><p>The pre-/post_exec scripts can access some internal variables from <codeclass="literal">ccollect</code>:</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div><p>Only available for <codeclass="literal">post_exec</code>:</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
<spanclass="emphasis"><em>remote_host</em></span>: name of host we backup to (empty if unused)
</li></ul></div></div><divclass="section"title="Using rsync protocol without ssh"><divclass="titlepage"><div><div><h3class="title"><aname="_using_rsync_protocol_without_ssh"></a>Using rsync protocol without ssh</h3></div></div></div><p>When you have a computer with little computing power, it may be useful to use
rsync without ssh, directly using the rsync protocol
(specify <spanclass="emphasis"><em>user@host::share</em></span> in <codeclass="literal">source</code>). You may wish to use
<codeclass="literal">rsync_options</code> to specify a password file to use for automatic backup.</p><p>Example:</p><preclass="screen">backup:~# cat /etc/ccollect/sources/sample.backup.host.org/source
this_is_the_rsync_password</pre><p>This hint was reported by Daniel Aubry.</p></div><divclass="section"title="Not excluding top-level directories"><divclass="titlepage"><div><div><h3class="title"><aname="_not_excluding_top_level_directories"></a>Not excluding top-level directories</h3></div></div></div><p>When you exclude "/proc" or "/mnt" from your backup, you may run into
trouble when you restore your backup. When you use "/proc/*" or "/mnt/\*"
instead, <codeclass="literal">ccollect</code> will backup empty directories.</p><divclass="note"title="Note"style="margin-left: 0.5in; margin-right: 0.5in;"><h3class="title">Note</h3><p>When those directories contain hidden files
they will still be transferred!</p></div><p>This hint was reported by Marcus Wagner.</p></div><divclass="section"title="Re-using already created rsync-backups"><divclass="titlepage"><div><div><h3class="title"><aname="_re_using_already_created_rsync_backups"></a>Re-using already created rsync-backups</h3></div></div></div><p>If you used <codeclass="literal">rsync</code> directly before you use <codeclass="literal">ccollect</code>, you can
daily.0</pre><p>Now you can use /home/backup/web1 as the <codeclass="literal">destination</code> for the backup.</p><divclass="note"title="Note"style="margin-left: 0.5in; margin-right: 0.5in;"><h3class="title">Note</h3><p>It does not matter anymore how you name your directory, as <codeclass="literal">ccollect</code> uses
the -c option from <codeclass="literal">ls</code> to find out which directory to clone from.</p></div><divclass="note"title="Note"style="margin-left: 0.5in; margin-right: 0.5in;"><h3class="title">Note</h3><p>Older versions (pre 0.6, iirc) had a problem, if you named the first backup
something like "daily.initial". It was needed to use the "<spanclass="strong"><strong>0</strong></span>" (or some
number that is lower than the current year) as extension. <codeclass="literal">ccollect</code>
used <codeclass="literal">sort</code> to find the latest backup. <codeclass="literal">ccollect</code> itself uses
<spanclass="emphasis"><em>interval.YEARMONTHDAY-HOURMINUTE.PID</em></span>. This notation was <spanclass="strong"><strong>always</strong></span> before
"daily.initial", as numbers are earlier in the list
which is produced by <codeclass="literal">sort</code>. So, if you had a directory named "daily.initial",
<codeclass="literal">ccollect</code> always diffed against this backup and transfered and deleted
files which where deleted in previous backups. This means you simply
wasted resources, but your backup had beer complete anyway.</p></div></div><divclass="section"title="Using pre_/post_exec"><divclass="titlepage"><div><div><h3class="title"><aname="_using_pre__post_exec"></a>Using pre_/post_exec</h3></div></div></div><p>Your pre_/post_exec script does not need to be a script, you can also
</li></ul></div><p>The only requirement is that it is executable.</p></div><divclass="section"title="Using source specific interval definitions"><divclass="titlepage"><div><div><h3class="title"><aname="_using_source_specific_interval_definitions"></a>Using source specific interval definitions</h3></div></div></div><p>When you are backing up multiple hosts via cron each night, it may be
a problem that host "big_server" may only have 4 daily backups, because
otherwise its backup device will be full. But for all other hosts
you want to keep 20 daily backups. In this case you would create
<codeclass="literal">/etc/ccollect/default/intervals/daily</code> containing "20" and
<codeclass="literal">/etc/ccollect/sources/big_server/intervals/daily</code> containing "4".</p><p>Source specific intervals always overwrite the default values.
If you have to specify it individually for every host, because
of different requirements, you can even omit creating
<codeclass="literal">/etc/ccollect/default/intervals/daily</code>.</p></div><divclass="section"title="Comparing backups"><divclass="titlepage"><div><div><h3class="title"><aname="_comparing_backups"></a>Comparing backups</h3></div></div></div><p>If you want to see what changed between two backups, you can use
<codeclass="literal">rsync</code> directly:</p><preclass="screen">[12:00] u0255:ddba034.netstream.ch# rsync -n -a --delete --stats --progress daily.20080324-0313.17841/ daily.20080325-0313.31148/</pre><p>This results in a listing of changes. Because we pass -n to rsync no transfer
is made (i.e. report only mode).</p><p>This hint was reported by Daniel Aubry.</p></div><divclass="section"title="Testing for host reachabilty"><divclass="titlepage"><div><div><h3class="title"><aname="_testing_for_host_reachabilty"></a>Testing for host reachabilty</h3></div></div></div><p>If you want to test whether the host you try to backup is reachable, you can use
# ping -c1 -q `cat "/etc/ccollect/sources/$name/source" | cut -d"@" -f2 | cut -d":" -f1`</pre><p>This prevents the deletion of old backups, if the host is not reachable.</p><p>This hint was reported by Daniel Aubry.</p></div><divclass="section"title="Easy check for errors"><divclass="titlepage"><div><div><h3class="title"><aname="_easy_check_for_errors"></a>Easy check for errors</h3></div></div></div><p>If you want to see whether there have been any errors while doing the backup,
you can run <codeclass="literal">ccollect</code> together with <codeclass="literal">ccollect_analyse_logs.sh</code>:</p><preclass="screen">$ ccollect | ccollect_analyse_logs.sh e</pre></div></div><divclass="section"title="F.A.Q."><divclass="titlepage"><div><div><h2class="title"style="clear: both"><aname="_f_a_q"></a>F.A.Q.</h2></div></div></div><divclass="section"title="What happens if one backup is broken or empty?"><divclass="titlepage"><div><div><h3class="title"><aname="_what_happens_if_one_backup_is_broken_or_empty"></a>What happens if one backup is broken or empty?</h3></div></div></div><p>Let us assume that one backup failed (connection broke or the source
hard disk had some failures). Therefore we’ve got one incomplete backup in our history.</p><p><codeclass="literal">ccollect</code> will transfer the missing files the next time you use it.
This leads to</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
backup instead</p></div><divclass="section"title="When backing up from localhost the destination is also included. Is this a bug?"><divclass="titlepage"><div><div><h3class="title"><aname="_when_backing_up_from_localhost_the_destination_is_also_included_is_this_a_bug"></a>When backing up from localhost the destination is also included. Is this a bug?</h3></div></div></div><p>No. <codeclass="literal">ccollect</code> passes your source definition directly to <codeclass="literal">rsync</code>. It
to <spanclass="emphasis"><em>source/exclude</em></span>. (Daniel Aubry reported this problem)</p></div><divclass="section"title='Why does ccollect say "Permission denied" with my pre-/postexec script?'><divclass="titlepage"><div><div><h3class="title"><aname="_why_does_ccollect_say_permission_denied_with_my_pre_postexec_script"></a>Why does ccollect say "Permission denied" with my pre-/postexec script?</h3></div></div></div><p>The most common error is that you have not given your script the correct
permissions. Try <codeclass="literal">chmod 0755 /etc/ccollect/sources/'yoursource'/*_exec`</code>.</p></div><divclass="section"title="Why does the backup job fail when part of the source is a link?"><divclass="titlepage"><div><div><h3class="title"><aname="_why_does_the_backup_job_fail_when_part_of_the_source_is_a_link"></a>Why does the backup job fail when part of the source is a link?</h3></div></div></div><p>When a part of your path you specified in the source is a
(symbolic, hard links are not possible for directories) link,
the backup <spanclass="strong"><strong>must</strong></span> fail.</p><p>First of all, let us have a look at how it looks like:</p><preclass="screen">==> ccollect 0.4: Beginning backup using interval taeglich <==
[testsource] Sa Apr 29 00:01:55 CEST 2006 Beginning to backup
[testsource] Currently 0 backup(s) exist(s), total keeping 10 backup(s).
[testsource] Beginning to backup, this may take some time...
[testsource] Sa Apr 29 00:01:55 CEST 2006 Transferring files...
[testsource] rsync: recv_generator: mkdir "/etc/ccollect/sources/testsource/destination/taeglich.2006-04-29-0001.3874/home/user/nico/projekte/ccollect" failed: No such file or directory (2)
[testsource] rsync: stat "/etc/ccollect/sources/testsource/destination/taeglich.2006-04-29-0001.3874/home/user/nico/projekte/ccollect" failed: No such file or directory (2)
[...]</pre><p>So what is the problem? It is very obvious when you look deeper into it:</p><preclass="screen">% cat /etc/ccollect/sources/testsource/source
% ls -l /etc/ccollect/sources/testsource/destination/taeglich.2006-04-29-0001.3874/home/user/nico
lrwxrwxrwx 1 nico nico 29 2006-04-29 00:01 projekte -> oeffentlich/computer/projekte</pre><p><codeclass="literal">rsync</code> creates the directory structure before it creates the symbolic link.
This link now links to something not reachable (dead link). It is
impossible to create subdirectories under the broken link.</p><p>In conclusion you cannot use paths with a linked part.</p><p>However, you can backup directories containing symbolic links
(in this case you could backup /home/user/nico, which contains
/home/user/nico/projekte and oeffentlich/computer/projekte).</p></div><divclass="section"title="How can I prevent missing the right time to enter my password?"><divclass="titlepage"><div><div><h3class="title"><aname="_how_can_i_prevent_missing_the_right_time_to_enter_my_password"></a>How can I prevent missing the right time to enter my password?</h3></div></div></div><p>As <codeclass="literal">ccollect</code> first deletes the old backups, it may take some time
until <codeclass="literal">rsync</code> requests the password for the <codeclass="literal">ssh</code> session from you.</p><p>The easiest way not to miss that point is running <codeclass="literal">ccollect</code> in <codeclass="literal">screen</code>,
which has the ability to monitor the output for activity. So as soon as
your screen beeps, after <codeclass="literal">ccollect</code> began to remove the last directory,
you can enter your password (have a look at screen(1), especially "C-a M"
and "C-a _", for more information).</p></div><divclass="section"title="Backup fails, if autofs is running, but sources not reachable"><divclass="titlepage"><div><div><h3class="title"><aname="_backup_fails_if_autofs_is_running_but_sources_not_reachable"></a>Backup fails, if autofs is running, but sources not reachable</h3></div></div></div><p>If you are trying to backup a system containing paths that are managed
by autofs, you may run into this error:</p><preclass="screen">2009-12-01-23:14:15: ccollect 0.8.1: Beginning backup using interval monatlich
[ikn] directory has vanished: "/home/users/nico/privat/firmen/ethz/autofs/scratch"
[ikn] directory has vanished: "/home/users/nico/privat/firmen/ethz/autofs/sgscratch"
[ikn] directory has vanished: "/home/users/nico/privat/firmen/ethz/autofs/supp"
[ikn] directory has vanished: "/home/users/nico/privat/firmen/ethz/autofs/sysadmin"
[ikn] rsync warning: some files vanished before they could be transferred (code 24) at main.c(1057) [sender=3.0.6]
[ikn] 2009-12-01-23:44:23: Source / is not readable. Skipping.</pre><p>Thus, if you are unsure whether autofs paths can be mounted during backup,
stop autofs in pre_exec and reenable it in post_exec.</p></div></div><divclass="section"title="Examples"><divclass="titlepage"><div><div><h2class="title"style="clear: both"><aname="_examples"></a>Examples</h2></div></div></div><divclass="section"title="A backup host configuration from scratch"><divclass="titlepage"><div><div><h3class="title"><aname="_a_backup_host_configuration_from_scratch"></a>A backup host configuration from scratch</h3></div></div></div><preclass="screen">srwali01:~# mkdir /etc/ccollect
/o> ccollect.sh: Beginning backup using interval taeglich
/=> Beginning to backup "local-root" ...
|-> 0 backup(s) already exist, keeping 28 backup(s).</pre><p>After that, I added some more sources:</p><preclass="screen">srwali01:~# cd /etc/ccollect/sources
srwali01:/etc/ccollect/sources/srwali03# mkdir /mnt/hdbackup/srwali03</pre></div><divclass="section"title="Using hard-links requires less disk space"><divclass="titlepage"><div><div><h3class="title"><aname="_using_hard_links_requires_less_disk_space"></a>Using hard-links requires less disk space</h3></div></div></div><preclass="screen"># du (coreutils) 5.2.1
147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:45.588</pre><p>The backup of our main fileserver:</p><preclass="screen">backup:~# df -h /home/backup/srsyg01/
the hardlinks allocate.</p></div><divclass="section"title="A collection of backups on the backup server"><divclass="titlepage"><div><div><h3class="title"><aname="_a_collection_of_backups_on_the_backup_server"></a>A collection of backups on the backup server</h3></div></div></div><p>All the data of my important hosts is backuped to eiche into
woechentlich.2006-02-12-11:51.2514 woechentlich.2006-03-16-23:00.4218</pre><p>And this incremental backup and the archive are copied to an external
usb harddisk (attention: you <spanclass="strong"><strong>should</strong></span> really use -H to backup the backup):</p><preclass="screen">[9:23] eiche:backup# df -h
