<html><head><metahttp-equiv="Content-Type"content="text/html; charset=ISO-8859-1"><title>ccollect - Installing, Configuring and Using</title><metaname="generator"content="DocBook XSL Stylesheets V1.75.1"></head><bodybgcolor="white"text="black"link="#0000FF"vlink="#840084"alink="#0000FF"><divlang="en"class="article"title="ccollect - Installing, Configuring and Using"><divclass="titlepage"><div><div><h2class="title"><aname="id569747"></a>ccollect - Installing, Configuring and Using</h2></div><div><divclass="author"><h3class="author"><spanclass="firstname">Nico</span><spanclass="surname">Schottelius</span></h3><codeclass="email"><<aclass="email"href="mailto:nico-ccollect__@__schottelius.org">nico-ccollect__@__schottelius.org</a>></code></div></div><div><divclass="revhistory"><tableborder="1"width="100%"summary="Revision history"><tr><thalign="left"valign="top"colspan="3"><b>Revision History</b></th></tr><tr><tdalign="left">Revision 0.7.1</td><tdalign="left">for ccollect 0.7.1, Initial Version from 2006-01-13</td><tdalign="left">NS</td></tr></table></div></div></div><hr></div><divclass="toc"><p><b>Table of Contents</b></p><dl><dt><spanclass="section"><ahref="#_introduction">Introduction</a></span></dt><dd><dl><dt><spanclass="section"><ahref="#_supported_and_tested_operating_systems_and_architectures">Supported and tested operating systems and architectures</a></span></dt><dt><spanclass="section"><ahref="#_why_you_could_only_backup_from_remote_hosts_not_to_them">Why you COULD only backup from remote hosts, not to them</a></span></dt><dt><spanclass="section"><ahref="#_incompatibilities">Incompatibilities</a></span></dt></dl></dd><dt><spanclass="section"><ahref="#_quick_start">Quick start</a></span></dt><dt><spanclass="section"><ahref="#_requirements">Requirements</a></span></dt><dd><dl><dt><spanclass="section"><ahref="#_installing_ccollect">Installing ccollect</a></span></dt><dt><spanclass="section"><ahref="#_using_ccollect">Using ccollect</a></span></dt></dl></dd><dt><spanclass="section"><ahref="#_installing">Installing</a></span></dt><dt><spanclass="section"><ahref="#_configuring">Configuring</a></span></dt><dd><dl><dt><spanclass="section"><ahref="#_runtime_options">Runtime options</a></span></dt><dt><spanclass="section"><ahref="#_general_configuration">General configuration</a></span></dt><dt><spanclass="section"><ahref="#_source_configuration">Source configuration</a></span></dt></dl></dd><dt><spanclass="section"><ahref="#_hints">Hints</a></span></dt><dd><dl><dt><spanclass="section"><ahref="#_smart_logging">Smart logging</a></span></dt><dt><spanclass="section"><ahref="#_using_a_different_ssh_port">Using a different ssh port</a></span></dt><dt><spanclass="section"><ahref="#_using_source_names_or_interval_in_pre_post_exec_scripts">Using source names or interval in pre_/post_exec scripts</a></span></dt><dt><spanclass="section"><ahref="#_using_rsync_protocol_without_ssh">Using rsync protocol without ssh</a></span></dt><dt><spanclass="section"><ahref="#_not_excluding_top_level_directories">Not excluding top-level directories</a></span></dt><dt><spanclass="section"><ahref="#_re_using_already_created_rsync_backups">Re-using already created rsync-backups</a></span></dt><dt><spanclass="section"><ahref="#_using_pre_post_exec">Using pre_/post_exec</a></span></dt><dt><spanclass="section"><ahref="#_using_source_specific_interval_definitions">Using source specific interval definitions</a></span></dt><dt><spanclass="section"><ahref="#_comparing_backups">Comparing backups</a></span></dt><dt><spanclass="section"><ahref="#_testing_for_host_reachabilty">Testing for host reachabilty</a></span></dt><dt><spanclass="section"><ahref="#_easy_check_for_errors">Easy check for errors</a></span></dt></dl></dd><dt><spanclass="section"><ahref="#_f_a_q">F.A.Q.</a></span></dt><dd><dl><dt><spanclass="section"><ahref="#_what_happens_if_one_backup_is_broken_or_empty">What happens if one backup is broken or empty?</a></span></dt><dt><spanclass="section"><ahref="#_
using hardlinks and <codeclass="literal">rsync</code></p><divclass="section"title="Introduction"><divclass="titlepage"><div><div><h2class="title"style="clear: both"><aname="_introduction"></a>Introduction</h2></div></div></div><p><codeclass="literal">ccollect</code> is a backup utility written in the sh-scripting language.
bourne shell compatible (like <spanclass="emphasis"><em>dash</em></span>, <spanclass="emphasis"><em>ksh</em></span>, <spanclass="emphasis"><em>zsh</em></span>, <spanclass="emphasis"><em>bash</em></span>, …).</p><divclass="section"title="Supported and tested operating systems and architectures"><divclass="titlepage"><div><div><h3class="title"><aname="_supported_and_tested_operating_systems_and_architectures"></a>Supported and tested operating systems and architectures</h3></div></div></div><p><codeclass="literal">ccollect</code> was successfully tested on the following platforms:</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div><p>It <spanclass="strong"><strong>should</strong></span> run on any Unix that supports <codeclass="literal">rsync</code> and has a POSIX-compatible
bourne shell. If your platform is not listed above and you have it successfully
running, please drop me a mail.</p></div><divclass="section"title="Why you COULD only backup from remote hosts, not to them"><divclass="titlepage"><div><div><h3class="title"><aname="_why_you_could_only_backup_from_remote_hosts_not_to_them"></a>Why you COULD only backup from remote hosts, not to them</h3></div></div></div><p>While considering the design of ccollect, I thought about enabling
so you have the choice whether you want to use it or not.</p><divclass="section"title="Reason"><divclass="titlepage"><div><div><h4class="title"><aname="_reason"></a>Reason</h4></div></div></div><p>If you want to backup <spanclass="strong"><strong>TO</strong></span> a remote host, you have to loosen security on it.</p><p>Imagine the following situation: You backup your farm of webservers <spanclass="strong"><strong>TO</strong></span>
compromised.</p><p>Your backup server will be compromised, too.</p><p>And the attacker will have access to all data on the other webservers.</p></div><divclass="section"title="Doing it securely"><divclass="titlepage"><div><div><h4class="title"><aname="_doing_it_securely"></a>Doing it securely</h4></div></div></div><p>Think of it the other way round: The backup server (now behind a
All other backups are still secure.</p></div></div><divclass="section"title="Incompatibilities"><divclass="titlepage"><div><div><h3class="title"><aname="_incompatibilities"></a>Incompatibilities</h3></div></div></div><divclass="section"title="Versions 0.6 and 0.7"><divclass="titlepage"><div><div><h4class="title"><aname="_versions_0_6_and_0_7"></a>Versions 0.6 and 0.7</h4></div></div></div><divclass="itemizedlist"title="The format of destination changed:"><pclass="title"><b>The format of <codeclass="literal">destination</code> changed:</b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div><p>You can update your configuration using <codeclass="literal">tools/config-pre-0.7-to-0.7.sh</code>.</p><divclass="itemizedlist"title="Added remote_host"><pclass="title"><b>Added <spanclass="emphasis"><em>remote_host</em></span></b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div></div><divclass="section"title="Versions 0.5 and 0.6"><divclass="titlepage"><div><div><h4class="title"><aname="_versions_0_5_and_0_6"></a>Versions 0.5 and 0.6</h4></div></div></div><divclass="itemizedlist"title="The format of rsync_options changed:"><pclass="title"><b>The format of <codeclass="literal">rsync_options</code> changed:</b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div><p>You can update your configuration using <codeclass="literal">tools/config-pre-0.6-to-0.6.sh</code>.</p><divclass="itemizedlist"title="The name of the backup directories changed:"><pclass="title"><b>The name of the backup directories changed:</b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
XXXXX (- comes before digit).</p></div><divclass="section"title="Versions 0.4 and 0.5"><divclass="titlepage"><div><div><h4class="title"><aname="_versions_0_4_and_0_5"></a>Versions 0.4 and 0.5</h4></div></div></div><p>Not a real incompatibilty, but seems to fit in this section:</p><divclass="itemizedlist"title="0.5 does NOT require"><pclass="title"><b>0.5 does <spanclass="strong"><strong>NOT</strong></span> require</b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div><p>anymore!</p></div><divclass="section"title="Versions < 0.4 and 0.4"><divclass="titlepage"><div><div><h4class="title"><aname="_versions_lt_0_4_and_0_4"></a>Versions < 0.4 and 0.4</h4></div></div></div><p>Since <codeclass="literal">ccollect</code> 0.4 there are several incompatibilities with earlier
versions:</p><divclass="itemizedlist"title="List of incompatibilities"><pclass="title"><b>List of incompatibilities</b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
subdirectory:</p><preclass="screen">[10:05] hydrogenium:ccollect-0.4# ./tools/config-pre-0.4-to-0.4.sh /etc/ccollect</pre></div></div></div><divclass="section"title="Quick start"><divclass="titlepage"><div><div><h2class="title"style="clear: both"><aname="_quick_start"></a>Quick start</h2></div></div></div><p>For those who do not want to read the whole long document:</p><preclass="screen"># get latest ccollect tarball from http://www.nico.schottelius.org/software/ccollect/
execution of some backups.</p></div><divclass="section"title="Requirements"><divclass="titlepage"><div><div><h2class="title"style="clear: both"><aname="_requirements"></a>Requirements</h2></div></div></div><divclass="section"title="Installing ccollect"><divclass="titlepage"><div><div><h3class="title"><aname="_installing_ccollect"></a>Installing ccollect</h3></div></div></div><p>For the installation you need at least</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
the latest ccollect package (<aclass="ulink"href="http://www.nico.schottelius.org/software/ccollect/"target="_top">http://www.nico.schottelius.org/software/ccollect/</a>)
</li></ul></div></div><divclass="section"title="Using ccollect"><divclass="titlepage"><div><div><h3class="title"><aname="_using_ccollect"></a>Using ccollect</h3></div></div></div><divclass="itemizedlist"title="Running ccollect requires the following tools to be installed:"><pclass="title"><b>Running ccollect requires the following tools to be installed:</b></p><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div></div></div><divclass="section"title="Installing"><divclass="titlepage"><div><div><h2class="title"style="clear: both"><aname="_installing"></a>Installing</h2></div></div></div><p>Either type <spanclass="emphasis"><em>make install</em></span> or simply copy it to a directory in your
<spanclass="emphasis"><em>./tools/report_success.sh</em></span>.</p></div><divclass="section"title="Configuring"><divclass="titlepage"><div><div><h2class="title"style="clear: both"><aname="_configuring"></a>Configuring</h2></div></div></div><p>For configuration aid have a look at the above mentioned tools, which can assist
<codeclass="literal">tools/report_success.sh</code>.</p><divclass="section"title="Runtime options"><divclass="titlepage"><div><div><h3class="title"><aname="_runtime_options"></a>Runtime options</h3></div></div></div><p><codeclass="literal">ccollect</code> looks for its configuration in <spanclass="emphasis"><em>/etc/ccollect</em></span> or, if set, in
the directory specified by the variable <spanclass="emphasis"><em>$CCOLLECT_CONF</em></span>:</p><preclass="screen"># sh-compatible (dash, zsh, mksh, ksh, bash, ...)
$ CCOLLECT_CONF=/your/config/dir ccollect.sh ...
# csh
$ ( setenv CCOLLECT_CONF /your/config/dir ; ccollect.sh ... )</pre><p>When you start <codeclass="literal">ccollect</code>, you have to specify in which interval
to backup (daily, weekly, yearly; you can specify the names yourself, see below)
and which sources to backup (or -a to backup all sources).</p><p>The interval specifies how many backups are kept.</p><p>There are also some self-explanatory parameters you can pass to ccollect,
simply use <codeclass="literal">ccollect.sh --help</code> for info.</p></div><divclass="section"title="General configuration"><divclass="titlepage"><div><div><h3class="title"><aname="_general_configuration"></a>General configuration</h3></div></div></div><p>The general configuration can be found in $CCOLLECT_CONF/defaults or
configuration.</p><p>All configuration entries are plain-text files (use UTF-8 for non-ascii characters).</p><divclass="section"title="Interval definition"><divclass="titlepage"><div><div><h4class="title"><aname="_interval_definition"></a>Interval definition</h4></div></div></div><p>The interval definition can be found in
the same as the name of the interval: <codeclass="literal">intervals/'<interval name>'</code>.</p><p>The content of this file should be a single line containing a number.
This number defines how many versions of this interval are kept.</p><p>Example:</p><preclass="screen"> [10:23] zaphodbeeblebrox:ccollect-0.2% ls -l conf/defaults/intervals/
4</pre><p>This means to keep 28 daily backups, 12 monthly backups and 4 weekly.</p></div><divclass="section"title="General pre- and post-execution"><divclass="titlepage"><div><div><h4class="title"><aname="_general_pre_and_post_execution"></a>General pre- and post-execution</h4></div></div></div><p>If you add <spanclass="emphasis"><em>$CCOLLECT_CONF/defaults/<codeclass="literal">pre_exec</code></em></span> or
<spanclass="emphasis"><em>/etc/ccollect/defaults/<codeclass="literal">pre_exec</code></em></span> (same with <codeclass="literal">post_exec</code>), <codeclass="literal">ccollect</code>
will start <codeclass="literal">pre_exec</code> before the whole backup process and
<codeclass="literal">post_exec</code> after backup of all sources is done.</p><p>The following example describes how to report free disk space in
human readable format before and after the whole backup process:</p><preclass="screen">[13:00] hydrogenium:~# mkdir -p /etc/ccollect/defaults/
explanation.</p></div></div><divclass="section"title="Source configuration"><divclass="titlepage"><div><div><h3class="title"><aname="_source_configuration"></a>Source configuration</h3></div></div></div><p>Each source configuration exists in <spanclass="emphasis"><em>$CCOLLECT_CONF/sources/$name</em></span> or
<spanclass="emphasis"><em>/etc/ccollect/sources/$name</em></span>.</p><p>The name you choose for the subdirectory describes the source.</p><p>Each source contains at least the following files:</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div><p>Additionally a source may have the following files:</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
<codeclass="literal">very_verbose</code> be very verbose (<codeclass="literal">mkdir -v</code>, <codeclass="literal">rm -v</code> and <codeclass="literal">rsync -vv</code>)
/home/nico/vpn</pre><divclass="section"title='Detailed description of "source"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_source"></a>Detailed description of "source"</h4></div></div></div><p><codeclass="literal">source</code> describes a <codeclass="literal">rsync</code> compatible source (one line only).</p><p>For instance <spanclass="emphasis"><em>backup_user@foreign_host:/home/server/video</em></span>.
of <codeclass="literal">rsync</code>(1).</p></div><divclass="section"title='Detailed description of "destination"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_destination"></a>Detailed description of "destination"</h4></div></div></div><p><codeclass="literal">destination</code> must be a text file containing the destination directory.
<codeclass="literal">destination</code><spanclass="strong"><strong>USED</strong></span> to be a link to the destination directory in
earlier versions, so do not be confused if you see such examples.</p><p>Example:</p><preclass="screen"> [11:36] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/destination
/home/nico/backupdir</pre></div><divclass="section"title='Detailed description of "remote_host"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_remote_host"></a>Detailed description of "remote_host"</h4></div></div></div><p><codeclass="literal">remote_host</code> must be a text file containing the destination host.
If this file is existing, you are backing up your data <spanclass="strong"><strong>TO</strong></span> this host
and <spanclass="strong"><strong>not</strong></span> to you local host.</p><p><spanclass="strong"><strong>Warning</strong></span>: You need to have <codeclass="literal">ssh</code> access to the remote host. <codeclass="literal">rsync</code> and
<codeclass="literal">ccollect</code> will connect to that host via <codeclass="literal">ssh</code>. <codeclass="literal">ccollect</code> needs the shell
access, because it needs to find out how many backups exist on the remote
host and to be able to delete them.</p><p>Example:</p><preclass="screen"> [10:17] denkbrett:ccollect-0.7.0% cat conf/sources/remote1/remote_host
home.schottelius.org</pre><p>It may contain all the ssh-specific values like <spanclass="emphasis"><em>myuser@yourhost.ch</em></span>.</p></div><divclass="section"title='Detailed description of "verbose"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_verbose"></a>Detailed description of "verbose"</h4></div></div></div><p><codeclass="literal">verbose</code> tells <codeclass="literal">ccollect</code> that the log should contain verbose messages.</p><p>If this file exists in the source specification <spanclass="strong"><strong>-v</strong></span> will be passed to <codeclass="literal">rsync</code>.</p><p>`
Example:</p><preclass="screen"> [11:35] zaphodbeeblebrox:ccollect-0.2% touch conf/sources/testsource1/verbose</pre></div><divclass="section"title='Detailed description of "very_verbose"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_very_verbose"></a>Detailed description of "very_verbose"</h4></div></div></div><p><codeclass="literal">very_verbose</code> tells <codeclass="literal">ccollect</code> that it should log very verbosely.</p><p>If this file exists in the source specification <spanclass="strong"><strong>-v</strong></span> will be passed to
<codeclass="literal">rsync</code>, <codeclass="literal">rm</code> and <codeclass="literal">mkdir</code>.</p><p>Example:</p><preclass="screen"> [23:67] nohost:~% touch conf/sources/testsource1/very_verbose</pre></div><divclass="section"title='Detailed description of "summary"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_summary"></a>Detailed description of "summary"</h4></div></div></div><p>If you create the file <codeclass="literal">summary</code> in the source definition,
<codeclass="literal">ccollect</code> will present you a nice summary at the end.</p><preclass="screen">backup:~# touch /etc/ccollect/sources/root/summary
backup:~# ccollect.sh werktags root
==> ccollect.sh: Beginning backup using interval werktags <==
[root] Beginning to backup this source ...
[root] Currently 3 backup(s) exist, total keeping 50 backup(s).
[root] Beginning to backup, this may take some time...
[root] Hard linking...
[root] Transferring files...
[root]
[root] Number of files: 84183
[root] Number of files transferred: 32
[root] Total file size: 26234080536 bytes
[root] Total transferred file size: 9988252 bytes
[root] Literal data: 9988252 bytes
[root] Matched data: 0 bytes
[root] File list size: 3016771
[root] File list generation time: 1.786 seconds
[root] File list transfer time: 0.000 seconds
[root] Total bytes sent: 13009119
[root] Total bytes received: 2152
[root]
[root] sent 13009119 bytes received 2152 bytes 2891393.56 bytes/sec
[root] total size is 26234080536 speedup is 2016.26
[root] Successfully finished backup.
==> Finished ccollect.sh <==</pre><p>You could also combine it with <codeclass="literal">verbose</code> or <codeclass="literal">very_verbose</code>, but these
already print some statistics (though not all / the same as presented by
<codeclass="literal">summary</code>).</p></div><divclass="section"title='Detailed description of "exclude"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_exclude"></a>Detailed description of "exclude"</h4></div></div></div><p><codeclass="literal">exclude</code> specifies a list of paths to exclude. The entries are seperated by a newline (\n).</p><p>Example:</p><preclass="screen"> [11:35] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
something with spaces is not a problem</pre></div><divclass="section"title='Detailed description of "intervals/"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_intervals"></a>Detailed description of "intervals/"</h4></div></div></div><p>When you create the subdirectory <codeclass="literal">intervals/</code> in your source configuration
directory, you can specify individiual intervals for this specific source.
Each file in this directory describes an interval.</p><p>Example:</p><preclass="screen"> [11:37] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervals/
20</pre></div><divclass="section"title='Detailled description of "rsync_options"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailled_description_of_rsync_options"></a>Detailled description of "rsync_options"</h4></div></div></div><p>When you create the file <codeclass="literal">rsync_options</code> in your source configuration,
to use for automatic backup over the rsync-protocol.</p><p>Example:</p><preclass="screen"> [23:42] hydrogenium:ccollect-0.2% cat conf/sources/test_rsync/rsync_options
--password-file=/home/user/backup/protected_password_file</pre></div><divclass="section"title='Detailled description of "pre_exec" and "post_exec"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailled_description_of_pre_exec_and_post_exec"></a>Detailled description of "pre_exec" and "post_exec"</h4></div></div></div><p>When you create <codeclass="literal">pre_exec</code> and / or <codeclass="literal">post_exec</code> in your source
configuration, <codeclass="literal">ccollect</code> will execute this command before and
respectively after doing the backup for <spanclass="strong"><strong>this specific</strong></span> source.
If you want to have pre-/post-exec before and after <spanclass="strong"><strong>all</strong></span>
backups, see above for general configuration.</p><p>Example:</p><preclass="screen">[13:09] hydrogenium:ccollect-0.3% cat conf/sources/with_exec/pre_exec
df -h</pre></div><divclass="section"title='Detailed description of "delete_incomplete"'><divclass="titlepage"><div><div><h4class="title"><aname="_detailed_description_of_delete_incomplete"></a>Detailed description of "delete_incomplete"</h4></div></div></div><p>If you create the file <codeclass="literal">delete_incomplete</code> in a source specification directory,
the user.</p></div></div></div><divclass="section"title="Hints"><divclass="titlepage"><div><div><h2class="title"style="clear: both"><aname="_hints"></a>Hints</h2></div></div></div><divclass="section"title="Smart logging"><divclass="titlepage"><div><div><h3class="title"><aname="_smart_logging"></a>Smart logging</h3></div></div></div><p>Since ccollect-0.6.1 you can use the ccollect-logwrapper.sh(1) for logging.
ccollect-logwrapper.sh (11722): Finished.</pre></div><divclass="section"title="Using a different ssh port"><divclass="titlepage"><div><div><h3class="title"><aname="_using_a_different_ssh_port"></a>Using a different ssh port</h3></div></div></div><p>Mostly easy is to use your ~/.ssh/config file:</p><preclass="screen">host mx2.schottelius.org
HostKeyAlias bruehe.schottelius.org</pre></div><divclass="section"title="Using source names or interval in pre_/post_exec scripts"><divclass="titlepage"><div><div><h3class="title"><aname="_using_source_names_or_interval_in_pre_post_exec_scripts"></a>Using source names or interval in pre_/post_exec scripts</h3></div></div></div><p>The pre-/post_exec scripts can access some internal variables from <codeclass="literal">ccollect</code>:</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div></div><divclass="section"title="Using rsync protocol without ssh"><divclass="titlepage"><div><div><h3class="title"><aname="_using_rsync_protocol_without_ssh"></a>Using rsync protocol without ssh</h3></div></div></div><p>When you have a computer with little computing power, it may be useful to use
rsync without ssh, directly using the rsync protocol
(specify <spanclass="emphasis"><em>user@host::share</em></span> in <codeclass="literal">source</code>). You may wish to use
<codeclass="literal">rsync_options</code> to specify a password file to use for automatic backup.</p><p>Example:</p><preclass="screen">backup:~# cat /etc/ccollect/sources/sample.backup.host.org/source
this_is_the_rsync_password</pre><p>This hint was reported by Daniel Aubry.</p></div><divclass="section"title="Not excluding top-level directories"><divclass="titlepage"><div><div><h3class="title"><aname="_not_excluding_top_level_directories"></a>Not excluding top-level directories</h3></div></div></div><p>When you exclude "/proc" or "/mnt" from your backup, you may run into
trouble when you restore your backup. When you use "/proc/*" or "/mnt/\*"
instead, <codeclass="literal">ccollect</code> will backup empty directories.</p><divclass="note"title="Note"style="margin-left: 0.5in; margin-right: 0.5in;"><h3class="title">Note</h3><p>When those directories contain hidden files
they will still be transferred!</p></div><p>This hint was reported by Marcus Wagner.</p></div><divclass="section"title="Re-using already created rsync-backups"><divclass="titlepage"><div><div><h3class="title"><aname="_re_using_already_created_rsync_backups"></a>Re-using already created rsync-backups</h3></div></div></div><p>If you used <codeclass="literal">rsync</code> directly before you use <codeclass="literal">ccollect</code>, you can
daily.0</pre><p>Now you can use /home/backup/web1 as the <codeclass="literal">destination</code> for the backup.</p><divclass="note"title="Note"style="margin-left: 0.5in; margin-right: 0.5in;"><h3class="title">Note</h3><p>It does not matter anymore how you name your directory, as <codeclass="literal">ccollect</code> uses
the -c option from <codeclass="literal">ls</code> to find out which directory to clone from.</p></div><divclass="note"title="Note"style="margin-left: 0.5in; margin-right: 0.5in;"><h3class="title">Note</h3><p>Older versions (pre 0.6, iirc) had a problem, if you named the first backup
something like "daily.initial". It was needed to use the "<spanclass="strong"><strong>0</strong></span>" (or some
number that is lower than the current year) as extension. <codeclass="literal">ccollect</code>
used <codeclass="literal">sort</code> to find the latest backup. <codeclass="literal">ccollect</code> itself uses
<spanclass="emphasis"><em>interval.YEARMONTHDAY-HOURMINUTE.PID</em></span>. This notation was <spanclass="strong"><strong>always</strong></span> before
"daily.initial", as numbers are earlier in the list
which is produced by <codeclass="literal">sort</code>. So, if you had a directory named "daily.initial",
<codeclass="literal">ccollect</code> always diffed against this backup and transfered and deleted
files which where deleted in previous backups. This means you simply
wasted resources, but your backup had beer complete anyway.</p></div></div><divclass="section"title="Using pre_/post_exec"><divclass="titlepage"><div><div><h3class="title"><aname="_using_pre_post_exec"></a>Using pre_/post_exec</h3></div></div></div><p>Your pre_/post_exec script does not need to be a script, you can also
use a link to</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
</li></ul></div><p>The only requirement is that it is executable.</p></div><divclass="section"title="Using source specific interval definitions"><divclass="titlepage"><div><div><h3class="title"><aname="_using_source_specific_interval_definitions"></a>Using source specific interval definitions</h3></div></div></div><p>When you are backing up multiple hosts via cron each night, it may be
a problem that host "big_server" may only have 4 daily backups, because
otherwise its backup device will be full. But for all other hosts
you want to keep 20 daily backups. In this case you would create
<codeclass="literal">/etc/ccollect/default/intervals/daily</code> containing "20" and
<codeclass="literal">/etc/ccollect/sources/big_server/intervals/daily</code> containing "4".</p><p>Source specific intervals always overwrite the default values.
If you have to specify it individually for every host, because
of different requirements, you can even omit creating
<codeclass="literal">/etc/ccollect/default/intervals/daily</code>.</p></div><divclass="section"title="Comparing backups"><divclass="titlepage"><div><div><h3class="title"><aname="_comparing_backups"></a>Comparing backups</h3></div></div></div><p>If you want to see what changed between two backups, you can use
<codeclass="literal">rsync</code> directly:</p><preclass="screen">[12:00] u0255:ddba034.netstream.ch# rsync -n -a --delete --stats --progress daily.20080324-0313.17841/ daily.20080325-0313.31148/</pre><p>This results in a listing of changes. Because we pass -n to rsync no transfer
is made (i.e. report only mode)"</p><p>This hint was reported by Daniel Aubry.</p></div><divclass="section"title="Testing for host reachabilty"><divclass="titlepage"><div><div><h3class="title"><aname="_testing_for_host_reachabilty"></a>Testing for host reachabilty</h3></div></div></div><p>If you want to test whether the host you try to backup is reachable, you can use
# ping -c1 -q `cat "/etc/ccollect/sources/$name/source" | cut -d"@" -f2 | cut -d":" -f1`</pre><p>This prevents the deletion of old backups, if the host is not reachable.</p><p>This hint was reported by Daniel Aubry.</p></div><divclass="section"title="Easy check for errors"><divclass="titlepage"><div><div><h3class="title"><aname="_easy_check_for_errors"></a>Easy check for errors</h3></div></div></div><p>If you want to see whether there have been any errors while doing the backup,
you can run <codeclass="literal">ccollect</code> together with <codeclass="literal">ccollect_analyse_logs.sh</code>:</p><preclass="screen">$ ccollect | ccollect_analyse_logs.sh e</pre></div></div><divclass="section"title="F.A.Q."><divclass="titlepage"><div><div><h2class="title"style="clear: both"><aname="_f_a_q"></a>F.A.Q.</h2></div></div></div><divclass="section"title="What happens if one backup is broken or empty?"><divclass="titlepage"><div><div><h3class="title"><aname="_what_happens_if_one_backup_is_broken_or_empty"></a>What happens if one backup is broken or empty?</h3></div></div></div><p>Let us assume that one backup failed (connection broke or the source
hard disk had some failures). Therefore we’ve got one incomplete backup in our history.</p><p><codeclass="literal">ccollect</code> will transfer the missing files the next time you use it.
This leads to</p><divclass="itemizedlist"><ulclass="itemizedlist"type="disc"><liclass="listitem">
backup instead</p></div><divclass="section"title="When backing up from localhost the destination is also included. Is this a bug?"><divclass="titlepage"><div><div><h3class="title"><aname="_when_backing_up_from_localhost_the_destination_is_also_included_is_this_a_bug"></a>When backing up from localhost the destination is also included. Is this a bug?</h3></div></div></div><p>No. <codeclass="literal">ccollect</code> passes your source definition directly to <codeclass="literal">rsync</code>. It
to <spanclass="emphasis"><em>source/exclude</em></span>. (Daniel Aubry reported this problem)</p></div><divclass="section"title='Why does ccollect say "Permission denied" with my pre-/postexec script?'><divclass="titlepage"><div><div><h3class="title"><aname="_why_does_ccollect_say_permission_denied_with_my_pre_postexec_script"></a>Why does ccollect say "Permission denied" with my pre-/postexec script?</h3></div></div></div><p>The most common error is that you have not given your script the correct
permissions. Try <codeclass="literal">chmod 0755 /etc/ccollect/sources/'yoursource'/*_exec</code>`.</p></div><divclass="section"title="Why does the backup job fail when part of the source is a link?"><divclass="titlepage"><div><div><h3class="title"><aname="_why_does_the_backup_job_fail_when_part_of_the_source_is_a_link"></a>Why does the backup job fail when part of the source is a link?</h3></div></div></div><p>When a part of your path you specified in the source is a
(symbolic, hard links are not possible for directories) link,
the backup <spanclass="strong"><strong>must</strong></span> fail.</p><p>First of all, let us have a look at how it looks like:</p><preclass="screen">==> ccollect 0.4: Beginning backup using interval taeglich <==
[testsource] Sa Apr 29 00:01:55 CEST 2006 Beginning to backup
[testsource] Currently 0 backup(s) exist(s), total keeping 10 backup(s).
[testsource] Beginning to backup, this may take some time...
[testsource] Sa Apr 29 00:01:55 CEST 2006 Transferring files...
[testsource] rsync: recv_generator: mkdir "/etc/ccollect/sources/testsource/destination/taeglich.2006-04-29-0001.3874/home/user/nico/projekte/ccollect" failed: No such file or directory (2)
[testsource] rsync: stat "/etc/ccollect/sources/testsource/destination/taeglich.2006-04-29-0001.3874/home/user/nico/projekte/ccollect" failed: No such file or directory (2)
[...]</pre><p>So what is the problem? It is very obvious when you look deeper into it:</p><preclass="screen">% cat /etc/ccollect/sources/testsource/source
% ls -l /etc/ccollect/sources/testsource/destination/taeglich.2006-04-29-0001.3874/home/user/nico
lrwxrwxrwx 1 nico nico 29 2006-04-29 00:01 projekte -> oeffentlich/computer/projekte</pre><p><codeclass="literal">rsync</code> creates the directory structure before it creates the symbolic link.
This link now links to something not reachable (dead link). It is
impossible to create subdirectories under the broken link.</p><p>In conclusion you cannot use paths with a linked part.</p><p>However, you can backup directories containing symbolic links
(in this case you could backup /home/user/nico, which contains
/home/user/nico/projekte and oeffentlich/computer/projekte).</p></div><divclass="section"title="How can I prevent missing the right time to enter my password?"><divclass="titlepage"><div><div><h3class="title"><aname="_how_can_i_prevent_missing_the_right_time_to_enter_my_password"></a>How can I prevent missing the right time to enter my password?</h3></div></div></div><p>As <codeclass="literal">ccollect</code> first deletes the old backups, it may take some time
until <codeclass="literal">rsync</code> requests the password for the <codeclass="literal">ssh</code> session from you.</p><p>The easiest way not to miss that point is running <codeclass="literal">ccollect</code> in <codeclass="literal">screen</code>,
which has the ability to monitor the output for activity. So as soon as
your screen beeps, after <codeclass="literal">ccollect</code> began to remove the last directory,
you can enter your password (have a look at screen(1), especially "C-a M"
and "C-a _", for more information).</p></div></div><divclass="section"title="Examples"><divclass="titlepage"><div><div><h2class="title"style="clear: both"><aname="_examples"></a>Examples</h2></div></div></div><divclass="section"title="A backup host configuration from scratch"><divclass="titlepage"><div><div><h3class="title"><aname="_a_backup_host_configuration_from_scratch"></a>A backup host configuration from scratch</h3></div></div></div><preclass="screen">srwali01:~# mkdir /etc/ccollect
/o> ccollect.sh: Beginning backup using interval taeglich
/=> Beginning to backup "local-root" ...
|-> 0 backup(s) already exist, keeping 28 backup(s).</pre><p>After that, I added some more sources:</p><preclass="screen">srwali01:~# cd /etc/ccollect/sources
srwali01:/etc/ccollect/sources/srwali03# mkdir /mnt/hdbackup/srwali03</pre></div><divclass="section"title="Using hard-links requires less disk space"><divclass="titlepage"><div><div><h3class="title"><aname="_using_hard_links_requires_less_disk_space"></a>Using hard-links requires less disk space</h3></div></div></div><preclass="screen"># du (coreutils) 5.2.1
147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:45.588</pre><p>The backup of our main fileserver:</p><preclass="screen">backup:~# df -h /home/backup/srsyg01/
the hardlinks allocate.</p></div><divclass="section"title="A collection of backups on the backup server"><divclass="titlepage"><div><div><h3class="title"><aname="_a_collection_of_backups_on_the_backup_server"></a>A collection of backups on the backup server</h3></div></div></div><p>All the data of my important hosts is backuped to eiche into
woechentlich.2006-02-12-11:51.2514 woechentlich.2006-03-16-23:00.4218</pre><p>And this incremental backup and the archive are copied to an external
usb harddisk (attention: you <spanclass="strong"><strong>should</strong></span> really use -H to backup the backup):</p><preclass="screen">[9:23] eiche:backup# df -h
