# get latest ccollect tarball from http://www.nico.schottelius.org/software/ccollect/ @@ -624,15 +606,15 @@ du -s ~/DASI /bin # report success echo "Please report success using ./tools/report_success.sh"
ccollect is included into gentoo portage. Thanks to René Nussbaumer. +ccollect is included into gentoo portage. +Thanks to René Nussbaumer. ### Debian -
Marcus Wagner makes ccollect available as Debian packages.
-
To get ccollect insert the following line into your /etc/apt/sources.list:
- deb http://deb.notestc.de/ clinux/
-
| Revision History | ||
|---|---|---|
| Revision 0.7.1 | for ccollect 0.7.1, Initial Version from 2006-01-13 | NS |
Table of Contents
(pseudo) incremental backup +
| Revision History | ||
|---|---|---|
| Revision 0.7.1 | for ccollect 0.7.1, Initial Version from 2006-01-13 | NS |
Table of Contents
(pseudo) incremental backup
with different exclude lists
-using hardlinks and rsync
ccollect is a backup utility written in the sh-scripting language.
+using hardlinks and rsync
ccollect is a backup utility written in the sh-scripting language.
It does not depend on a specific shell, only /bin/sh needs to be
-bourne shell compatible (like dash, ksh, zsh, bash, …).
ccollect was successfully tested on the following platforms:
ccollect was successfully tested on the following platforms:
It should run on any Unix that supports rsync and has a POSIX-compatible
bourne shell. If your platform is not listed above and you have it successfully
-running, please drop me a mail.
While considering the design of ccollect, I thought about enabling +running, please drop me a mail.
While considering the design of ccollect, I thought about enabling backup to remote hosts. Though this sounds like a nice feature (Backup my notebook to the server now.), in my opinion it is a bad idea to backup to a remote host.
But as more and more people requested this feature, it was implemented, -so you have the choice whether you want to use it or not.
If you want to backup TO a remote host, you have to loosen security on it.
Imagine the following situation: You backup your farm of webservers TO +so you have the choice whether you want to use it or not.
If you want to backup TO a remote host, you have to loosen security on it.
Imagine the following situation: You backup your farm of webservers TO a backup host somewhere else. Now one of your webservers which has access to your backup host gets -compromised.
Your backup server will be compromised, too.
And the attacker will have access to all data on the other webservers.
Think of it the other way round: The backup server (now behind a +compromised.
Your backup server will be compromised, too.
And the attacker will have access to all data on the other webservers.
Think of it the other way round: The backup server (now behind a firewall, not accessable from outside) connects to the webservers and pulls the data from them. If someone gets access to one of the webservers, this person will perhaps not even see your machine. If the attacker sees connections from a host to the compromised machine, she will not be able to log in on the backup machine. -All other backups are still secure.
+The argument order changed: +
If you did not use arguments (most people do not), nothing will +change for you.
.
The format of destination changed:
You can update your configuration using tools/config-pre-0.7-to-0.7.sh.
Added remote_host
You can update your configuration using tools/config-pre-0.7-to-0.7.sh.
Added remote_host
The format of rsync_options changed:
rsync)
-You can update your configuration using tools/config-pre-0.6-to-0.6.sh.
The name of the backup directories changed:
You can update your configuration using tools/config-pre-0.6-to-0.6.sh.
The name of the backup directories changed:
For the second change there is no updated needed, as XXXX- is always before -XXXXX (- comes before digit).
Not a real incompatibilty, but seems to fit in this section:
0.5 does NOT require
Not a real incompatibilty, but seems to fit in this section:
0.5 does NOT require
anymore!
Since ccollect 0.4 there are several incompatibilities with earlier
-versions:
List of incompatibilities
anymore!
Since ccollect 0.4 there are several incompatibilities with earlier
+versions:
List of incompatibilities
pax (Posix) is now required, cp -al (GNU specific) is removed
-You can convert your old configuration directory using
config-pre-0.4-to-0.4.sh, which can be found in the tools/
-subdirectory:
[10:05] hydrogenium:ccollect-0.4# ./tools/config-pre-0.4-to-0.4.sh /etc/ccollect
For those who do not want to read the whole long document:
# get latest ccollect tarball from http://www.nico.schottelius.org/software/ccollect/ # replace value for CCV with the current version export CCV=0.7.1 @@ -118,37 +125,37 @@ du -s ~/DASI /bin # report success echo "Please report success using ./tools/report_success.sh"
Cutting and pasting the complete section above to your shell will result in the download of ccollect, the creation of a sample configuration and the -execution of some backups.
For the installation you need at least
For the installation you need at least
cp and chmod or install
-make
-asciidoc
-Either type make install or simply copy it to a directory in your $PATH and execute chmod 0755 /path/to/ccollect.sh. If you would like to use the new management scripts (available since 0.6), copy -the following scripts to a directory in $PATH:
tools/ccollect_add_source.sh
-tools/ccollect_analyse_logs.sh.sh
-tools/ccollect_delete_source.sh
-tools/ccollect_list_intervals.sh
-tools/ccollect_logwrapper.sh
After having installed and used ccollect, report success using -./tools/report_success.sh.
For configuration aid have a look at the above mentioned tools, which can assist +./tools/report_success.sh.
For configuration aid have a look at the above mentioned tools, which can assist
you quite well. When you are successfully using ccollect, report success using
-tools/report_success.sh.
ccollect looks for its configuration in /etc/ccollect or, if set, in
+tools/report_success.sh.
ccollect looks for its configuration in /etc/ccollect or, if set, in
the directory specified by the variable $CCOLLECT_CONF:
# sh-compatible (dash, zsh, mksh, ksh, bash, ...) $ CCOLLECT_CONF=/your/config/dir ccollect.sh ... @@ -156,13 +163,13 @@ $ CCOLLECT_CONF=/your/config/dir ccollect.sh ... $ ( setenv CCOLLECT_CONF /your/config/dir ; ccollect.sh ... )
When you start ccollect, you have to specify in which interval
to backup (daily, weekly, yearly; you can specify the names yourself, see below)
and which sources to backup (or -a to backup all sources).
The interval specifies how many backups are kept.
There are also some self-explanatory parameters you can pass to ccollect,
-simply use ccollect.sh --help for info.
The general configuration can be found in $CCOLLECT_CONF/defaults or
+simply use ccollect.sh —help for info.
The general configuration can be found in $CCOLLECT_CONF/defaults or /etc/ccollect/defaults. All options specified there are generally valid for all source definitions, although the values can be overwritten in the source -configuration.
All configuration entries are plain-text files (use UTF-8 for non-ascii characters).
The interval definition can be found in +configuration.
All configuration entries are plain-text files (use UTF-8 for non-ascii characters).
The interval definition can be found in
$CCOLLECT_CONF/defaults/intervals/ or /etc/ccollect/defaults/intervals.
Each file in this directory specifies an interval. The name of the file is
-the same as the name of the interval: intervals/'<interval name>'.
The content of this file should be a single line containing a number.
+the same as the name of the interval: intervals/<interval name>.
The content of this file should be a single line containing a number. This number defines how many versions of this interval are kept.
Example:
[10:23] zaphodbeeblebrox:ccollect-0.2% ls -l conf/defaults/intervals/
insgesamt 12
-rw-r--r-- 1 nico users 3 2005-12-08 10:24 daily
@@ -171,7 +178,7 @@ This number defines how many versions of this interval are kept.Example:<
[10:23] zaphodbeeblebrox:ccollect-0.2% cat conf/defaults/intervals/*
28
12
- 4
This means to keep 28 daily backups, 12 monthly backups and 4 weekly.
If you add $CCOLLECT_CONF/defaults/pre_exec or
+ 4
This means to keep 28 daily backups, 12 monthly backups and 4 weekly.
If you add $CCOLLECT_CONF/defaults/pre_exec or
/etc/ccollect/defaults/pre_exec (same with post_exec), ccollect
will start pre_exec before the whole backup process and
post_exec after backup of all sources is done.
The following example describes how to report free disk space in @@ -180,31 +187,37 @@ human readable format before and after the whole backup process:
If you add $CCOLLECT_CONF/defaults/
delete_incomplete, this +[13:01] hydrogenium:~# ln -s /etc/ccollect/defaults/pre_exec /etc/ccollect/defaults/post_exec
Each source configuration exists in $CCOLLECT_CONF/sources/$name or -/etc/ccollect/sources/$name.
The name you choose for the subdirectory describes the source.
Each source contains at least the following files:
Each source configuration exists in $CCOLLECT_CONF/sources/$name or +/etc/ccollect/sources/$name.
The name you choose for the subdirectory describes the source.
Each source contains at least the following files:
source (a text file containing the rsync compatible path to backup)
-destination (a text file containing the directory we should backup to)
-Additionally a source may have the following files:
Additionally a source may have the following files:
verbose whether to be verbose (passes -v to rsync)
-very_verbose be very verbose (mkdir -v, rm -v and rsync -vv)
-summary create a transfer summary when rsync finished
-exclude exclude list for rsync. newline seperated list.
-rsync_options extra options for rsync. newline seperated list.
-pre_exec program to execute before backing up this source
-post_exec program to execute after backing up this source
-delete_incomplete delete incomplete backups
-remote_host host to backup to
+rsync_failure_codes list of rsync exit codes that indicate complete failure
+mtime Sort backup directories based on their modification time
+quiet_if_down Suppress error messages if source is not connectable
Example:
[10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2
insgesamt 12
lrwxrwxrwx 1 nico users 20 2005-11-17 16:44 destination
@@ -221,21 +234,21 @@ explanation.source describes a rsync compatible source (one line only).
For instance backup_user@foreign_host:/home/server/video. + /home/nico/vpn
source describes a rsync compatible source (one line only).
For instance backup_user@foreign_host:/home/server/video.
To use the rsync protocol without the ssh-tunnel, use
rsync::USER@HOST/SRC. For more information have a look at the manpage
-of rsync(1).
destination must be a text file containing the destination directory.
+of rsync(1).
destination must be a text file containing the destination directory.
destination USED to be a link to the destination directory in
earlier versions, so do not be confused if you see such examples.
Example:
[11:36] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/destination - /home/nico/backupdir
remote_host must be a text file containing the destination host.
+ /home/nico/backupdir
remote_host must be a text file containing the destination host.
If this file is existing, you are backing up your data TO this host
and not to you local host.
Warning: You need to have ssh access to the remote host. rsync and
ccollect will connect to that host via ssh. ccollect needs the shell
access, because it needs to find out how many backups exist on the remote
host and to be able to delete them.
Example:
[10:17] denkbrett:ccollect-0.7.0% cat conf/sources/remote1/remote_host - home.schottelius.org
It may contain all the ssh-specific values like myuser@yourhost.ch.
verbose tells ccollect that the log should contain verbose messages.
If this file exists in the source specification -v will be passed to rsync.
` -Example:
[11:35] zaphodbeeblebrox:ccollect-0.2% touch conf/sources/testsource1/verbose
very_verbose tells ccollect that it should log very verbosely.
If this file exists in the source specification -v will be passed to
-rsync, rm and mkdir.
Example:
[23:67] nohost:~% touch conf/sources/testsource1/very_verbose
If you create the file summary in the source definition,
+ home.schottelius.org
It may contain all the ssh-specific values like myuser@yourhost.ch.
verbose tells ccollect that the log should contain verbose messages.
If this file exists in the source specification -v will be passed to rsync.
` +Example:
[11:35] zaphodbeeblebrox:ccollect-0.2% touch conf/sources/testsource1/verbose
very_verbose tells ccollect that it should log very verbosely.
If this file exists in the source specification -v will be passed to
+rsync, rm and mkdir.
Example:
[23:67] nohost:~% touch conf/sources/testsource1/very_verbose
If you create the file summary in the source definition,
ccollect will present you a nice summary at the end.
backup:~# touch /etc/ccollect/sources/root/summary backup:~# ccollect.sh werktags root ==> ccollect.sh: Beginning backup using interval werktags <== @@ -262,11 +275,11 @@ backup:~# ccollect.sh werktags root [root] Successfully finished backup. ==> Finished ccollect.sh <==
You could also combine it with verbose or very_verbose, but these
already print some statistics (though not all / the same as presented by
-summary).
exclude specifies a list of paths to exclude. The entries are seperated by a newline (\n).
Example:
[11:35] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
+summary).exclude specifies a list of paths to exclude. The entries are seperated by a newline (\n).
Example:
[11:35] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli
- something with spaces is not a problemWhen you create the subdirectory intervals/ in your source configuration
+ something with spaces is not a problem
When you create the subdirectory intervals/ in your source configuration
directory, you can specify individiual intervals for this specific source.
Each file in this directory describes an interval.
Example:
[11:37] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervals/
insgesamt 8
@@ -274,12 +287,12 @@ Each file in this directory describes an interval.Example:
When you create the file rsync_options in your source configuration,
+ 20
When you create the file rsync_options in your source configuration,
all the parameters in this file will be passed to rsync. This
way you can pass additional options to rsync. For instance you can tell rsync
-to show progress ("--progress"), or which -password-file ("--password-file")
+to show progress ("—progress"), or which -password-file ("—password-file")
to use for automatic backup over the rsync-protocol.
Example:
[23:42] hydrogenium:ccollect-0.2% cat conf/sources/test_rsync/rsync_options - --password-file=/home/user/backup/protected_password_file
When you create pre_exec and / or post_exec in your source
+ --password-file=/home/user/backup/protected_password_file
When you create pre_exec and / or post_exec in your source
configuration, ccollect will execute this command before and
respectively after doing the backup for this specific source.
If you want to have pre-/post-exec before and after all
@@ -292,16 +305,32 @@ df -h
#!/bin/sh
# Show whats free after
-df -h
If you create the file delete_incomplete in a source specification directory,
+df -h
If you have the file rsync_failure_codes in your source configuration
+directory, it should contain a newline-separated list of numbers representing
+rsync exit codes. If rsync exits with any code in this list, a marker will
+be left in the destination directory indicating failure of this backup. If
+you have enabled delete_incomplete, then this backup will be deleted during
+the next ccollect run on the same interval.
By default, ccollect.sh chooses the most recent backup directory for cloning or +the oldest for deletion based on the directory's last change time (ctime). +With this option, the sorting is done based on modification time (mtime). With +this version of ccollect, the ctime and mtime of your backups will normally +be the same and this option has no effect. However, if you, for example, move +your backups to another hard disk using cp -a or rsync -a, you should use this +option because the ctimes are not preserved during such operations.
If you have any backups in your repository made with ccollect version 0.7.1 or +earlier, do not use this option.
By default, ccollect.sh emits a series of error messages if a source is not +connectable. With this option enabled, ccollect still reports that the +source is not connectable but the associated error messages generated by +rsync or ssh are suppressed. You may want to use this option for sources, +like notebook PCs, that are often disconnected.
Since ccollect-0.6.1 you can use the ccollect-logwrapper.sh(1) for logging. You call it the same way you call ccollect.sh and it will create a logfile containing the output of ccollect.sh. For more information look at the manpage ccollect-logwrapper. The following is an example running ccollect-logwrapper.sh:
u0219 ~ # ~chdscni9/ccollect-logwrapper.sh daily u0160.nshq.ch.netstream.com ccollect-logwrapper.sh (11722): Starting with arguments: daily u0160.nshq.ch.netstream.com -ccollect-logwrapper.sh (11722): Finished.
Mostly easy is to use your ~/.ssh/config file:
host mx2.schottelius.org +ccollect-logwrapper.sh (11722): Finished.
Mostly easy is to use your ~/.ssh/config file:
host mx2.schottelius.org
Port 2342If you only use that port for backup only and normally want to use another port, you can add HostName and "HostKeyAlias" (if you also have different keys on the different ports):
Host hhydrogenium
@@ -312,16 +341,16 @@ keys on the different ports):Host hhydrogenium
Host bruehe
Hostname bruehe.schottelius.org
Port 22
- HostKeyAlias bruehe.schottelius.orgThe pre-/post_exec scripts can access some internal variables from ccollect:
The pre-/post_exec scripts can access some internal variables from ccollect:
When you have a computer with little computing power, it may be useful to use
rsync without ssh, directly using the rsync protocol
(specify user@host::share in source). You may wish to use
rsync_options to specify a password file to use for automatic backup.
Example:
backup:~# cat /etc/ccollect/sources/sample.backup.host.org/source @@ -331,11 +360,11 @@ backup:~# cat /etc/ccollect/sources/sample.backup.host.org/rsync_options --password-file=/etc/ccollect/sources/sample.backup.host.org/rsync_password backup:~# cat /etc/ccollect/sources/sample.backup.host.org/rsync_password -this_is_the_rsync_password
This hint was reported by Daniel Aubry.
When you exclude "/proc" or "/mnt" from your backup, you may run into
-trouble when you restore your backup. When you use "/proc/*" or "/mnt/\*"
-instead, ccollect will backup empty directories.
When those directories contain hidden files +this_is_the_rsync_password
This hint was reported by Daniel Aubry.
When you exclude "/proc" or "/mnt" from your backup, you may run into
+trouble when you restore your backup. When you use "/proc/*" or "/mnt/*"
+instead, ccollect will backup empty directories.
When those directories contain hidden files (those beginning with a dot (.)), -they will still be transferred!
This hint was reported by Marcus Wagner.
If you used rsync directly before you use ccollect, you can
+they will still be transferred!
This hint was reported by Marcus Wagner.
If you used rsync directly before you use ccollect, you can
use this old backup as initial backup for ccollect: You
simply move it into a directory below the destination directory
and name it "interval.0".
Example:
backup:/home/backup/web1# ls @@ -348,8 +377,8 @@ backup:/home/backup/web1# mkdir daily.0 backup:/home/backup/web1# mv * daily.0 2>/dev/null backup:/home/backup/web1# ls -daily.0
Now you can use /home/backup/web1 as the destination for the backup.
It does not matter anymore how you name your directory, as ccollect uses
-the -c option from ls to find out which directory to clone from.
Older versions (pre 0.6, iirc) had a problem, if you named the first backup +daily.0
Now you can use /home/backup/web1 as the destination for the backup.
It does not matter anymore how you name your directory, as ccollect uses
+the -c option from ls to find out which directory to clone from.
Older versions (pre 0.6, iirc) had a problem, if you named the first backup
something like "daily.initial". It was needed to use the "0" (or some
number that is lower than the current year) as extension. ccollect
used sort to find the latest backup. ccollect itself uses
@@ -358,12 +387,12 @@ used sort to find the latest backup. sort. So, if you had a directory named "daily.initial",
ccollect always diffed against this backup and transfered and deleted
files which where deleted in previous backups. This means you simply
-wasted resources, but your backup had beer complete anyway.
Your pre_/post_exec script does not need to be a script, you can also -use a link to
Your pre_/post_exec script does not need to be a script, you can also +use a link to
The only requirement is that it is executable.
When you are backing up multiple hosts via cron each night, it may be +
The only requirement is that it is executable.
When you are backing up multiple hosts via cron each night, it may be
a problem that host "big_server" may only have 4 daily backups, because
otherwise its backup device will be full. But for all other hosts
you want to keep 20 daily backups. In this case you would create
@@ -371,26 +400,26 @@ you want to keep 20 daily backups. In this case you would create
/etc/ccollect/sources/big_server/intervals/daily containing "4".
Source specific intervals always overwrite the default values.
If you have to specify it individually for every host, because
of different requirements, you can even omit creating
-/etc/ccollect/default/intervals/daily.
If you want to see what changed between two backups, you can use
+/etc/ccollect/default/intervals/daily.
If you want to see what changed between two backups, you can use
rsync directly:
[12:00] u0255:ddba034.netstream.ch# rsync -n -a --delete --stats --progress daily.20080324-0313.17841/ daily.20080325-0313.31148/
This results in a listing of changes. Because we pass -n to rsync no transfer -is made (i.e. report only mode)"
This hint was reported by Daniel Aubry.
If you want to test whether the host you try to backup is reachable, you can use +is made (i.e. report only mode)"
This hint was reported by Daniel Aubry.
If you want to test whether the host you try to backup is reachable, you can use the following script as source specific pre-exec:
#!/bin/sh -# ping -c1 -q `cat "/etc/ccollect/sources/$name/source" | cut -d"@" -f2 | cut -d":" -f1`
This prevents the deletion of old backups, if the host is not reachable.
This hint was reported by Daniel Aubry.
Let us assume that one backup failed (connection broke or the source -hard disk had some failures). Therefore we’ve got one incomplete backup in our history.
ccollect will transfer the missing files the next time you use it.
-This leads to
This prevents the deletion of old backups, if the host is not reachable.
This hint was reported by Daniel Aubry.
Let us assume that one backup failed (connection broke or the source +hard disk had some failures). Therefore we've got one incomplete backup in our history.
ccollect will transfer the missing files the next time you use it.
+This leads to
If the whole ccollect process was interrupted, ccollect (since 0.6) can
detect that and remove the incomplete backups, so you can clone from a complete
-backup instead
No. ccollect passes your source definition directly to rsync. It
+backup instead
No. ccollect passes your source definition directly to rsync. It
does not try to analyze it. So it actually does not know if a source
comes from local harddisk or from a remote server. And it does not want
to. When you backup from the local harddisk (which is perhaps not
even a good idea when thinking of security), add the destination
-to source/exclude. (Daniel Aubry reported this problem)
The most common error is that you have not given your script the correct
-permissions. Try chmod 0755 /etc/ccollect/sources/'yoursource'/*_exec`.
When a part of your path you specified in the source is a +to source/exclude. (Daniel Aubry reported this problem)
The most common error is that you have not given your script the correct
+permissions. Try chmod 0755 /etc/ccollect/sources/yoursource/*_exec`.
When a part of your path you specified in the source is a (symbolic, hard links are not possible for directories) link, the backup must fail.
First of all, let us have a look at how it looks like:
==> ccollect 0.4: Beginning backup using interval taeglich <== [testsource] Sa Apr 29 00:01:55 CEST 2006 Beginning to backup @@ -409,12 +438,12 @@ lrwxrwxrwx 1 nico nico 29 2006-04-29 00:01 projekte -> oeffentlich/computer/p This link now links to something not reachable (dead link). It is impossible to create subdirectories under the broken link.In conclusion you cannot use paths with a linked part.
However, you can backup directories containing symbolic links (in this case you could backup /home/user/nico, which contains -/home/user/nico/projekte and oeffentlich/computer/projekte).
As ccollect first deletes the old backups, it may take some time
+/home/user/nico/projekte and oeffentlich/computer/projekte).
As ccollect first deletes the old backups, it may take some time
until rsync requests the password for the ssh session from you.
The easiest way not to miss that point is running ccollect in screen,
which has the ability to monitor the output for activity. So as soon as
your screen beeps, after ccollect began to remove the last directory,
you can enter your password (have a look at screen(1), especially "C-a M"
-and "C-a _", for more information).
srwali01:~# mkdir /etc/ccollect srwali01:~# mkdir -p /etc/ccollect/defaults/intervals/ srwali01:~# echo 28 > /etc/ccollect/defaults/intervals/taeglich srwali01:~# echo 52 > /etc/ccollect/defaults/intervals/woechentlich @@ -459,7 +488,7 @@ srwali01:/etc/ccollect/sources/srwali03# cat > exclude << EOF > EOF srwali01:/etc/ccollect/sources/srwali03# echo 'root@10.103.2.3:/' > source srwali01:/etc/ccollect/sources/srwali03# echo /mnt/hdbackup/srwali03 > destination -srwali01:/etc/ccollect/sources/srwali03# mkdir /mnt/hdbackup/srwali03
# du (coreutils) 5.2.1 +srwali01:/etc/ccollect/sources/srwali03# mkdir /mnt/hdbackup/srwali03
# du (coreutils) 5.2.1 [10:53] srsyg01:sources% du -sh ~/backupdir 4.6M /home/nico/backupdir [10:53] srsyg01:sources% du -sh ~/backupdir/* @@ -504,7 +533,7 @@ du (GNU coreutils) 5.93 12G hydrogenium/durcheinander.2006-01-17-00:27.13820 1.5G hydrogenium/durcheinander.2006-01-25-23:18.31328 200M hydrogenium/durcheinander.2006-01-26-00:11.3332
In the second report (without -l) the sizes include the space the inodes of -the hardlinks allocate.
All the data of my important hosts is backuped to eiche into +the hardlinks allocate.
All the data of my important hosts is backuped to eiche into /mnt/schwarzesloch/backup:
[9:24] eiche:backup# ls * creme: woechentlich.2006-01-26-22:22.4153 woechentlich.2006-02-12-11:48.2461 @@ -544,7 +573,7 @@ DDIR=/mnt/usb/backup rsync -av -H --delete /mnt/schwarzesloch/ "$DDIR/schwarzesloch/" -rsync -av -H --delete /mnt/archiv/ "$DDIR/archiv/"
Truncated output from ps axuwwwf:
S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily -p ddba034 ddba045 ddba046 ddba047 ddba049 ddna010 ddna011 +rsync -av -H --delete /mnt/archiv/ "$DDIR/archiv/"
Truncated output from ps axuwwwf:
S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily -p ddba034 ddba045 ddba046 ddba047 ddba049 ddna010 ddna011
S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba034
S+ 11:40 0:00 | | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba034
R+ 11:40 23:40 | | | | | \_ rsync -a --delete --numeric-ids --relative --delete-excluded --link-dest=/home/server/backup/ddba034
diff --git a/software/ccollect/doc/ccollect.html b/software/ccollect/doc/ccollect.html
index 79fb5c7b..16270350 100644
--- a/software/ccollect/doc/ccollect.html
+++ b/software/ccollect/doc/ccollect.html
@@ -3,8 +3,7 @@
-
-ccollect - Installing, Configuring and Using
+
+ccollect - Installing, Configuring and Using
@@ -392,22 +351,22 @@ for ccollect 0.7.1, Initial Version from 2006-01-13
-(pseudo) incremental backup
+
(pseudo) incremental backup
with different exclude lists
using hardlinks and rsync
1. Introduction
-ccollect is a backup utility written in the sh-scripting language.
+
ccollect is a backup utility written in the sh-scripting language.
It does not depend on a specific shell, only /bin/sh needs to be
bourne shell compatible (like dash, ksh, zsh, bash, …).
1.1. Supported and tested operating systems and architectures
-ccollect was successfully tested on the following platforms:
-
+ccollect was successfully tested on the following platforms:
+
-
-GNU/Linux on amd64/hppa/i386/ppc
+GNU/Linux on amd64/hppa/i386/ppc/ARM
-
@@ -431,35 +390,58 @@ OpenBSD on amd64
-It should run on any Unix that supports rsync and has a POSIX-compatible
+
It should run on any Unix that supports rsync and has a POSIX-compatible
bourne shell. If your platform is not listed above and you have it successfully
running, please drop me a mail.
1.2. Why you COULD only backup from remote hosts, not to them
-While considering the design of ccollect, I thought about enabling
+
While considering the design of ccollect, I thought about enabling
backup to remote hosts. Though this sounds like a nice feature
(Backup my notebook to the server now.), in my opinion it is a
bad idea to backup to a remote host.
-But as more and more people requested this feature, it was implemented,
+
But as more and more people requested this feature, it was implemented,
so you have the choice whether you want to use it or not.
1.2.1. Reason
-If you want to backup TO a remote host, you have to loosen security on it.
-Imagine the following situation: You backup your farm of webservers TO
+
If you want to backup TO a remote host, you have to loosen security on it.
+Imagine the following situation: You backup your farm of webservers TO
a backup host somewhere else.
Now one of your webservers which has access to your backup host gets
compromised.
-Your backup server will be compromised, too.
-And the attacker will have access to all data on the other webservers.
+Your backup server will be compromised, too.
+And the attacker will have access to all data on the other webservers.
1.2.2. Doing it securely
-Think of it the other way round: The backup server (now behind a
+
Think of it the other way round: The backup server (now behind a
firewall, not accessable from outside) connects to the
webservers and pulls the data from them. If someone gets access to one
of the webservers, this person will perhaps not even see your machine. If
the attacker sees connections from a host to the compromised
machine, she will not be able to log in on the backup machine.
All other backups are still secure.
-1.3. Incompatibilities
-1.3.1. Versions 0.6 and 0.7
-The format of destination changed:
+1.3. Incompatibilities and changes
+1.3.1. Versions 0.7 and 0.8
+
+-
+
+The argument order changed:
+
+
+-
+
+Old: "<interval name> [args] <sources to backup>"
+
+
+-
+
+New: "[args] <interval name> <sources to backup>"
+
+
+
+
+
+If you did not use arguments (most people do not), nothing will
+change for you.
+.
+1.3.2. Versions 0.6 and 0.7
+The format of destination changed:
-
Before 0.7 it was a (link to a) directory
@@ -471,16 +453,16 @@ As of 0.7 it is a textfile containing the destination
-You can update your configuration using tools/config-pre-0.7-to-0.7.sh.
-Added remote_host
+You can update your configuration using tools/config-pre-0.7-to-0.7.sh.
+Added remote_host
-
As of 0.7 it is possible to backup to hosts (see section remote_host below).
-1.3.2. Versions 0.5 and 0.6
-The format of rsync_options changed:
+1.3.3. Versions 0.5 and 0.6
+The format of rsync_options changed:
-
Before 0.6 it was whitespace delimeted
@@ -492,8 +474,8 @@ As of 0.6 it is newline seperated (so you can pass whitespaces to rsync
-You can update your configuration using tools/config-pre-0.6-to-0.6.sh.
-The name of the backup directories changed:
+You can update your configuration using tools/config-pre-0.6-to-0.6.sh.
+The name of the backup directories changed:
-
Before 0.6: "date +%Y-%m-%d-%H%M"
@@ -505,11 +487,11 @@ As of 0.6: "date +%Y%m%d-%H%M" (better readable, date is closer together)
-For the second change there is no updated needed, as XXXX- is always before
+
For the second change there is no updated needed, as XXXX- is always before
XXXXX (- comes before digit).
-1.3.3. Versions 0.4 and 0.5
-Not a real incompatibilty, but seems to fit in this section:
-0.5 does NOT require
+1.3.4. Versions 0.4 and 0.5
+Not a real incompatibilty, but seems to fit in this section:
+0.5 does NOT require
-
PaX
@@ -521,11 +503,11 @@ bc
-anymore!
-1.3.4. Versions < 0.4 and 0.4
-Since ccollect 0.4 there are several incompatibilities with earlier
+
anymore!
+1.3.5. Versions < 0.4 and 0.4
+Since ccollect 0.4 there are several incompatibilities with earlier
versions:
-List of incompatibilities
+List of incompatibilities
-
pax (Posix) is now required, cp -al (GNU specific) is removed
@@ -552,7 +534,7 @@ ccollect now reports when postexec returns non-zero
-You can convert your old configuration directory using
+
You can convert your old configuration directory using
config-pre-0.4-to-0.4.sh, which can be found in the tools/
subdirectory:
@@ -562,7 +544,7 @@ subdirectory:
2. Quick start
-For those who do not want to read the whole long document:
+For those who do not want to read the whole long document:
# get latest ccollect tarball from http://www.nico.schottelius.org/software/ccollect/
@@ -624,15 +606,15 @@ du -s ~/DASI /bin
# report success
echo "Please report success using ./tools/report_success.sh"
-Cutting and pasting the complete section above to your shell will result in
+
Cutting and pasting the complete section above to your shell will result in
the download of ccollect, the creation of a sample configuration and the
execution of some backups.
3. Requirements
3.1. Installing ccollect
-For the installation you need at least
-
+For the installation you need at least
+
-
the latest ccollect package (http://www.nico.schottelius.org/software/ccollect/)
@@ -655,7 +637,7 @@ for rebuilding the generated documentation: additionally asciidoc
3.2. Using ccollect
-Running ccollect requires the following tools to be installed:
+Running ccollect requires the following tools to be installed:
-
date
@@ -675,11 +657,11 @@ for rebuilding the generated documentation: additionally asciidoc
4. Installing
-Either type make install or simply copy it to a directory in your
+
Either type make install or simply copy it to a directory in your
$PATH and execute chmod 0755 /path/to/ccollect.sh. If you would
like to use the new management scripts (available since 0.6), copy
the following scripts to a directory in $PATH:
-
+
-
tools/ccollect_add_source.sh
@@ -706,16 +688,16 @@ the following scripts to a directory in $PATH:
-After having installed and used ccollect, report success using
+
After having installed and used ccollect, report success using
./tools/report_success.sh.
5. Configuring
-For configuration aid have a look at the above mentioned tools, which can assist
+
For configuration aid have a look at the above mentioned tools, which can assist
you quite well. When you are successfully using ccollect, report success using
tools/report_success.sh.
5.1. Runtime options
-ccollect looks for its configuration in /etc/ccollect or, if set, in
+
ccollect looks for its configuration in /etc/ccollect or, if set, in
the directory specified by the variable $CCOLLECT_CONF:
@@ -725,26 +707,26 @@ $ CCOLLECT_CONF=/your/config/dir ccollect.sh ...
# csh
$ ( setenv CCOLLECT_CONF /your/config/dir ; ccollect.sh ... )
-When you start ccollect, you have to specify in which interval
+
When you start ccollect, you have to specify in which interval
to backup (daily, weekly, yearly; you can specify the names yourself, see below)
and which sources to backup (or -a to backup all sources).
-The interval specifies how many backups are kept.
-There are also some self-explanatory parameters you can pass to ccollect,
-simply use ccollect.sh --help for info.
+The interval specifies how many backups are kept.
+There are also some self-explanatory parameters you can pass to ccollect,
+simply use ccollect.sh —help for info.
5.2. General configuration
-The general configuration can be found in $CCOLLECT_CONF/defaults or
+
The general configuration can be found in $CCOLLECT_CONF/defaults or
/etc/ccollect/defaults. All options specified there are generally valid for
all source definitions, although the values can be overwritten in the source
configuration.
-All configuration entries are plain-text files (use UTF-8 for non-ascii characters).
+All configuration entries are plain-text files (use UTF-8 for non-ascii characters).
5.2.1. Interval definition
-The interval definition can be found in
+
The interval definition can be found in
$CCOLLECT_CONF/defaults/intervals/ or /etc/ccollect/defaults/intervals.
Each file in this directory specifies an interval. The name of the file is
-the same as the name of the interval: intervals/'<interval name>'.
-The content of this file should be a single line containing a number.
+the same as the name of the interval: intervals/<interval name>.
+The content of this file should be a single line containing a number.
This number defines how many versions of this interval are kept.
-Example:
+Example:
[10:23] zaphodbeeblebrox:ccollect-0.2% ls -l conf/defaults/intervals/
@@ -757,13 +739,13 @@ This number defines how many versions of this interval are kept.
12
4
-This means to keep 28 daily backups, 12 monthly backups and 4 weekly.
+This means to keep 28 daily backups, 12 monthly backups and 4 weekly.
5.2.2. General pre- and post-execution
-If you add $CCOLLECT_CONF/defaults/pre_exec or
+
If you add $CCOLLECT_CONF/defaults/pre_exec or
/etc/ccollect/defaults/pre_exec (same with post_exec), ccollect
will start pre_exec before the whole backup process and
post_exec after backup of all sources is done.
-The following example describes how to report free disk space in
+
The following example describes how to report free disk space in
human readable format before and after the whole backup process:
@@ -775,15 +757,15 @@ human readable format before and after the whole backup process:
[13:01] hydrogenium:~# ln -s /etc/ccollect/defaults/pre_exec /etc/ccollect/defaults/post_exec
5.2.3. General delete_incomplete
-If you add $CCOLLECT_CONF/defaults/delete_incomplete, this
+
If you add $CCOLLECT_CONF/defaults/delete_incomplete, this
option applies for all sources. See below for a longer
explanation.
5.3. Source configuration
-Each source configuration exists in $CCOLLECT_CONF/sources/$name or
+
Each source configuration exists in $CCOLLECT_CONF/sources/$name or
/etc/ccollect/sources/$name.
-The name you choose for the subdirectory describes the source.
-Each source contains at least the following files:
-
+The name you choose for the subdirectory describes the source.
+Each source contains at least the following files:
+
-
source (a text file containing the rsync compatible path to backup)
@@ -795,8 +777,8 @@ explanation.
-Additionally a source may have the following files:
-
+Additionally a source may have the following files:
+
-
verbose whether to be verbose (passes -v to rsync)
@@ -842,8 +824,23 @@ explanation.
remote_host host to backup to
+-
+
+rsync_failure_codes list of rsync exit codes that indicate complete failure
+
+
+-
+
+mtime Sort backup directories based on their modification time
+
+
+-
+
+quiet_if_down Suppress error messages if source is not connectable
+
+
-Example:
+Example:
[10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2
@@ -865,56 +862,56 @@ explanation.
/home/nico/vpn
5.3.1. Detailed description of "source"
-source describes a rsync compatible source (one line only).
-For instance backup_user@foreign_host:/home/server/video.
+
source describes a rsync compatible source (one line only).
+For instance backup_user@foreign_host:/home/server/video.
To use the rsync protocol without the ssh-tunnel, use
rsync::USER@HOST/SRC. For more information have a look at the manpage
of rsync(1).
5.3.2. Detailed description of "destination"
-destination must be a text file containing the destination directory.
+
destination must be a text file containing the destination directory.
destination USED to be a link to the destination directory in
earlier versions, so do not be confused if you see such examples.
-Example:
+Example:
[11:36] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/destination
/home/nico/backupdir
5.3.3. Detailed description of "remote_host"
-remote_host must be a text file containing the destination host.
+
remote_host must be a text file containing the destination host.
If this file is existing, you are backing up your data TO this host
and not to you local host.
-Warning: You need to have ssh access to the remote host. rsync and
+
Warning: You need to have ssh access to the remote host. rsync and
ccollect will connect to that host via ssh. ccollect needs the shell
access, because it needs to find out how many backups exist on the remote
host and to be able to delete them.
-Example:
+Example:
[10:17] denkbrett:ccollect-0.7.0% cat conf/sources/remote1/remote_host
home.schottelius.org
-It may contain all the ssh-specific values like myuser@yourhost.ch.
+It may contain all the ssh-specific values like myuser@yourhost.ch.
5.3.4. Detailed description of "verbose"
-verbose tells ccollect that the log should contain verbose messages.
-If this file exists in the source specification -v will be passed to rsync.
-`
+
verbose tells ccollect that the log should contain verbose messages.
+If this file exists in the source specification -v will be passed to rsync.
+`
Example:
[11:35] zaphodbeeblebrox:ccollect-0.2% touch conf/sources/testsource1/verbose
5.3.5. Detailed description of "very_verbose"
-very_verbose tells ccollect that it should log very verbosely.
-If this file exists in the source specification -v will be passed to
+
very_verbose tells ccollect that it should log very verbosely.
+If this file exists in the source specification -v will be passed to
rsync, rm and mkdir.
-Example:
+Example:
[23:67] nohost:~% touch conf/sources/testsource1/very_verbose
5.3.6. Detailed description of "summary"
-If you create the file summary in the source definition,
+
If you create the file summary in the source definition,
ccollect will present you a nice summary at the end.
@@ -944,12 +941,12 @@ backup:~# ccollect.sh werktags root
[root] Successfully finished backup.
==> Finished ccollect.sh <==
-You could also combine it with verbose or very_verbose, but these
+
You could also combine it with verbose or very_verbose, but these
already print some statistics (though not all / the same as presented by
summary).
5.3.7. Detailed description of "exclude"
-exclude specifies a list of paths to exclude. The entries are seperated by a newline (\n).
-Example:
+exclude specifies a list of paths to exclude. The entries are seperated by a newline (\n).
+Example:
[11:35] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
@@ -959,10 +956,10 @@ already print some statistics (though not all / the same as presented by
something with spaces is not a problem
5.3.8. Detailed description of "intervals/"
-When you create the subdirectory intervals/ in your source configuration
+
When you create the subdirectory intervals/ in your source configuration
directory, you can specify individiual intervals for this specific source.
Each file in this directory describes an interval.
-Example:
+Example:
[11:37] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervals/
@@ -974,24 +971,24 @@ Each file in this directory describes an interval.
20
5.3.9. Detailled description of "rsync_options"
-When you create the file rsync_options in your source configuration,
+
When you create the file rsync_options in your source configuration,
all the parameters in this file will be passed to rsync. This
way you can pass additional options to rsync. For instance you can tell rsync
-to show progress ("--progress"), or which -password-file ("--password-file")
+to show progress ("—progress"), or which -password-file ("—password-file")
to use for automatic backup over the rsync-protocol.
-Example:
+Example:
[23:42] hydrogenium:ccollect-0.2% cat conf/sources/test_rsync/rsync_options
--password-file=/home/user/backup/protected_password_file
5.3.10. Detailled description of "pre_exec" and "post_exec"
-When you create pre_exec and / or post_exec in your source
+
When you create pre_exec and / or post_exec in your source
configuration, ccollect will execute this command before and
respectively after doing the backup for this specific source.
If you want to have pre-/post-exec before and after all
backups, see above for general configuration.
-Example:
+Example:
[13:09] hydrogenium:ccollect-0.3% cat conf/sources/with_exec/pre_exec
@@ -1006,15 +1003,38 @@ df -h
df -h
5.3.11. Detailed description of "delete_incomplete"
-If you create the file delete_incomplete in a source specification directory,
+
If you create the file delete_incomplete in a source specification directory,
ccollect will look for incomplete backups (when the whole ccollect process
was interrupted) and remove them. Without this file ccollect will only warn
the user.
+5.3.12. Detailed description of "rsync_failure_codes"
+If you have the file rsync_failure_codes in your source configuration
+directory, it should contain a newline-separated list of numbers representing
+rsync exit codes. If rsync exits with any code in this list, a marker will
+be left in the destination directory indicating failure of this backup. If
+you have enabled delete_incomplete, then this backup will be deleted during
+the next ccollect run on the same interval.
+5.3.13. Detailed description of "mtime"
+By default, ccollect.sh chooses the most recent backup directory for cloning or
+the oldest for deletion based on the directory's last change time (ctime).
+With this option, the sorting is done based on modification time (mtime). With
+this version of ccollect, the ctime and mtime of your backups will normally
+be the same and this option has no effect. However, if you, for example, move
+your backups to another hard disk using cp -a or rsync -a, you should use this
+option because the ctimes are not preserved during such operations.
+If you have any backups in your repository made with ccollect version 0.7.1 or
+earlier, do not use this option.
+5.3.14. Detailed description of "quiet_if_down"
+By default, ccollect.sh emits a series of error messages if a source is not
+connectable. With this option enabled, ccollect still reports that the
+source is not connectable but the associated error messages generated by
+rsync or ssh are suppressed. You may want to use this option for sources,
+like notebook PCs, that are often disconnected.
6. Hints
6.1. Smart logging
-Since ccollect-0.6.1 you can use the ccollect-logwrapper.sh(1) for logging.
+
Since ccollect-0.6.1 you can use the ccollect-logwrapper.sh(1) for logging.
You call it the same way you call ccollect.sh and it will create a
logfile containing the output of ccollect.sh. For more information look
at the manpage ccollect-logwrapper. The following is an example running
@@ -1026,13 +1046,13 @@ ccollect-logwrapper.sh (11722): Starting with arguments: daily u0160.nshq.ch.net
ccollect-logwrapper.sh (11722): Finished.
6.2. Using a different ssh port
-Mostly easy is to use your ~/.ssh/config file:
+Mostly easy is to use your ~/.ssh/config file:
host mx2.schottelius.org
Port 2342
-If you only use that port for backup only and normally want to use another port,
+
If you only use that port for backup only and normally want to use another port,
you can add HostName and "HostKeyAlias" (if you also have different
keys on the different ports):
@@ -1048,8 +1068,8 @@ Host bruehe
HostKeyAlias bruehe.schottelius.org
6.3. Using source names or interval in pre_/post_exec scripts
-The pre-/post_exec scripts can access some internal variables from ccollect:
-
+The pre-/post_exec scripts can access some internal variables from ccollect:
+
-
INTERVAL: The interval specified on the command line
@@ -1073,11 +1093,11 @@ name: the name of the currently being backuped source (not available for
6.4. Using rsync protocol without ssh
-When you have a computer with little computing power, it may be useful to use
+
When you have a computer with little computing power, it may be useful to use
rsync without ssh, directly using the rsync protocol
(specify user@host::share in source). You may wish to use
rsync_options to specify a password file to use for automatic backup.
-Example:
+Example:
backup:~# cat /etc/ccollect/sources/sample.backup.host.org/source
@@ -1089,10 +1109,10 @@ backup:~# cat /etc/ccollect/sources/sample.backup.host.org/rsync_options
backup:~# cat /etc/ccollect/sources/sample.backup.host.org/rsync_password
this_is_the_rsync_password
-This hint was reported by Daniel Aubry.
+This hint was reported by Daniel Aubry.
6.5. Not excluding top-level directories
-When you exclude "/proc" or "/mnt" from your backup, you may run into
-trouble when you restore your backup. When you use "/proc/*" or "/mnt/\*"
+
When you exclude "/proc" or "/mnt" from your backup, you may run into
+trouble when you restore your backup. When you use "/proc/*" or "/mnt/*"
instead, ccollect will backup empty directories.
@@ -1100,19 +1120,19 @@ instead, ccollect will backup empty directories.
Note
-When those directories contain hidden files
+
When those directories contain hidden files
(those beginning with a dot (.)),
they will still be transferred!
-This hint was reported by Marcus Wagner.
+This hint was reported by Marcus Wagner.
6.6. Re-using already created rsync-backups
-If you used rsync directly before you use ccollect, you can
+
If you used rsync directly before you use ccollect, you can
use this old backup as initial backup for ccollect: You
simply move it into a directory below the destination directory
and name it "interval.0".
-Example:
+Example:
backup:/home/backup/web1# ls
@@ -1127,14 +1147,14 @@ backup:/home/backup/web1# mv * daily.0 2>/dev/null
backup:/home/backup/web1# ls
daily.0
-Now you can use /home/backup/web1 as the destination for the backup.
+Now you can use /home/backup/web1 as the destination for the backup.
Note
-It does not matter anymore how you name your directory, as ccollect uses
+
It does not matter anymore how you name your directory, as ccollect uses
the -c option from ls to find out which directory to clone from.
@@ -1145,7 +1165,7 @@ the -c option from ls to find out which directory to clone from.Note
-Older versions (pre 0.6, iirc) had a problem, if you named the first backup
+
Older versions (pre 0.6, iirc) had a problem, if you named the first backup
something like "daily.initial". It was needed to use the "0" (or some
number that is lower than the current year) as extension. ccollect
used sort to find the latest backup. ccollect itself uses
@@ -1159,9 +1179,9 @@ wasted resources, but your backup had beer complete anyway.
6.7. Using pre_/post_exec
-Your pre_/post_exec script does not need to be a script, you can also
+
Your pre_/post_exec script does not need to be a script, you can also
use a link to
-
+
-
an existing program
@@ -1173,40 +1193,40 @@ an already written script
-The only requirement is that it is executable.
+The only requirement is that it is executable.
6.8. Using source specific interval definitions
-When you are backing up multiple hosts via cron each night, it may be
+
When you are backing up multiple hosts via cron each night, it may be
a problem that host "big_server" may only have 4 daily backups, because
otherwise its backup device will be full. But for all other hosts
you want to keep 20 daily backups. In this case you would create
/etc/ccollect/default/intervals/daily containing "20" and
/etc/ccollect/sources/big_server/intervals/daily containing "4".
-Source specific intervals always overwrite the default values.
+
Source specific intervals always overwrite the default values.
If you have to specify it individually for every host, because
of different requirements, you can even omit creating
/etc/ccollect/default/intervals/daily.
6.9. Comparing backups
-If you want to see what changed between two backups, you can use
+
If you want to see what changed between two backups, you can use
rsync directly:
[12:00] u0255:ddba034.netstream.ch# rsync -n -a --delete --stats --progress daily.20080324-0313.17841/ daily.20080325-0313.31148/
-This results in a listing of changes. Because we pass -n to rsync no transfer
+
This results in a listing of changes. Because we pass -n to rsync no transfer
is made (i.e. report only mode)"
-This hint was reported by Daniel Aubry.
+This hint was reported by Daniel Aubry.
6.10. Testing for host reachabilty
-If you want to test whether the host you try to backup is reachable, you can use
+
If you want to test whether the host you try to backup is reachable, you can use
the following script as source specific pre-exec:
#!/bin/sh
# ping -c1 -q `cat "/etc/ccollect/sources/$name/source" | cut -d"@" -f2 | cut -d":" -f1`
-This prevents the deletion of old backups, if the host is not reachable.
-This hint was reported by Daniel Aubry.
+This prevents the deletion of old backups, if the host is not reachable.
+This hint was reported by Daniel Aubry.
6.11. Easy check for errors
-If you want to see whether there have been any errors while doing the backup,
+
If you want to see whether there have been any errors while doing the backup,
you can run ccollect together with ccollect_analyse_logs.sh:
@@ -1216,11 +1236,11 @@ you can run ccollect together with ccollect_analyse_logs.sh:7. F.A.Q.
7.1. What happens if one backup is broken or empty?
-Let us assume that one backup failed (connection broke or the source
-hard disk had some failures). Therefore we’ve got one incomplete backup in our history.
-ccollect will transfer the missing files the next time you use it.
+
Let us assume that one backup failed (connection broke or the source
+hard disk had some failures). Therefore we've got one incomplete backup in our history.
+ccollect will transfer the missing files the next time you use it.
This leads to
-
+
-
more transferred files
@@ -1232,24 +1252,24 @@ much greater disk space usage, as no hardlinks can be used
-If the whole ccollect process was interrupted, ccollect (since 0.6) can
+
If the whole ccollect process was interrupted, ccollect (since 0.6) can
detect that and remove the incomplete backups, so you can clone from a complete
backup instead
7.2. When backing up from localhost the destination is also included. Is this a bug?
-No. ccollect passes your source definition directly to rsync. It
+
No. ccollect passes your source definition directly to rsync. It
does not try to analyze it. So it actually does not know if a source
comes from local harddisk or from a remote server. And it does not want
to. When you backup from the local harddisk (which is perhaps not
even a good idea when thinking of security), add the destination
to source/exclude. (Daniel Aubry reported this problem)
7.3. Why does ccollect say "Permission denied" with my pre-/postexec script?
-The most common error is that you have not given your script the correct
-permissions. Try chmod 0755 /etc/ccollect/sources/'yoursource'/*_exec`.
+The most common error is that you have not given your script the correct
+permissions. Try chmod 0755 /etc/ccollect/sources/yoursource/*_exec`.
7.4. Why does the backup job fail when part of the source is a link?
-When a part of your path you specified in the source is a
+
When a part of your path you specified in the source is a
(symbolic, hard links are not possible for directories) link,
the backup must fail.
-First of all, let us have a look at how it looks like:
+First of all, let us have a look at how it looks like:
==> ccollect 0.4: Beginning backup using interval taeglich <==
@@ -1262,7 +1282,7 @@ the backup must fail.
[testsource] rsync: stat "/etc/ccollect/sources/testsource/destination/taeglich.2006-04-29-0001.3874/home/user/nico/projekte/ccollect" failed: No such file or directory (2)
[...]
-So what is the problem? It is very obvious when you look deeper into it:
+So what is the problem? It is very obvious when you look deeper into it:
% cat /etc/ccollect/sources/testsource/source
@@ -1272,17 +1292,17 @@ lrwxrwxrwx 1 nico nico 29 2005-12-02 23:28 /home/user/nico/projekte -> oeffen
% ls -l /etc/ccollect/sources/testsource/destination/taeglich.2006-04-29-0001.3874/home/user/nico
lrwxrwxrwx 1 nico nico 29 2006-04-29 00:01 projekte -> oeffentlich/computer/projekte
-rsync creates the directory structure before it creates the symbolic link.
+
rsync creates the directory structure before it creates the symbolic link.
This link now links to something not reachable (dead link). It is
impossible to create subdirectories under the broken link.
-In conclusion you cannot use paths with a linked part.
-However, you can backup directories containing symbolic links
+
In conclusion you cannot use paths with a linked part.
+However, you can backup directories containing symbolic links
(in this case you could backup /home/user/nico, which contains
/home/user/nico/projekte and oeffentlich/computer/projekte).
7.5. How can I prevent missing the right time to enter my password?
-As ccollect first deletes the old backups, it may take some time
+
As ccollect first deletes the old backups, it may take some time
until rsync requests the password for the ssh session from you.
-The easiest way not to miss that point is running ccollect in screen,
+
The easiest way not to miss that point is running ccollect in screen,
which has the ability to monitor the output for activity. So as soon as
your screen beeps, after ccollect began to remove the last directory,
you can enter your password (have a look at screen(1), especially "C-a M"
@@ -1316,7 +1336,7 @@ srwali01:/etc/ccollect/sources/local-root# ccollect.sh taeglich local-root
/=> Beginning to backup "local-root" ...
|-> 0 backup(s) already exist, keeping 28 backup(s).
-After that, I added some more sources:
+After that, I added some more sources:
srwali01:~# cd /etc/ccollect/sources
@@ -1365,7 +1385,7 @@ srwali01:/etc/ccollect/sources# du -sh /mnt/hdbackup/wl6/*
147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:42.312
147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:45.588
-The backup of our main fileserver:
+The backup of our main fileserver:
backup:~# df -h /home/backup/srsyg01/
@@ -1387,7 +1407,7 @@ backup:~# du -sh /home/backup/srsyg01/*
backup:~# du --version | head -n1
du (coreutils) 5.2.1
-Newer versions of du also detect the hardlinks, so we can even compare
+
Newer versions of du also detect the hardlinks, so we can even compare
the sizes directly with du:
@@ -1404,10 +1424,10 @@ du (GNU coreutils) 5.93
1.5G hydrogenium/durcheinander.2006-01-25-23:18.31328
200M hydrogenium/durcheinander.2006-01-26-00:11.3332
-In the second report (without -l) the sizes include the space the inodes of
+
In the second report (without -l) the sizes include the space the inodes of
the hardlinks allocate.
8.3. A collection of backups on the backup server
-All the data of my important hosts is backuped to eiche into
+
All the data of my important hosts is backuped to eiche into
/mnt/schwarzesloch/backup:
@@ -1432,7 +1452,7 @@ woechentlich.2006-02-06-08:22.15994 woechentlich.2006-03-02-23:00.17346
woechentlich.2006-02-06-19:40.16321 woechentlich.2006-03-09-23:00.29317
woechentlich.2006-02-12-11:51.2514 woechentlich.2006-03-16-23:00.4218
-And this incremental backup and the archive are copied to an external
+
And this incremental backup and the archive are copied to an external
usb harddisk (attention: you should really use -H to backup the backup):
@@ -1458,7 +1478,7 @@ rsync -av -H --delete /mnt/schwarzesloch/ "$DDIR/schwarzesloch/"
rsync -av -H --delete /mnt/archiv/ "$DDIR/archiv/"
8.4. Processes running when doing ccollect -p
-Truncated output from ps axuwwwf:
+Truncated output from ps axuwwwf:
S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily -p ddba034 ddba045 ddba046 ddba047 ddba049 ddna010 ddna011
@@ -1500,13 +1520,13 @@ rsync -av -H --delete /mnt/archiv/ "$DDIR/archiv/"
S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddna011
S+ 11:40 0:00 | | | \_ sed s:^:\[ddna011\] :
-As you can see, six processes are deleting old backups, while one backup
+
As you can see, six processes are deleting old backups, while one backup
(ddba034) is already copying data.
diff --git a/software/ccollect/doc/ccollect.text b/software/ccollect/doc/ccollect.text
index 984cdcf8..b762ecf3 100644
--- a/software/ccollect/doc/ccollect.text
+++ b/software/ccollect/doc/ccollect.text
@@ -21,7 +21,7 @@ Supported and tested operating systems and architectures
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
`ccollect` was successfully tested on the following platforms:
-- GNU/Linux on amd64/hppa/i386/ppc
+- GNU/Linux on amd64/hppa/i386/ppc/ARM
- FreeBSD on amd64/i386
- Mac OS X 10.5
- NetBSD on alpha/amd64/i386/sparc/sparc64
@@ -68,8 +68,22 @@ machine, she will not be able to log in on the backup machine.
All other backups are still secure.
-Incompatibilities
-~~~~~~~~~~~~~~~~~
+Incompatibilities and changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+Versions 0.7 and 0.8
+^^^^^^^^^^^^^^^^^^^^^
+
+. The argument order changed:
+- Old: " [args] "
+- New: "[args] "
+
+If you did not use arguments (most people do not), nothing will
+change for you.
+
+.
+
Versions 0.6 and 0.7
@@ -359,6 +373,9 @@ Additionally a source may have the following files:
- `delete_incomplete` delete incomplete backups
- `remote_host` host to backup to
+ - `rsync_failure_codes` list of rsync exit codes that indicate complete failure
+ - `mtime` Sort backup directories based on their modification time
+ - `quiet_if_down` Suppress error messages if source is not connectable
Example:
@@ -574,6 +591,36 @@ If you create the file `delete_incomplete` in a source specification directory,
was interrupted) and remove them. Without this file `ccollect` will only warn
the user.
+Detailed description of "rsync_failure_codes"
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+If you have the file `rsync_failure_codes` in your source configuration
+directory, it should contain a newline-separated list of numbers representing
+rsync exit codes. If rsync exits with any code in this list, a marker will
+be left in the destination directory indicating failure of this backup. If
+you have enabled delete_incomplete, then this backup will be deleted during
+the next ccollect run on the same interval.
+
+Detailed description of "mtime"
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+By default, ccollect.sh chooses the most recent backup directory for cloning or
+the oldest for deletion based on the directory's last change time (ctime).
+With this option, the sorting is done based on modification time (mtime). With
+this version of ccollect, the ctime and mtime of your backups will normally
+be the same and this option has no effect. However, if you, for example, move
+your backups to another hard disk using cp -a or rsync -a, you should use this
+option because the ctimes are not preserved during such operations.
+
+If you have any backups in your repository made with ccollect version 0.7.1 or
+earlier, do not use this option.
+
+Detailed description of "quiet_if_down"
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+By default, ccollect.sh emits a series of error messages if a source is not
+connectable. With this option enabled, ccollect still reports that the
+source is not connectable but the associated error messages generated by
+rsync or ssh are suppressed. You may want to use this option for sources,
+like notebook PCs, that are often disconnected.
+
Hints
-----
diff --git a/software/smtp_logger.mdwn b/software/smtp_logger.mdwn
new file mode 100644
index 00000000..47167722
--- /dev/null
+++ b/software/smtp_logger.mdwn
@@ -0,0 +1,67 @@
+## Introduction
+
+smtp_logger is a logging smtp proxy. It is designed to aid mail server
+administrators debugging their smtp connections.
+
+## How does it work?
+
+ ------------------
+ | SMTP Client(s) |
+ ------------------
+ |
+ | want to connect to smtp-real.example.org
+ |
+ ------------------
+ | Router/firewall|
+ ------------------
+ |
+ | redirects data to loghost
+ |
+ ------------------
+ | loghost |
+ ------------------
+ |
+ | Saves connection data and forwards them
+ |
+ ------------------
+ | smtp-real | processes the mail
+ ------------------
+
+Have a look at the README file for further instructions.
+
+## How to get smtp_logger
+
+### Releases
+
+ * **smtp_logger-0.1**: the first public release
+ ([[tar|smtp_logger-0.1.tar]]
+ [[lzma|smtp_logger-0.1.tar.lzma]]
+ [[gz|smtp_logger-0.1.tar.gz]]
+ [[bz2|smtp_logger-0.1.tar.bz2]])
+
+### Development
+
+The latest development code can be found in git.
+You can view latest changes in
+[gitweb](http://git.schottelius.org/?p=smtp_logger)
+or clone the latest sources using
+
+ git clone git://git.schottelius.org/smtp_logger
+
+To submit changes, simply follow the instructions
+on [how to setup a public git repo](http://book.git-scm.com/4_setting_up_a_public_repository.html).
+
+There are also some
+[CIA bot statistics](http://cia.vc/stats/project/smtp_logger) available.
+
+## Support
+
+### IRC
+
+ * [#cLinux](irc://irc.freenode.org/#cLinux) - Multi language (German/English)
+
+### Mail
+
+You can also [[contact me directly|about]].
+
+[[!tag unix]]
diff --git a/software/smtp_logger/smtp_logger-0.1.tar b/software/smtp_logger/smtp_logger-0.1.tar
new file mode 100644
index 00000000..77653405
Binary files /dev/null and b/software/smtp_logger/smtp_logger-0.1.tar differ
diff --git a/software/smtp_logger/smtp_logger-0.1.tar.bz2 b/software/smtp_logger/smtp_logger-0.1.tar.bz2
new file mode 100644
index 00000000..6c07b368
Binary files /dev/null and b/software/smtp_logger/smtp_logger-0.1.tar.bz2 differ
diff --git a/software/smtp_logger/smtp_logger-0.1.tar.gz b/software/smtp_logger/smtp_logger-0.1.tar.gz
new file mode 100644
index 00000000..b547865d
Binary files /dev/null and b/software/smtp_logger/smtp_logger-0.1.tar.gz differ
diff --git a/software/smtp_logger/smtp_logger-0.1.tar.lzma b/software/smtp_logger/smtp_logger-0.1.tar.lzma
new file mode 100644
index 00000000..352997a6
Binary files /dev/null and b/software/smtp_logger/smtp_logger-0.1.tar.lzma differ