(pseudo) incremental backup with different exclude lists -using hardlinks and rsync
rsync| Revision History | ||
|---|---|---|
| Revision 0.8.1 | for ccollect 0.8.1, Initial Version from 2006-01-13 | NS |
Table of Contents
(pseudo) incremental backup +
| Revision History | ||
|---|---|---|
| Revision 2.0 | for ccollect 2.0, Initial Version from 2006-01-13 | NS |
Table of Contents
(pseudo) incremental backup
with different exclude lists
using hardlinks and rsync
ccollect is a backup utility written in the sh-scripting language.
It does not depend on a specific shell, only /bin/sh needs to be
@@ -27,7 +27,16 @@ webservers and pulls the data from
of the webservers, this person will perhaps not even see your machine. If
the attacker sees connections from a host to the compromised
machine, she will not be able to log in on the backup machine.
-All other backups are still secure.
The argument order changed:
If you have the file rsync_failure_codes in your source configuration
directory, it should contain a newline-separated list of numbers representing
-rsync exit codes. If rsync exits with any code in this list, a marker will
+rsync exit codes. If rsync exits with any code in this list, a marker will
be left in the destination directory indicating failure of this backup. If
you have enabled delete_incomplete, then this backup will be deleted during
the next ccollect run on the same interval.
By default, ccollect.sh chooses the most recent backup directory for cloning or the oldest for deletion based on the directory’s last change time (ctime). -With this option, the sorting is done based on modification time (mtime). With +With this option, the sorting is done based on modification time (mtime). With this version of ccollect, the ctime and mtime of your backups will normally be the same and this option has no effect. However, if you, for example, move your backups to another hard disk using cp -a or rsync -a, you should use this @@ -380,7 +389,7 @@ keys on the different ports):
Host hhydrogenium
Host bruehe
Hostname bruehe.schottelius.org
Port 22
- HostKeyAlias bruehe.schottelius.orgThe pre-/post_exec scripts can access some internal variables from ccollect:
The pre-/post_exec scripts can access some internal variables from ccollect:
sort to find the latest backup. sort. So, if you had a directory named "daily.initial",
ccollect always diffed against this backup and transfered and deleted
files which where deleted in previous backups. This means you simply
-wasted resources, but your backup had beer complete anyway.Your pre_/post_exec script does not need to be a script, you can also use a link to
destination
to source/exclude. (Daniel Aubry reported this problem)The most common error is that you have not given your script the correct
-permissions. Try chmod 0755 /etc/ccollect/sources/'yoursource'/*_exec`.
When a part of your path you specified in the source is a
+permissions. Try chmod 0755 /etc/ccollect/sources/'yoursource'/*_exec`.
When a part of your path you specified in the source is a (symbolic, hard links are not possible for directories) link, the backup must fail.
First of all, let us have a look at how it looks like:
==> ccollect 0.4: Beginning backup using interval taeglich <== [testsource] Sa Apr 29 00:01:55 CEST 2006 Beginning to backup @@ -484,7 +493,21 @@ untilrsyncrequests the password for theccollectbegan to remove the last directory, you can enter your password (have a look at screen(1), especially "C-a M" -and "C-a _", for more information).
srwali01:~# mkdir /etc/ccollect +and "C-a _", for more information).
If you are trying to backup a system containing paths that are managed +by autofs, you may run into this error:
2009-12-01-23:14:15: ccollect 0.8.1: Beginning backup using interval monatlich +[ikn] 2009-12-01-23:14:15: Beginning to backup +[ikn] 2009-12-01-23:14:15: Executing /home/users/nico/ethz/ccollect/sources/ikn/pre_exec ... +Enter LUKS passphrase: +[ikn] Command successful. +[ikn] key slot 0 unlocked. +[ikn] 2009-12-01-23:14:23: Finished /home/users/nico/ethz/ccollect/sources/ikn/pre_exec (return code 0). [ikn] directory has vanished: "/home/users/nico/privat/firmen/ethz/autofs/projects" +[ikn] directory has vanished: "/home/users/nico/privat/firmen/ethz/autofs/scratch" +[ikn] directory has vanished: "/home/users/nico/privat/firmen/ethz/autofs/sgscratch" +[ikn] directory has vanished: "/home/users/nico/privat/firmen/ethz/autofs/supp" +[ikn] directory has vanished: "/home/users/nico/privat/firmen/ethz/autofs/sysadmin" +[ikn] rsync warning: some files vanished before they could be transferred (code 24) at main.c(1057) [sender=3.0.6] +[ikn] 2009-12-01-23:44:23: Source / is not readable. Skipping.
Thus, if you are unsure whether autofs paths can be mounted during backup, +stop autofs in pre_exec and reenable it in post_exec.
srwali01:~# mkdir /etc/ccollect srwali01:~# mkdir -p /etc/ccollect/defaults/intervals/ srwali01:~# echo 28 > /etc/ccollect/defaults/intervals/taeglich srwali01:~# echo 52 > /etc/ccollect/defaults/intervals/woechentlich @@ -614,7 +637,7 @@ DDIR=/mnt/usb/backup rsync -av -H --delete /mnt/schwarzesloch/ "$DDIR/schwarzesloch/" -rsync -av -H --delete /mnt/archiv/ "$DDIR/archiv/"
Truncated output from ps axuwwwf:
S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily -p ddba034 ddba045 ddba046 ddba047 ddba049 ddna010 ddna011 +rsync -av -H --delete /mnt/archiv/ "$DDIR/archiv/"
Truncated output from ps axuwwwf:
S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily -j ddba034 ddba045 ddba046 ddba047 ddba049 ddna010 ddna011
S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba034
S+ 11:40 0:00 | | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba034
R+ 11:40 23:40 | | | | | \_ rsync -a --delete --numeric-ids --relative --delete-excluded --link-dest=/home/server/backup/ddba034
diff --git a/software/ccollect/documentation/ccollect.html b/software/ccollect/documentation/ccollect.html
index 760526ea..b1858848 100644
--- a/software/ccollect/documentation/ccollect.html
+++ b/software/ccollect/documentation/ccollect.html
@@ -2,15 +2,25 @@
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
-
-
+
+
ccollect - Installing, Configuring and Using
+
-
+
ccollect - Installing, Configuring and Using
-<nico-ccollect__@__schottelius.org>
-version 0.8.1,
-for ccollect 0.8.1, Initial Version from 2006-01-13
+<nico-ccollect__@__schottelius.org>
+version 2.0,
+for ccollect 2.0, Initial Version from 2006-01-13
+
(pseudo) incremental backup
with different exclude lists
-using hardlinks and rsync
+using hardlinks and rsync
+
1. Introduction
-ccollect is a backup utility written in the sh-scripting language.
-It does not depend on a specific shell, only /bin/sh needs to be
+
ccollect is a backup utility written in the sh-scripting language.
+It does not depend on a specific shell, only /bin/sh needs to be
bourne shell compatible (like dash, ksh, zsh, bash, …).
-1.1. Supported and tested operating systems and architectures
-ccollect was successfully tested on the following platforms:
+
+1.1. Supported and tested operating systems and architectures
+ccollect was successfully tested on the following platforms:
-
@@ -431,16 +783,19 @@ OpenBSD on amd64
-It should run on any Unix that supports rsync and has a POSIX-compatible
+
It should run on any Unix that supports rsync and has a POSIX-compatible
bourne shell. If your platform is not listed above and you have it successfully
running, please drop me a mail.
-1.2. Why you COULD only backup from remote hosts, not to them
+
+
+1.2. Why you COULD only backup from remote hosts, not to them
While considering the design of ccollect, I thought about enabling
backup to remote hosts. Though this sounds like a nice feature
("Backup my notebook to the server now."), in my opinion it is a
bad idea to backup to a remote host.
But as more and more people requested this feature, it was implemented,
so you have the choice whether you want to use it or not.
+
1.2.1. Reason
If you want to backup TO a remote host, you have to loosen security on it.
Imagine the following situation: You backup your farm of webservers TO
@@ -449,6 +804,8 @@ Now one of your webservers which has access to your backup host gets
compromised.
Your backup server will be compromised, too.
And the attacker will have access to all data on the other webservers.
+
+
1.2.2. Doing it securely
Think of it the other way round: The backup server (now behind a
firewall, not accessable from outside) connects to the
@@ -457,8 +814,43 @@ of the webservers, this person will perhaps not even see your machine. If
the attacker sees connections from a host to the compromised
machine, she will not be able to log in on the backup machine.
All other backups are still secure.
-1.3. Incompatibilities and changes
-1.3.1. Versions 0.7 and 0.8
+
+
+
+1.3. Incompatibilities and changes
+
+1.3.1. Versions 0.9 and 1.0
+
+-
+
+Added "Error: " prefix in _exit_err()
+
+
+
+
+
+1.3.2. Versions 0.8 and 0.9
+
+-
+
+Renamed script to ccollect (.sh is not needed)
+
+
+-
+
+Removed feature to backup to a host via ccollect, added new tool
+ (FIXME: insert name here) that takes care of this via tunnel
+
+
+-
+
+Perhaps creating subdirectory of source name (idea from Stefan Schlörholz)
+
+
+
+
+
+1.3.3. Versions 0.7 and 0.8
The argument order changed:
-
@@ -497,8 +889,10 @@ New: Options in $CCOLLECT_CONF/defaults are used as defaults (see below)
-1.3.2. Versions 0.6 and 0.7
-The format of destination changed:
+
+
+1.3.4. Versions 0.6 and 0.7
+The format of destination changed:
-
Before 0.7 it was a (link to a) directory
@@ -510,7 +904,7 @@ As of 0.7 it is a textfile containing the destination
-You can update your configuration using tools/config-pre-0.7-to-0.7.sh.
+You can update your configuration using tools/config-pre-0.7-to-0.7.sh.
Added remote_host
-
@@ -518,8 +912,10 @@ As of 0.7 it is possible to backup to hosts (see section remote
-1.3.3. Versions 0.5 and 0.6
-The format of rsync_options changed:
+
+
+1.3.5. Versions 0.5 and 0.6
+The format of rsync_options changed:
-
Before 0.6 it was whitespace delimeted
@@ -527,11 +923,11 @@ Before 0.6 it was whitespace delimeted
-
-As of 0.6 it is newline seperated (so you can pass whitespaces to rsync)
+As of 0.6 it is newline seperated (so you can pass whitespaces to rsync)
-You can update your configuration using tools/config-pre-0.6-to-0.6.sh.
+You can update your configuration using tools/config-pre-0.6-to-0.6.sh.
The name of the backup directories changed:
-
@@ -546,7 +942,9 @@ As of 0.6: "date +%Y%m%d-%H%M" (better readable, date is closer together)
For the second change there is no updated needed, as XXXX- is always before
XXXXX (- comes before digit).
-1.3.4. Versions 0.4 and 0.5
+
+
+1.3.6. Versions 0.4 and 0.5
Not a real incompatibilty, but seems to fit in this section:
0.5 does NOT require
-
@@ -561,13 +959,15 @@ bc
anymore!
-1.3.5. Versions < 0.4 and 0.4
-Since ccollect 0.4 there are several incompatibilities with earlier
+
+
+1.3.7. Versions < 0.4 and 0.4
+Since ccollect 0.4 there are several incompatibilities with earlier
versions:
List of incompatibilities
-
-pax (Posix) is now required, cp -al (GNU specific) is removed
+pax (Posix) is now required, cp -al (GNU specific) is removed
-
@@ -592,19 +992,23 @@ ccollect now reports when postexec returns non-zero
You can convert your old configuration directory using
-config-pre-0.4-to-0.4.sh, which can be found in the tools/
+config-pre-0.4-to-0.4.sh, which can be found in the tools/
subdirectory:
-[10:05] hydrogenium:ccollect-0.4# ./tools/config-pre-0.4-to-0.4.sh /etc/ccollect
+[10:05] hydrogenium:ccollect-0.4# ./tools/config-pre-0.4-to-0.4.sh /etc/ccollect
+
+
+
+
2. Quick start
For those who do not want to read the whole long document:
-# get latest ccollect tarball from http://www.nico.schottelius.org/software/ccollect/
+# get latest ccollect tarball from http://www.nico.schottelius.org/software/ccollect/
# replace value for CCV with the current version
export CCV=0.8.1
@@ -661,15 +1065,18 @@ echo "Let's see how much space we used with two backups and compare it to /bin"
du -s ~/DASI /bin
# report success
-echo "Please report success using ./tools/report_success.sh"
+echo "Please report success using ./tools/report_success.sh"
Cutting and pasting the complete section above to your shell will result in
the download of ccollect, the creation of a sample configuration and the
execution of some backups.
+
+
3. Requirements
-3.1. Installing ccollect
+
+3.1. Installing ccollect
For the installation you need at least
-
@@ -679,39 +1086,44 @@ the latest ccollect package (3.2. Using ccollect
-Running ccollect requires the following tools to be installed:
--
-
-date
-
-
--
-
-rsync
-
-
--
-
-ssh (if you want to use rsync over ssh, which is recommened for security)
+for rebuilding the generated documentation: additionally asciidoc
+
+3.2. Using ccollect
+Running ccollect requires the following tools to be installed:
+-
+
+date
+
+
+-
+
+rsync
+
+
+-
+
+ssh (if you want to use rsync over ssh, which is recommened for security)
+
+
+
+
+
+
+
4. Installing
Either type make install or simply copy it to a directory in your
@@ -721,75 +1133,81 @@ following scripts to a directory in $PATH:
-
-tools/ccollect_add_source.sh
+tools/ccollect_add_source.sh
-
-tools/ccollect_analyse_logs.sh.sh
+tools/ccollect_analyse_logs.sh.sh
-
-tools/ccollect_delete_source.sh
+tools/ccollect_delete_source.sh
-
-tools/ccollect_list_intervals.sh
+tools/ccollect_list_intervals.sh
-
-tools/ccollect_logwrapper.sh
+tools/ccollect_logwrapper.sh
After having installed and used ccollect, report success using
./tools/report_success.sh.
+
+
5. Configuring
For configuration aid have a look at the above mentioned tools, which can assist
-you quite well. When you are successfully using ccollect, I would be happy if
+you quite well. When you are successfully using ccollect, I would be happy if
you add a link to your website, stating "I backup with ccollect", which points
to the ccollect homepage. So more people now about ccollect, use it and
-improve it. You can also report success using tools/report_success.sh.
-5.1. Runtime options
-ccollect looks for its configuration in /etc/ccollect or, if set, in
+improve it. You can also report success using tools/report_success.sh.
+
+5.1. Runtime options
+ccollect looks for its configuration in /etc/ccollect or, if set, in
the directory specified by the variable $CCOLLECT_CONF:
-# sh-compatible (dash, zsh, mksh, ksh, bash, ...)
+# sh-compatible (dash, zsh, mksh, ksh, bash, ...)
$ CCOLLECT_CONF=/your/config/dir ccollect.sh ...
# csh
-$ ( setenv CCOLLECT_CONF /your/config/dir ; ccollect.sh ... )
+$ ( setenv CCOLLECT_CONF /your/config/dir ; ccollect.sh ... )
-When you start ccollect, you have to specify in which interval
+
When you start ccollect, you have to specify in which interval
to backup (daily, weekly, yearly; you can specify the names yourself, see below)
and which sources to backup (or -a to backup all sources).
The interval specifies how many backups are kept.
There are also some self-explanatory parameters you can pass to ccollect,
-simply use ccollect.sh --help for info.
-5.2. General configuration
+simply use ccollect.sh --help for info.
+
+
+5.2. General configuration
The general configuration can be found in $CCOLLECT_CONF/defaults or
/etc/ccollect/defaults. All options specified there are generally valid for
all source definitions, although the values can be overwritten in the source
configuration.
All configuration entries are plain-text files
(use UTF-8 for non-ascii characters).
+
5.2.1. Interval definition
The interval definition can be found in
$CCOLLECT_CONF/defaults/intervals/ or /etc/ccollect/defaults/intervals.
Each file in this directory specifies an interval. The name of the file is
-the same as the name of the interval: intervals/'<interval name>'.
+the same as the name of the interval: intervals/'<interval name>'.
The content of this file should be a single line containing a number.
This number defines how many versions of this interval are kept.
Example:
- [10:23] zaphodbeeblebrox:ccollect-0.2% ls -l conf/defaults/intervals/
+ [10:23] zaphodbeeblebrox:ccollect-0.2% ls -l conf/defaults/intervals/
insgesamt 12
-rw-r--r-- 1 nico users 3 2005-12-08 10:24 daily
-rw-r--r-- 1 nico users 3 2005-12-08 11:36 monthly
@@ -797,31 +1215,33 @@ This number defines how many versions of this interval are kept.
[10:23] zaphodbeeblebrox:ccollect-0.2% cat conf/defaults/intervals/*
28
12
- 4
+ 4
This means to keep 28 daily backups, 12 monthly backups and 4 weekly.
+
+
5.2.2. General pre- and post-execution
-If you add $CCOLLECT_CONF/defaults/pre_exec or
-/etc/ccollect/defaults/pre_exec (same with post_exec), ccollect
-will start pre_exec before the whole backup process and
-post_exec after backup of all sources is done.
-If pre_exec exits with a non-zero return code, the whole backup
+
If you add $CCOLLECT_CONF/defaults/pre_exec or
+/etc/ccollect/defaults/pre_exec (same with post_exec), ccollect
+will start pre_exec before the whole backup process and
+post_exec after backup of all sources is done.
+If pre_exec exits with a non-zero return code, the whole backup
process will be aborted.
-The pre_exec and post_exec script can access the following exported variables:
+The pre_exec and post_exec script can access the following exported variables:
-
-INTERVAL: the interval selected (daily)
+INTERVAL: the interval selected (daily)
-
-no_sources: number of sources to backup (2)
+no_sources: number of sources to backup (2)
-
-source_$no: name of the source, $no starts at 0 ($source_0)
+source_$no: name of the source, $no starts at 0 ($source_0)
@@ -829,14 +1249,17 @@ process will be aborted.
human readable format before and after the whole backup process:
-[13:00] hydrogenium:~# mkdir -p /etc/ccollect/defaults/
+[13:00] hydrogenium:~# mkdir -p /etc/ccollect/defaults/
[13:00] hydrogenium:~# echo '#!/bin/sh' > /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# echo '' >> /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# echo 'df -h' >> /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# chmod 0755 /etc/ccollect/defaults/pre_exec
-[13:01] hydrogenium:~# ln -s /etc/ccollect/defaults/pre_exec /etc/ccollect/defaults/post_exec
+[13:01] hydrogenium:~# ln -s /etc/ccollect/defaults/pre_exec /etc/ccollect/defaults/post_exec
-5.3. Source configuration
+
+
+
+5.3. Source configuration
Each source configuration exists in $CCOLLECT_CONF/sources/$name or
/etc/ccollect/sources/$name.
The name you choose for the subdirectory describes the source.
@@ -844,12 +1267,12 @@ human readable format before and after the whole backup process:
-
-source (a text file containing the rsync compatible path to backup)
+source (a text file containing the rsync compatible path to backup)
-
-destination (a text file containing the directory we should backup to)
+destination (a text file containing the directory we should backup to)
@@ -857,69 +1280,69 @@ human readable format before and after the whole backup process:
-
-pre_exec program to execute before backing up this source
+pre_exec program to execute before backing up this source
-
-post_exec program to execute after backing up this source
+post_exec program to execute after backing up this source
-
-verbose whether to be verbose (passes -v to rsync)
+verbose whether to be verbose (passes -v to rsync)
-
-very_verbose be very verbose (mkdir -v, rm -v and rsync -vv)
+very_verbose be very verbose (mkdir -v, rm -v and rsync -vv)
-
-summary create a transfer summary when rsync finished
+summary create a transfer summary when rsync finished
-
-exclude exclude list for rsync. newline seperated list.
+exclude exclude list for rsync. newline seperated list.
-
-rsync_options extra options for rsync. newline seperated list.
+rsync_options extra options for rsync. newline seperated list.
-
-delete_incomplete delete incomplete backups
+delete_incomplete delete incomplete backups
-
-remote_host host to backup to
+remote_host host to backup to
-
-rsync_failure_codes list of rsync exit codes that indicate complete failure
+rsync_failure_codes list of rsync exit codes that indicate complete failure
-
-mtime Sort backup directories based on their modification time
+mtime Sort backup directories based on their modification time
-
-quiet_if_down Suppress error messages if source is not connectable
+quiet_if_down Suppress error messages if source is not connectable
Example:
- [10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2
+ [10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2
insgesamt 12
lrwxrwxrwx 1 nico users 20 2005-11-17 16:44 destination
-rw-r--r-- 1 nico users 62 2005-12-07 17:43 exclude
@@ -935,19 +1358,20 @@ human readable format before and after the whole backup process:
[10:48] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/intervals/daily
5
[10:48] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/source
- /home/nico/vpn
+ /home/nico/vpn
+
5.3.1. Default options
-If you add $CCOLLECT_CONF/defaults/option_name, the value will
+
If you add $CCOLLECT_CONF/defaults/option_name, the value will
be used in abscence of the option in a source. If you want to prevent
the default value to be used in a source, you can create the file
-$CCOLLECT_CONF/sources/$name/no_option_name (i.e. prefix it with
+$CCOLLECT_CONF/sources/$name/no_option_name (i.e. prefix it with
no_.
Example:
- [9:04] ikn2:ccollect% touch conf/defaults/verbose
- [9:04] ikn2:ccollect% touch conf/sources/local/no_verbose
+ [9:04] ikn2:ccollect% touch conf/defaults/verbose
+ [9:04] ikn2:ccollect% touch conf/sources/local/no_verbose
This enables the verbose option for all sources, but disables it for the
source local.
@@ -956,86 +1380,98 @@ the source specific version overrides the default one:
Example:
- [9:05] ikn2:ccollect% echo "backup-host" > conf/defaults/remote_host
- [9:05] ikn2:ccollect% echo "different-host" > conf/sources/local/remote_host
+ [9:05] ikn2:ccollect% echo "backup-host" > conf/defaults/remote_host
+ [9:05] ikn2:ccollect% echo "different-host" > conf/sources/local/remote_host
You can use all source options as defaults, with the exception of
-
-source
+source
-
-destination
+destination
-
-pre_exec
+pre_exec
-
-post_exec
+post_exec
+
+
5.3.2. Detailed description of "source"
-source describes a rsync compatible source (one line only).
+source describes a rsync compatible source (one line only).
For instance backup_user@foreign_host:/home/server/video.
-To use the rsync protocol without the ssh-tunnel, use
+To use the rsync protocol without the ssh-tunnel, use
rsync::USER@HOST/SRC. For more information have a look at the manpage
-of rsync(1).
+of rsync(1).
+
+
5.3.3. Detailed description of "destination"
-destination must be a text file containing the destination directory.
-destination USED to be a link to the destination directory in
+
destination must be a text file containing the destination directory.
+destination USED to be a link to the destination directory in
earlier versions, so do not be confused if you see such examples.
Example:
- [11:36] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/destination
- /home/nico/backupdir
+ [11:36] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/destination
+ /home/nico/backupdir
+
+
5.3.4. Detailed description of "remote_host"
-remote_host must be a text file containing the destination host.
+
remote_host must be a text file containing the destination host.
If this file is existing, you are backing up your data TO this host
and not to you local host.
-Warning: You need to have ssh access to the remote host. rsync and
-ccollect will connect to that host via ssh. ccollect needs the shell
+
Warning: You need to have ssh access to the remote host. rsync and
+ccollect will connect to that host via ssh. ccollect needs the shell
access, because it needs to find out how many backups exist on the remote
host and to be able to delete them.
Example:
- [10:17] denkbrett:ccollect-0.7.0% cat conf/sources/remote1/remote_host
- home.schottelius.org
+ [10:17] denkbrett:ccollect-0.7.0% cat conf/sources/remote1/remote_host
+ home.schottelius.org
It may contain all the ssh-specific values like myuser@yourhost.ch.
+
+
5.3.5. Detailed description of "verbose"
-verbose tells ccollect that the log should contain verbose messages.
-If this file exists in the source specification -v will be passed to rsync.
+verbose tells ccollect that the log should contain verbose messages.
+If this file exists in the source specification -v will be passed to rsync.
Example:
- [11:35] zaphodbeeblebrox:ccollect-0.2% touch conf/sources/testsource1/verbose
+ [11:35] zaphodbeeblebrox:ccollect-0.2% touch conf/sources/testsource1/verbose
+
+
5.3.6. Detailed description of "very_verbose"
-very_verbose tells ccollect that it should log very verbosely.
+very_verbose tells ccollect that it should log very verbosely.
If this file exists in the source specification -v will be passed to
-rsync, rm and mkdir.
+rsync, rm and mkdir.
Example:
- [23:67] nohost:~% touch conf/sources/testsource1/very_verbose
+ [23:67] nohost:~% touch conf/sources/testsource1/very_verbose
+
+
5.3.7. Detailed description of "summary"
-If you create the file summary in the source definition,
-ccollect will present you a nice summary at the end.
+If you create the file summary in the source definition,
+ccollect will present you a nice summary at the end.
-backup:~# touch /etc/ccollect/sources/root/summary
+backup:~# touch /etc/ccollect/sources/root/summary
backup:~# ccollect.sh werktags root
==> ccollect.sh: Beginning backup using interval werktags <==
[root] Beginning to backup this source ...
@@ -1059,39 +1495,45 @@ backup:~# ccollect.sh werktags root
[root] sent 13009119 bytes received 2152 bytes 2891393.56 bytes/sec
[root] total size is 26234080536 speedup is 2016.26
[root] Successfully finished backup.
-==> Finished ccollect.sh <==
+==> Finished ccollect.sh <==
-You could also combine it with verbose or very_verbose, but these
+
You could also combine it with verbose or very_verbose, but these
already print some statistics (though not all / the same as presented by
-summary).
+summary).
+
+
5.3.8. Detailed description of "exclude"
-exclude specifies a list of paths to exclude. The entries are seperated by a newline (\n).
+exclude specifies a list of paths to exclude. The entries are seperated by a newline (\n).
Example:
- [11:35] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
+ [11:35] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli
- something with spaces is not a problem
+ something with spaces is not a problem
+
+
5.3.9. Detailed description of "intervals/"
-When you create the subdirectory intervals/ in your source configuration
+
When you create the subdirectory intervals/ in your source configuration
directory, you can specify individiual intervals for this specific source.
Each file in this directory describes an interval.
Example:
- [11:37] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervals/
+ [11:37] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervals/
insgesamt 8
-rw-r--r-- 1 nico users 2 2005-12-07 17:38 daily
-rw-r--r-- 1 nico users 3 2005-12-14 11:33 yearly
[11:37] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/intervals/*
5
- 20
+ 20
+
+
5.3.10. Detailled description of "rsync_options"
-When you create the file rsync_options in your source configuration,
+
When you create the file rsync_options in your source configuration,
all the parameters in this file will be passed to rsync. This
way you can pass additional options to rsync. For instance you can tell rsync
to show progress ("--progress"), or which -password-file ("--password-file")
@@ -1099,18 +1541,20 @@ to use for automatic backup over the rsync-protocol.
Example:
- [23:42] hydrogenium:ccollect-0.2% cat conf/sources/test_rsync/rsync_options
- --password-file=/home/user/backup/protected_password_file
+ [23:42] hydrogenium:ccollect-0.2% cat conf/sources/test_rsync/rsync_options
+ --password-file=/home/user/backup/protected_password_file
+
+
5.3.11. Detailled description of "pre_exec" and "post_exec"
-When you create pre_exec and / or post_exec in your source
-configuration, ccollect will execute this command before and
+
When you create pre_exec and / or post_exec in your source
+configuration, ccollect will execute this command before and
respectively after doing the backup for this specific source.
If you want to have pre-/post-exec before and after all
backups, see above for general configuration.
-If pre_exec exits with a non-zero return code, the backup
-process of this source will be aborted (i.e. backup skipped).
-The post_exec script can access the following exported variables from
+
If pre_exec exits with a non-zero return code, the backup
+process of this source will be aborted (i.e. backup skipped).
+The post_exec script can access the following exported variables from
ccollect:
-
@@ -1120,24 +1564,24 @@ ccollect:
-destination_name: contains the base directory name (daily.20091031-1013.24496)
+destination_name: contains the base directory name (daily.20091031-1013.24496)
-destination_dir: full path (/tmp/ccollect/daily.20091031-1013.24496)
+destination_dir: full path (/tmp/ccollect/daily.20091031-1013.24496)
-destination_full: like destination_dir, but prepended with the remote_host, if set (host:/tmp/ccollect/daily.20091031-1013.24496 or /tmp/ccollect/daily.20091031-1013.24496)
+destination_full: like destination_dir, but prepended with the remote_host, if set (host:/tmp/ccollect/daily.20091031-1013.24496 or /tmp/ccollect/daily.20091031-1013.24496)
Example:
-[13:09] hydrogenium:ccollect-0.3% cat conf/sources/with_exec/pre_exec
+[13:09] hydrogenium:ccollect-0.3% cat conf/sources/with_exec/pre_exec
#!/bin/sh
# Show whats free before
@@ -1146,30 +1590,38 @@ df -h
#!/bin/sh
# Show whats free after
-df -h
+df -h
+
+
5.3.12. Detailed description of "delete_incomplete"
-If you create the file delete_incomplete in a source specification directory,
-ccollect will look for incomplete backups (when the whole ccollect process
-was interrupted) and remove them. Without this file ccollect will only warn
+
If you create the file delete_incomplete in a source specification directory,
+ccollect will look for incomplete backups (when the whole ccollect process
+was interrupted) and remove them. Without this file ccollect will only warn
the user.
+
+
5.3.13. Detailed description of "rsync_failure_codes"
-If you have the file rsync_failure_codes in your source configuration
+
If you have the file rsync_failure_codes in your source configuration
directory, it should contain a newline-separated list of numbers representing
-rsync exit codes. If rsync exits with any code in this list, a marker will
+rsync exit codes. If rsync exits with any code in this list, a marker will
be left in the destination directory indicating failure of this backup. If
you have enabled delete_incomplete, then this backup will be deleted during
the next ccollect run on the same interval.
+
+
5.3.14. Detailed description of "mtime"
By default, ccollect.sh chooses the most recent backup directory for cloning or
the oldest for deletion based on the directory’s last change time (ctime).
-With this option, the sorting is done based on modification time (mtime). With
+With this option, the sorting is done based on modification time (mtime). With
this version of ccollect, the ctime and mtime of your backups will normally
be the same and this option has no effect. However, if you, for example, move
your backups to another hard disk using cp -a or rsync -a, you should use this
option because the ctimes are not preserved during such operations.
If you have any backups in your repository made with ccollect version 0.7.1 or
earlier, do not use this option.
+
+
5.3.15. Detailed description of "quiet_if_down"
By default, ccollect.sh emits a series of error messages if a source is not
connectable. With this option enabled, ccollect still reports that the
@@ -1177,9 +1629,14 @@ source is not connectable but the associated error messages generated by
rsync or ssh are suppressed. You may want to use this option for sources,
like notebook PCs, that are often disconnected.
+
+
+
+
6. Hints
-6.1. Smart logging
+
+6.1. Smart logging
Since ccollect-0.6.1 you can use the ccollect-logwrapper.sh(1) for logging.
You call it the same way you call ccollect.sh and it will create a
logfile containing the output of ccollect.sh. For more information look
@@ -1187,23 +1644,25 @@ at the manpage ccollect-logwrapper. The following is an example running
ccollect-logwrapper.sh:
-u0219 ~ # ~chdscni9/ccollect-logwrapper.sh daily u0160.nshq.ch.netstream.com
+u0219 ~ # ~chdscni9/ccollect-logwrapper.sh daily u0160.nshq.ch.netstream.com
ccollect-logwrapper.sh (11722): Starting with arguments: daily u0160.nshq.ch.netstream.com
-ccollect-logwrapper.sh (11722): Finished.
+ccollect-logwrapper.sh (11722): Finished.
-6.2. Using a different ssh port
+
+
+6.2. Using a different ssh port
Mostly easy is to use your ~/.ssh/config file:
-host mx2.schottelius.org
- Port 2342
+host mx2.schottelius.org
+ Port 2342
If you only use that port for backup only and normally want to use another port,
you can add HostName and "HostKeyAlias" (if you also have different
keys on the different ports):
-Host hhydrogenium
+Host hhydrogenium
Hostname bruehe.schottelius.org
Port 666
HostKeyAlias hydrogenium
@@ -1211,10 +1670,12 @@ keys on the different ports):
Host bruehe
Hostname bruehe.schottelius.org
Port 22
- HostKeyAlias bruehe.schottelius.org
+ HostKeyAlias bruehe.schottelius.org
-6.3. Using source names or interval in pre_/post_exec scripts
-The pre-/post_exec scripts can access some internal variables from ccollect:
+
+
+6.3. Using source names or interval in pre_/post_exec scripts
+The pre-/post_exec scripts can access some internal variables from ccollect:
-
@@ -1238,7 +1699,7 @@ Host bruehe
-Only available for post_exec:
+Only available for post_exec:
-
@@ -1246,28 +1707,32 @@ Host bruehe
-6.4. Using rsync protocol without ssh
+
+
+6.4. Using rsync protocol without ssh
When you have a computer with little computing power, it may be useful to use
rsync without ssh, directly using the rsync protocol
-(specify user@host::share in source). You may wish to use
-rsync_options to specify a password file to use for automatic backup.
+(specify user@host::share in source). You may wish to use
+rsync_options to specify a password file to use for automatic backup.
Example:
-backup:~# cat /etc/ccollect/sources/sample.backup.host.org/source
+backup:~# cat /etc/ccollect/sources/sample.backup.host.org/source
backup@webserver::backup-share
backup:~# cat /etc/ccollect/sources/sample.backup.host.org/rsync_options
--password-file=/etc/ccollect/sources/sample.backup.host.org/rsync_password
backup:~# cat /etc/ccollect/sources/sample.backup.host.org/rsync_password
-this_is_the_rsync_password
+this_is_the_rsync_password
This hint was reported by Daniel Aubry.
-6.5. Not excluding top-level directories
+
+
+6.5. Not excluding top-level directories
When you exclude "/proc" or "/mnt" from your backup, you may run into
trouble when you restore your backup. When you use "/proc/*" or "/mnt/\*"
-instead, ccollect will backup empty directories.
+instead, ccollect will backup empty directories.
@@ -1281,15 +1746,17 @@ they will still be transferred!
This hint was reported by Marcus Wagner.
-6.6. Re-using already created rsync-backups
-If you used rsync directly before you use ccollect, you can
-use this old backup as initial backup for ccollect: You
+
+
+6.6. Re-using already created rsync-backups
+If you used rsync directly before you use ccollect, you can
+use this old backup as initial backup for ccollect: You
simply move it into a directory below the destination directory
and name it "interval.0".
Example:
-backup:/home/backup/web1# ls
+backup:/home/backup/web1# ls
bin dev etc initrd lost+found mnt root srv usr vmlinuz
boot doc home lib media opt sbin tmp var vmlinuz.old
@@ -1299,17 +1766,17 @@ backup:/home/backup/web1# mkdir daily.0
backup:/home/backup/web1# mv * daily.0 2>/dev/null
backup:/home/backup/web1# ls
-daily.0
+daily.0
-Now you can use /home/backup/web1 as the destination for the backup.
+Now you can use /home/backup/web1 as the destination for the backup.
Note
-It does not matter anymore how you name your directory, as ccollect uses
-the -c option from ls to find out which directory to clone from.
+It does not matter anymore how you name your directory, as ccollect uses
+the -c option from ls to find out which directory to clone from.
@@ -1321,18 +1788,20 @@ the -c option from ls to find out which directory to clone from.
Older versions (pre 0.6, iirc) had a problem, if you named the first backup
something like "daily.initial". It was needed to use the "0" (or some
-number that is lower than the current year) as extension. ccollect
-used sort to find the latest backup. ccollect itself uses
+number that is lower than the current year) as extension. ccollect
+used sort to find the latest backup. ccollect itself uses
interval.YEARMONTHDAY-HOURMINUTE.PID. This notation was always before
"daily.initial", as numbers are earlier in the list
-which is produced by sort. So, if you had a directory named "daily.initial",
-ccollect always diffed against this backup and transfered and deleted
+which is produced by sort. So, if you had a directory named "daily.initial",
+ccollect always diffed against this backup and transfered and deleted
files which where deleted in previous backups. This means you simply
wasted resources, but your backup had beer complete anyway.
-6.7. Using pre_/post_exec
+
+
+6.7. Using pre_/post_exec
Your pre_/post_exec script does not need to be a script, you can also
use a link to
@@ -1348,51 +1817,63 @@ an already written script
The only requirement is that it is executable.
-6.8. Using source specific interval definitions
+
+
+6.8. Using source specific interval definitions
When you are backing up multiple hosts via cron each night, it may be
a problem that host "big_server" may only have 4 daily backups, because
otherwise its backup device will be full. But for all other hosts
you want to keep 20 daily backups. In this case you would create
-/etc/ccollect/default/intervals/daily containing "20" and
-/etc/ccollect/sources/big_server/intervals/daily containing "4".
+/etc/ccollect/default/intervals/daily containing "20" and
+/etc/ccollect/sources/big_server/intervals/daily containing "4".
Source specific intervals always overwrite the default values.
If you have to specify it individually for every host, because
of different requirements, you can even omit creating
-/etc/ccollect/default/intervals/daily.
-6.9. Comparing backups
+/etc/ccollect/default/intervals/daily.
+
+
+6.9. Comparing backups
If you want to see what changed between two backups, you can use
-rsync directly:
+rsync directly:
-[12:00] u0255:ddba034.netstream.ch# rsync -n -a --delete --stats --progress daily.20080324-0313.17841/ daily.20080325-0313.31148/
+[12:00] u0255:ddba034.netstream.ch# rsync -n -a --delete --stats --progress daily.20080324-0313.17841/ daily.20080325-0313.31148/
This results in a listing of changes. Because we pass -n to rsync no transfer
is made (i.e. report only mode).
This hint was reported by Daniel Aubry.
-6.10. Testing for host reachabilty
+
+
+6.10. Testing for host reachabilty
If you want to test whether the host you try to backup is reachable, you can use
the following script as source specific pre-exec:
-#!/bin/sh
-# ping -c1 -q `cat "/etc/ccollect/sources/$name/source" | cut -d"@" -f2 | cut -d":" -f1`
+#!/bin/sh
+# ping -c1 -q `cat "/etc/ccollect/sources/$name/source" | cut -d"@" -f2 | cut -d":" -f1`
This prevents the deletion of old backups, if the host is not reachable.
This hint was reported by Daniel Aubry.
-6.11. Easy check for errors
+
+
+6.11. Easy check for errors
If you want to see whether there have been any errors while doing the backup,
-you can run ccollect together with ccollect_analyse_logs.sh:
+you can run ccollect together with ccollect_analyse_logs.sh:
-$ ccollect | ccollect_analyse_logs.sh e
+$ ccollect | ccollect_analyse_logs.sh e
+
+Let us assume that one backup failed (connection broke or the source hard disk had some failures). Therefore we’ve got one incomplete backup in our history.
ccollect will transfer the missing files the next time you use it. +
ccollect will transfer the missing files the next time you use it.
This leads to
If the whole ccollect process was interrupted, ccollect (since 0.6) can +
If the whole ccollect process was interrupted, ccollect (since 0.6) can
detect that and remove the incomplete backups, so you can clone from a complete
backup instead
No. ccollect passes your source definition directly to rsync. It +
No. ccollect passes your source definition directly to rsync. It
does not try to analyze it. So it actually does not know if a source
comes from local harddisk or from a remote server. And it does not want
to. When you backup from the local harddisk (which is perhaps not
-even a good idea when thinking of security), add the destination
+even a good idea when thinking of security), add the destination
to source/exclude. (Daniel Aubry reported this problem)
The most common error is that you have not given your script the correct -permissions. Try chmod 0755 /etc/ccollect/sources/'yoursource'/*_exec`.
chmod 0755 /etc/ccollect/sources/'yoursource'/*_exec`.When a part of your path you specified in the source is a (symbolic, hard links are not possible for directories) link, the backup must fail.
First of all, let us have a look at how it looks like:
==> ccollect 0.4: Beginning backup using interval taeglich <== +==> ccollect 0.4: Beginning backup using interval taeglich <== [testsource] Sa Apr 29 00:01:55 CEST 2006 Beginning to backup [testsource] Currently 0 backup(s) exist(s), total keeping 10 backup(s). [testsource] Beginning to backup, this may take some time... @@ -1434,40 +1921,69 @@ the backup must fail.
So what is the problem? It is very obvious when you look deeper into it:
% cat /etc/ccollect/sources/testsource/source ++lrwxrwxrwx 1 nico nico 29 2006-04-29 00:01 projekte -> oeffentlich/computer/projekte% cat /etc/ccollect/sources/testsource/source /home/user/nico/projekte/ccollect/ccollect-0.4 % ls -l /home/user/nico/projekte lrwxrwxrwx 1 nico nico 29 2005-12-02 23:28 /home/user/nico/projekte -> oeffentlich/computer/projekte % ls -l /etc/ccollect/sources/testsource/destination/taeglich.2006-04-29-0001.3874/home/user/nico -lrwxrwxrwx 1 nico nico 29 2006-04-29 00:01 projekte -> oeffentlich/computer/projekte
rsync creates the directory structure before it creates the symbolic link. +
rsync creates the directory structure before it creates the symbolic link.
This link now links to something not reachable (dead link). It is
impossible to create subdirectories under the broken link.
In conclusion you cannot use paths with a linked part.
However, you can backup directories containing symbolic links (in this case you could backup /home/user/nico, which contains /home/user/nico/projekte and oeffentlich/computer/projekte).
As ccollect first deletes the old backups, it may take some time -until rsync requests the password for the ssh session from you.
The easiest way not to miss that point is running ccollect in screen, +
As ccollect first deletes the old backups, it may take some time
+until rsync requests the password for the ssh session from you.
The easiest way not to miss that point is running ccollect in screen,
which has the ability to monitor the output for activity. So as soon as
-your screen beeps, after ccollect began to remove the last directory,
+your screen beeps, after ccollect began to remove the last directory,
you can enter your password (have a look at screen(1), especially "C-a M"
and "C-a _", for more information).
If you are trying to backup a system containing paths that are managed +by autofs, you may run into this error:
srwali01:~# mkdir /etc/ccollect ++2009-12-01-23:14:15: ccollect 0.8.1: Beginning backup using interval monatlich +[ikn] 2009-12-01-23:14:15: Beginning to backup +[ikn] 2009-12-01-23:14:15: Executing /home/users/nico/ethz/ccollect/sources/ikn/pre_exec ... +Enter LUKS passphrase: +[ikn] Command successful. +[ikn] key slot 0 unlocked. +[ikn] 2009-12-01-23:14:23: Finished /home/users/nico/ethz/ccollect/sources/ikn/pre_exec (return code 0). [ikn] directory has vanished: "/home/users/nico/privat/firmen/ethz/autofs/projects" +[ikn] directory has vanished: "/home/users/nico/privat/firmen/ethz/autofs/scratch" +[ikn] directory has vanished: "/home/users/nico/privat/firmen/ethz/autofs/sgscratch" +[ikn] directory has vanished: "/home/users/nico/privat/firmen/ethz/autofs/supp" +[ikn] directory has vanished: "/home/users/nico/privat/firmen/ethz/autofs/sysadmin" +[ikn] rsync warning: some files vanished before they could be transferred (code 24) at main.c(1057) [sender=3.0.6] +[ikn] 2009-12-01-23:44:23: Source / is not readable. Skipping.
Thus, if you are unsure whether autofs paths can be mounted during backup, +stop autofs in pre_exec and reenable it in post_exec.
srwali01:~# mkdir /etc/ccollect
srwali01:~# mkdir -p /etc/ccollect/defaults/intervals/
srwali01:~# echo 28 > /etc/ccollect/defaults/intervals/taeglich
srwali01:~# echo 52 > /etc/ccollect/defaults/intervals/woechentlich
@@ -1488,12 +2004,12 @@ srwali01:/etc/ccollect/sources/local-root# mkdir /mnt/hdbackup/local-root
srwali01:/etc/ccollect/sources/local-root# ccollect.sh taeglich local-root
/o> ccollect.sh: Beginning backup using interval taeglich
/=> Beginning to backup "local-root" ...
-|-> 0 backup(s) already exist, keeping 28 backup(s).After that, I added some more sources:
srwali01:~# cd /etc/ccollect/sources ++srwali01:/etc/ccollect/sources/srwali03# mkdir /mnt/hdbackup/srwali03srwali01:~# cd /etc/ccollect/sources srwali01:/etc/ccollect/sources# mkdir windos-wl6 srwali01:/etc/ccollect/sources# cd windos-wl6/ srwali01:/etc/ccollect/sources/windos-wl6# echo /mnt/win/SYS/WL6 > source @@ -1517,12 +2033,14 @@ srwali01:/etc/ccollect/sources/srwali03# cat > exclude << EOF > EOF srwali01:/etc/ccollect/sources/srwali03# echo 'root@10.103.2.3:/' > source srwali01:/etc/ccollect/sources/srwali03# echo /mnt/hdbackup/srwali03 > destination -srwali01:/etc/ccollect/sources/srwali03# mkdir /mnt/hdbackup/srwali03
# du (coreutils) 5.2.1 ++147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:45.588# du (coreutils) 5.2.1 [10:53] srsyg01:sources% du -sh ~/backupdir 4.6M /home/nico/backupdir [10:53] srsyg01:sources% du -sh ~/backupdir/* @@ -1537,12 +2055,12 @@ srwali01:/etc/ccollect/sources# du -sh /mnt/hdbackup/wl6/ 186M /mnt/hdbackup/wl6/ srwali01:/etc/ccollect/sources# du -sh /mnt/hdbackup/wl6/* 147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:42.312 -147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:45.588
The backup of our main fileserver:
backup:~# df -h /home/backup/srsyg01/ ++du (coreutils) 5.2.1backup:~# df -h /home/backup/srsyg01/ Filesystem Size Used Avail Use% Mounted on /dev/mapper/backup--01-srsyg01 591G 451G 111G 81% /home/backup/srsyg01 @@ -1559,13 +2077,13 @@ backup:~# du -sh /home/backup/srsyg01/* 439G /home/backup/srsyg01/daily.2006-02-03-01:00.22567 16K /home/backup/srsyg01/lost+found backup:~# du --version | head -n1 -du (coreutils) 5.2.1
Newer versions of du also detect the hardlinks, so we can even compare the sizes directly with du:
[8:16] eiche:~# du --version | head -n 1 ++200M hydrogenium/durcheinander.2006-01-26-00:11.3332[8:16] eiche:~# du --version | head -n 1 du (GNU coreutils) 5.93 [8:17] eiche:schwarzesloch# du -slh hydrogenium/* 19G hydrogenium/durcheinander.0 @@ -1576,16 +2094,18 @@ du (GNU coreutils) 5.93 19G hydrogenium/durcheinander.0 12G hydrogenium/durcheinander.2006-01-17-00:27.13820 1.5G hydrogenium/durcheinander.2006-01-25-23:18.31328 -200M hydrogenium/durcheinander.2006-01-26-00:11.3332
In the second report (without -l) the sizes include the space the inodes of the hardlinks allocate.
All the data of my important hosts is backuped to eiche into /mnt/schwarzesloch/backup:
[9:24] eiche:backup# ls * ++woechentlich.2006-02-12-11:51.2514 woechentlich.2006-03-16-23:00.4218[9:24] eiche:backup# ls * creme: woechentlich.2006-01-26-22:22.4153 woechentlich.2006-02-12-11:48.2461 woechentlich.2006-01-26-22:23.4180 woechentlich.2006-02-18-23:00.7898 @@ -1604,13 +2124,13 @@ woechentlich.2006-02-04-10:32.13766 woechentlich.2006-02-16-23:00.6185 woechentlich.2006-02-05-23:02.15093 woechentlich.2006-02-23-23:00.11783 woechentlich.2006-02-06-08:22.15994 woechentlich.2006-03-02-23:00.17346 woechentlich.2006-02-06-19:40.16321 woechentlich.2006-03-09-23:00.29317 -woechentlich.2006-02-12-11:51.2514 woechentlich.2006-03-16-23:00.4218
And this incremental backup and the archive are copied to an external usb harddisk (attention: you should really use -H to backup the backup):
[9:23] eiche:backup# df -h ++rsync -av -H --delete /mnt/archiv/ "$DDIR/archiv/"[9:23] eiche:backup# df -h Filesystem Size Used Avail Use% Mounted on rootfs 14G 8.2G 4.9G 63% / /dev/root 14G 8.2G 4.9G 63% / @@ -1629,13 +2149,15 @@ DDIR=/mnt/usb/backup rsync -av -H --delete /mnt/schwarzesloch/ "$DDIR/schwarzesloch/" -rsync -av -H --delete /mnt/archiv/ "$DDIR/archiv/"
Truncated output from ps axuwwwf:
Truncated output from ps axuwwwf:
S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily -p ddba034 ddba045 ddba046 ddba047 ddba049 ddna010 ddna011 +S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddna011 R+ 12:08 0:00 | | | | \_ rm -rf /etc/ccollect/sources/ddna011/destination/daily.2006-10-20-1502.7824 S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddna011 - S+ 11:40 0:00 | | | \_ sed s:^:\[ddna011\] : + S+ 11:40 0:00 | | | \_ sed s:^:\[ddna011\] :S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily -j ddba034 ddba045 ddba046 ddba047 ddba049 ddna010 ddna011 S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba034 S+ 11:40 0:00 | | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba034 R+ 11:40 23:40 | | | | | \_ rsync -a --delete --numeric-ids --relative --delete-excluded --link-dest=/home/server/backup/ddba034 @@ -1672,15 +2194,19 @@ rsync -av -H --delete /mnt/archiv/ "$DDIR/archiv/"
As you can see, six processes are deleting old backups, while one backup (ddba034) is already copying data.
ccollect is a backup utility written in the sh-scripting language.
+
ccollect is a backup utility written in the sh-scripting language.
It does not depend on a specific shell, only /bin/sh needs to be
bourne shell compatibel (like dash, ksh, zsh, bash, …).
For more information refer to the manual titled "ccollect - Installing, Configuring and Using" (available as text (asciidoc), texinfo or html).
ccollect - (pseudo) incremental backup with different exclude lists using hardlinks and rsync
ccollect.sh [args] <interval name> <sources to backup>
ccollect is a backup utility written in the sh-scripting language. -It does not depend on a specific shell, only /bin/sh needs to be +
ccollect is a backup utility written in the sh-scripting language.
+It does not depend on a specific shell, only /bin/sh needs to be
bourne shell compatibel (like dash, ksh, zsh, bash, …).
For more information refer to the manual titled "ccollect - Installing, Configuring and Using" (available as text (asciidoc), texinfo or html).
+ Backup all sources specified in /etc/ccollect/sources +
++ Specifies the number of jobs to run simultaneously. + If max is not specified then parallelise all jobs. +
+- Parallelise backup processes -
-- Backup all sources specified in /etc/ccollect/sources + Parallelise backup processes (deprecated from 2.0)
ccollect_add_source(1), ccollect_analyse_logs(1), ccollect_logwrapper(1) ccollect_delete_source(1), ccollect_list_intervals(1)
Nico Schottelius <nico-ccollect--@--schottelius.org>
Main web site: http://www.nico.schottelius.org/software/ccollect/
Copyright (C) 2006-2008 Nico Schottelius. Free use of this software is granted under the terms of the GNU General Public License Version 3 (GPLv3).
ccollect_add_source.sh creates a new backup source for use with ccollect(1). +
ccollect_add_source.sh creates a new backup source for use with ccollect(1). It copies the files from to the source directory with the hostname below $CCOLLECT_CONF/sources. It is designed to run on a backup server to create new directories for new hosts.
ccollect_add_source - create new source for ccollect(1)
ccollect_add_source.sh <hostnames to create sources for>
ccollect_add_source.sh creates a new backup source for use with ccollect(1). @@ -403,6 +758,8 @@ It copies the files from to the source directory with the hostname below $CCOLLECT_CONF/sources. It is designed to run on a backup server to create new directories for new hosts.
A link to the directory where to store the backups. Below this directory
- ccollect_add_source.sh will create a directory with the hostname you
- specified on the command line. A common valua for destination_base is
+ ccollect_add_source.sh will create a directory with the hostname you
+ specified on the command line. A common valua for destination_base is
/home/server/backup.
- source_prefix is put before the hostname, source_postfix is appended
- after it. A common value for source_prefix maybe root@ and :/
- for source_postfix.
+ source_prefix is put before the hostname, source_postfix is appended
+ after it. A common value for source_prefix maybe root@ and :/
+ for source_postfix.
ccollect(1), ccollect_analyse_logs.sh, ccollect_delete_source(1), ccollect_list_intervals(1), ccollect_logwrapper(1),
Nico Schottelius <nico-ccollect--@--schottelius.org>
Main web site: http://www.nico.schottelius.org/software/ccollect/
Copyright (C) 2007-2008 Nico Schottelius. Free use of this software is granted under the terms of the GNU General Public License Version 3 (GPLv3).
ccollect_analyse_logs.sh reads the logfiles from stdin. You have to specify +
ccollect_analyse_logs.sh reads the logfiles from stdin. You have to specify at least one of the three loglevels (*i*nformational, *w*arning, *e*rror). Any combination of them is allowed.
ccollect_analyse_logs - analyse logs produced by ccollect(1)
ccollect_analyse_logs.sh [iwe]
ccollect_analyse_logs.sh reads the logfiles from stdin. You have to specify at least one of the three loglevels (*i*nformational, *w*arning, *e*rror). Any combination of them is allowed.
ccollect(1), ccollect_add_source.sh, ccollect_delete_source(1), ccollect_list_intervals(1), ccollect_logwrapper(1),
Nico Schottelius <nico-ccollect--@--schottelius.org>
Main web site: http://www.nico.schottelius.org/software/ccollect/
Copyright (C) 2007-2008 Nico Schottelius. Free use of this software is granted under the terms of the GNU General Public License Version 3 (GPLv3).
ccollect_delete_source.sh deletes backup sources from ccollect(1) and optional +
ccollect_delete_source.sh deletes backup sources from ccollect(1) and optional also the backups created for that source.
-d:
Delete also the destination directory. add_ccollect_source.sh will change
to the source/name/destination directory, get the absolute name and delete
diff --git a/software/ccollect/documentation/man/ccollect_delete_source.html b/software/ccollect/documentation/man/ccollect_delete_source.html
index a66b04ab..0bdfcd66 100644
--- a/software/ccollect/documentation/man/ccollect_delete_source.html
+++ b/software/ccollect/documentation/man/ccollect_delete_source.html
@@ -2,15 +2,25 @@
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
ccollect_delete_source - delete sources from ccollect(1)
ccollect_delete_source.sh [-d] [-f] <hostnames to create sources for>
ccollect_delete_source.sh deletes backup sources from ccollect(1) and optional also the backups created for that source.
-d:
- Delete also the destination directory. add_ccollect_source.sh will change
+ Delete also the destination directory. add_ccollect_source.sh will change
to the source/name/destination directory, get the absolute name and delete
it recursively.
-f:
Force deletion. Do not ask. Very handy for people who know what they do.
Very dangerous for everyone else.
ccollect(1), ccollect_add_source(1), ccollect_add_source(1), ccollect_logwrapper(1), ccollect_list_intervals(1)
Nico Schottelius <nico-ccollect--@--schottelius.org>
Main web site: http://www.nico.schottelius.org/software/ccollect/
Copyright (C) 2007-2008 Nico Schottelius. Free use of this software is granted under the terms of the GNU General Public License Version 3 (GPLv3).
ccollect_list_intervals.sh shows intervals specified in the configuration +
ccollect_list_intervals.sh shows intervals specified in the configuration for ccollect(1). It displays the name of each interval, followed by a colon followed by the number backups to keep.
<nico-ccollect--@--schottelius.org>ccollect_list_intervals - list available intervals from ccollect(1)
ccollect_list_intervals.sh
ccollect_list_intervals.sh shows intervals specified in the configuration for ccollect(1). It displays the name of each interval, followed by a colon followed by the number backups to keep.
collect(1), ccollect_add_source(1), ccollect_analyse_logs(1), ccollect_delete_source(1), ccollect_logwrapper(1)
Nico Schottelius <nico-ccollect--@--schottelius.org>
Main web site: http://www.nico.schottelius.org/software/ccollect/
Copyright (C) 2007-2008 Nico Schottelius. Free use of this software is granted under the terms of the GNU General Public License Version 3 (GPLv3).
ccollect_logwrapper.sh creates a unique logfile below +
ccollect_logwrapper.sh creates a unique logfile below $CCOLLECT_CONF/logwrapper and redirects ccollect(1) output (stdout and stderr) to it.
ccollect_logwrapper - start ccollect(1) and create a unique logfile
ccollect_logwrapper.sh <ccollect options>
ccollect_logwrapper.sh creates a unique logfile below $CCOLLECT_CONF/logwrapper and redirects ccollect(1) output (stdout and stderr) to it.
Options are passed directly to ccollect(1).
ccollect(1), ccollect_add_source(1), ccollect_analyse_logs(1), ccollect_delete_source(1), ccollect_list_intervals(1)
Nico Schottelius <nico-ccollect--@--schottelius.org>
Main web site: http://www.nico.schottelius.org/software/ccollect/
Copyright (C) 2007-2008 Nico Schottelius. Free use of this software is granted under the terms of the GNU General Public License Version 3 (GPLv3).