ccollect/doc/ccollect.text

676 lines
27 KiB
Text
Raw Normal View History

2006-01-13 11:36:20 +00:00
ccollect - Installing, Configuring and Using
============================================
Nico Schottelius <nico-linux-ccollect__@__schottelius.org>
0.4.0, for ccollect 0.4, Initial Version from 2006-01-13
2006-01-13 11:36:20 +00:00
:Author Initials: NS
(pseudo) incremental backup
with different exclude lists
using hardlinks and `rsync`
Introduction
------------
ccollect is a backup utitily written in the sh-scripting language.
It does not depend on a specific shell, only `/bin/sh` needs to be
2006-01-22 12:27:12 +00:00
bourne shell compatibel (like 'dash', 'ksh', 'zsh', 'bash', ...).
2006-01-13 11:36:20 +00:00
Why you can only backup TO localhost
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
While thinking about the design of ccollect, I thought about enabling
backup to *remote* hosts. Though this sounds like a nice feature
('Backup my notebook to the server now.'), it is in my opinion a
bad idea to backup to a remote host, because you have to open
security at your backup host. Think of the following situation: You backup
your farm of webservers *to* a backup host somewhere else. One of
your webservers gets compromised, then your backup server will be compromised,
too. Think of it the other way round: The backup server (now behind a
firewall using NAT and strong firewall rules) connects to the
2006-02-05 23:46:50 +00:00
webservers and pulls the data *from* them. If someone gets access to one
of the webservers, the person will perhaps not even see your machine. If
2006-01-13 11:36:20 +00:00
he/she sees that there are connections from a host to the compromised
machine, he/she will not be able to login to the backup machine.
All other backups are still secure.
Incompatibilities
~~~~~~~~~~~~~~~~~
Versions < 0.4 and 0.4
^^^^^^^^^^^^^^^^^^^^^^
Since `ccollect` 0.4 there are several incompatibilities with earlier
versions:
.List of incompatibilities
- `pax` (Posix) is now required, `cp -al` (GNU specific) is removed
- "interval" was written with two 'l' (ell), which is wrong in English
- Changed the name of backup directories, removed the colon in the interval
2006-04-25 13:28:54 +00:00
- ccollect will now exit, when preexec returns non-zero
- ccollect now reports when postexec returns non-zero
You can convert your old configuration directory using
`config-pre-0.4-to-0.4.sh`, which can be found in the *tools/*
subdirectory:
--------------------------------------------------------------------------------
[10:05] hydrogenium:ccollect-0.4# ./tools/config-pre-0.4-to-0.4.sh /etc/ccollect
--------------------------------------------------------------------------------
2006-01-13 11:36:20 +00:00
Requirements
------------
Installing ccollect
~~~~~~~~~~~~~~~~~~~
For the installation, you need at least
2006-01-22 12:27:12 +00:00
- either `cp` and `chmod` or `install`
- for more comfort: `make`
- for rebuilding the generated documentation: additionally `asciidoc`
2006-01-13 11:36:20 +00:00
Using ccollect
~~~~~~~~~~~~~~
2006-01-16 17:57:31 +00:00
.Running ccollect requires the following tools installed:
2006-01-13 11:36:20 +00:00
- `bc`
- `pax` *NEW* (since ccollect 0.4, replaces previously used `cp -al`)
- `date`
2006-01-13 11:36:20 +00:00
- `rsync`
- `ssh` (if you want to use rsync over ssh, which is recommened for security)
Installing
----------
Either type 'make install' or simply copy it to a directory in your
$PATH and execute 'chmod *0755* /path/to/ccollect.sh'.
Configuring
-----------
Runtime options
~~~~~~~~~~~~~~~
`ccollect` looks for its configuration in '/etc/ccollect' or, if set, in
the directory specified by the variable '$CCOLLECT_CONF'
(use 'CCOLLECT_CONF=/your/config/dir ccollect.sh' on the shell).
When you start `ccollect`, you have either to specify which interval
2006-01-13 11:36:20 +00:00
to backup (daily, weekly, yearly; you can specify the names yourself, see below).
The interval is used to specify how many backups to keep.
2006-01-13 11:36:20 +00:00
There are also some self explaining parameters you can pass to ccollect, simply use
2006-04-25 13:32:26 +00:00
`ccollect.sh --help` for info.
2006-01-13 11:36:20 +00:00
General configuration
~~~~~~~~~~~~~~~~~~~~~
The general configuration can be found below $CCOLLECT_CONF/defaults or
/etc/ccollect/defaults. All options specified here are generally valid for
all source definitions. Though the values can be overwritten in the source
configuration.
All configuration entries are plain-text (use UTF-8 if you use
non ASCII characters) files.
Interval definition
2006-01-13 11:36:20 +00:00
^^^^^^^^^^^^^^^^^^^^
The interval definition can be found below
'$CCOLLECT_CONF/defaults/intervals/' or '/etc/ccollect/defaults/intervals'.
Every file below this directory specifies an interval. The name of the file is the
name of the interval: `intervals/'<interval name>'`.
2006-01-13 11:36:20 +00:00
The content of this file should be a single line containing a number.
This number defines how many versions of this interval to keep.
2006-01-13 11:36:20 +00:00
Example:
-------------------------------------------------------------------------
[10:23] zaphodbeeblebrox:ccollect-0.2% ls -l conf/defaults/intervals/
2006-01-13 11:36:20 +00:00
insgesamt 12
-rw-r--r-- 1 nico users 3 2005-12-08 10:24 daily
-rw-r--r-- 1 nico users 3 2005-12-08 11:36 monthly
-rw-r--r-- 1 nico users 2 2005-12-08 11:36 weekly
[10:23] zaphodbeeblebrox:ccollect-0.2% cat conf/defaults/intervals/*
2006-01-13 11:36:20 +00:00
28
12
4
--------------------------------------------------------------------------------
This means to keep 28 daily backups, 12 monthly backups and 4 weekly.
2006-02-05 23:46:50 +00:00
General pre- and post-execution
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you add '$CCOLLECT_CONF/defaults/`pre_exec`' or
'/etc/ccollect/defaults/`pre_exec`' (same with `post_exec`), `ccollect`
will start `pre_exec` before the whole backup process and
`post_exec` after backup of all sources is done.
2006-02-05 23:46:50 +00:00
The following example describes how to report free disk space in
human readable format before and after the whole backup process:
-------------------------------------------------------------------------
[13:00] hydrogenium:~# mkdir -p /etc/ccollect/defaults/
[13:00] hydrogenium:~# echo '#!/bin/sh' > /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# echo '' >> /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# echo 'df -h' >> /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# chmod 0755 /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# ln -s /etc/ccollect/defaults/pre_exec /etc/ccollect/defaults/post_exec
-------------------------------------------------------------------------
2006-02-05 23:46:50 +00:00
2006-01-13 11:36:20 +00:00
Source configuration
~~~~~~~~~~~~~~~~~~~~
Each source configuration exists below '$CCOLLECT_CONF/sources/$name' or
'/etc/ccollect/sources/$name'.
The name you choose for the subdirectory describes the source.
Each source has at least the following files:
- `source` (a text file containing the `rsync` compatible path to backup)
2006-01-13 11:36:20 +00:00
- `destination` (a link to the directory we should backup to)
Additionally a source may have the following files:
- `verbose` whether to be verbose (passes -v to `rsync`)
2006-04-25 13:32:26 +00:00
- `very_verbose` be very verbose (-v also for `mkdir`, `pax`, `rm`)
2006-01-24 23:14:58 +00:00
- `summary` create a transfer summary when `rsync` finished
2006-01-22 12:27:12 +00:00
- `exclude` exclude list for `rsync`. newline ('\n') seperated list.
- `rsync_options` extra options to pass to `rsync`
2006-01-13 11:36:20 +00:00
2006-04-25 13:32:26 +00:00
- `pre_exec` program to execute before backuping *this* source
- `post_exec` program to execute after backuping *this* source
2006-01-22 12:27:12 +00:00
2006-02-05 23:46:50 +00:00
2006-01-13 11:36:20 +00:00
Example:
--------------------------------------------------------------------------------
[10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2
insgesamt 12
lrwxrwxrwx 1 nico users 20 2005-11-17 16:44 destination -> /home/nico/backupdir
-rw-r--r-- 1 nico users 62 2005-12-07 17:43 exclude
drwxr-xr-x 2 nico users 4096 2005-12-07 17:38 intervals
2006-01-13 11:36:20 +00:00
-rw-r--r-- 1 nico users 15 2005-11-17 16:44 source
[10:47] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli
[10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervals
2006-01-13 11:36:20 +00:00
insgesamt 4
-rw-r--r-- 1 nico users 2 2005-12-07 17:38 daily
[10:48] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/intervals/daily
2006-01-13 11:36:20 +00:00
5
[10:48] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/source
/home/nico/vpn
--------------------------------------------------------------------------------
2006-02-05 23:46:50 +00:00
2006-01-13 11:36:20 +00:00
Detailled description of "source"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`source` describes a `rsync` compatible source (one line only).
2006-01-13 11:36:20 +00:00
For instance 'backup_user@foreign_host:/home/server/video'.
To use the `rsync` protocol without the `ssh`-tunnel, use
'rsync::USER@HOST/SRC'. For more information have a look at the manpage
of `rsync`(1).
2006-01-13 11:36:20 +00:00
2006-02-05 23:46:50 +00:00
2006-01-13 11:36:20 +00:00
Detailled description of "verbose"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`verbose` tells `ccollect` that the log should contain verbose messages.
If this file exists in the source specification *-v* will be passed to `rsync`.
2006-02-05 23:46:50 +00:00
2006-01-13 11:36:20 +00:00
Example:
--------------------------------------------------------------------------------
[11:35] zaphodbeeblebrox:ccollect-0.2% touch conf/sources/testsource1/verbose
--------------------------------------------------------------------------------
2006-02-05 23:46:50 +00:00
2006-01-22 09:50:20 +00:00
Detailled description of "very_verbose"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`very_verbose` tells `ccollect` that it should log very verbose.
If this file exists in the source specification *-v* will be passed to
2006-04-25 13:32:26 +00:00
`rsync`, `pax`, `rm` and `mkdir`.
2006-01-22 09:50:20 +00:00
2006-02-05 23:46:50 +00:00
2006-01-22 09:50:20 +00:00
Example:
--------------------------------------------------------------------------------
[23:67] nohost:~% touch conf/sources/testsource1/very_verbose
--------------------------------------------------------------------------------
2006-02-05 23:46:50 +00:00
2006-01-24 23:14:58 +00:00
Detailled description of "summary"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you create the file `summary` below the source definition,
`ccollect` will present you with a nice summary at the end.
-------------------------------------------------------------------------------
backup:~# touch /etc/ccollect/sources/root/summary
backup:~# ccollect.sh werktags root
==> ccollect.sh: Beginning backup using interval werktags <==
2006-01-24 23:14:58 +00:00
[root] Beginning to backup this source ...
[root] Currently 3 backup(s) exist, total keeping 50 backup(s).
[root] Beginning to backup, this may take some time...
[root] Hard linking...
[root] Transferring files...
[root]
[root] Number of files: 84183
[root] Number of files transferred: 32
[root] Total file size: 26234080536 bytes
[root] Total transferred file size: 9988252 bytes
[root] Literal data: 9988252 bytes
[root] Matched data: 0 bytes
[root] File list size: 3016771
[root] File list generation time: 1.786 seconds
[root] File list transfer time: 0.000 seconds
[root] Total bytes sent: 13009119
[root] Total bytes received: 2152
[root]
[root] sent 13009119 bytes received 2152 bytes 2891393.56 bytes/sec
[root] total size is 26234080536 speedup is 2016.26
[root] Successfully finished backup.
==> Finished ccollect.sh <==
-------------------------------------------------------------------------------
You could also combine it with `verbose` or `very_verbose`, but they
already print some statistics (but not all / the same as presented by
`summary`).
2006-01-13 11:36:20 +00:00
Detailled description of "exclude"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`exclude` specifies a list of paths to exclude. The entries are new line (\n)
seperated.
Example:
--------------------------------------------------------------------------------
[11:35] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli
something with spaces is not a problem
--------------------------------------------------------------------------------
Detailled description of "destination"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`destination` must be a link to the destination directory.
2006-02-05 23:46:50 +00:00
2006-01-13 11:36:20 +00:00
Example:
--------------------------------------------------------------------------------
[11:36] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/destination
lrwxrwxrwx 1 nico users 20 2005-11-17 16:44 conf/sources/testsource2/destination -> /home/nico/backupdir
--------------------------------------------------------------------------------
2006-02-05 23:46:50 +00:00
To speak truth, this is not fully correct. `ccollect` will also backup
your data, if `destination` is a directory. But do you really want to have
a backup below /etc?
2006-01-13 11:36:20 +00:00
Detailled description of "intervals/"
2006-01-13 11:36:20 +00:00
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When you create a subdirectory `intervals/` within your source configuration
directory, you can specify individiual intervals for this specific source.
Each file below this directory describes an interval.
2006-01-13 11:36:20 +00:00
2006-02-05 23:46:50 +00:00
2006-01-13 11:36:20 +00:00
Example:
--------------------------------------------------------------------------------
[11:37] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervals/
2006-01-13 11:36:20 +00:00
insgesamt 8
-rw-r--r-- 1 nico users 2 2005-12-07 17:38 daily
-rw-r--r-- 1 nico users 3 2005-12-14 11:33 yearly
[11:37] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/intervals/*
2006-01-13 11:36:20 +00:00
5
20
--------------------------------------------------------------------------------
2006-02-05 23:46:50 +00:00
Detailled description of "rsync_options"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2006-01-16 17:57:31 +00:00
When you create the file `rsync_options` below your source configuration,
all the parameters found in this file will be passed to rsync. This
2006-01-16 17:57:31 +00:00
way you can pass additional options to rsync. For instance you can tell rsync
to show progress ("--progress") or which -password-file ("--password-file")
to use for automatic backup over the rsync-protocol.
2006-02-05 23:46:50 +00:00
Example:
--------------------------------------------------------------------------------
[23:42] hydrogenium:ccollect-0.2% cat conf/sources/test_rsync/rsync_options
--password-file=/home/user/backup/protected_password_file
--------------------------------------------------------------------------------
2006-02-05 23:46:50 +00:00
2006-01-22 12:27:12 +00:00
Detailled description of "pre_exec" and "post_exec"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When you create `pre_exec` and / or `post_exec` below your source
configuration, `ccollect` will execute this command before,
respective after doing the backup for *this specific* source.
If you want to have pre-/post-exec before and after *all*
backups, see above for general configuration.
2006-02-05 23:46:50 +00:00
2006-01-22 12:27:12 +00:00
Example:
--------------------------------------------------------------------------------
[13:09] hydrogenium:ccollect-0.3% cat conf/sources/with_exec/pre_exec
#!/bin/sh
# Show whats free before
df -h
[13:09] hydrogenium:ccollect-0.3% cat conf/sources/with_exec/post_exec
#!/bin/sh
# Show whats free after
df -h
--------------------------------------------------------------------------------
2006-01-13 11:36:20 +00:00
2006-02-05 23:46:50 +00:00
2006-01-16 17:57:31 +00:00
Hints
-----
Using rsync protocol without ssh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When you have a computer with little computing power, it may be useful to use
rsync without ssh, directly using the rsync protocol
(specify 'user@host::share' in `source`). You may wish to use
`rsync_options` to specify a password file to use for automatic backup.
Example:
--------------------------------------------------------------------------------
backup:~# cat /etc/ccollect/sources/sample.backup.host.org/source
backup@webserver::backup-share
backup:~# cat /etc/ccollect/sources/sample.backup.host.org/rsync_options
--password-file=/etc/ccollect/sources/sample.backup.host.org/rsync_password
backup:~# cat /etc/ccollect/sources/sample.backup.host.org/rsync_password
this_is_the_rsync_password
--------------------------------------------------------------------------------
This hint was reported by Daniel Aubry.
Not-excluding top-level directories
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When you exclude "/proc" or "/mnt" from your backup, you may run into
trouble when you restore your backup. When you use "/proc/\*" or "/mnt/\*"
instead `ccollect` will backup empty directories.
[NOTE]
===========================================
When those directories contain hidden files
2006-01-16 18:01:20 +00:00
(those beginning with a dot (*.*)),
2006-01-16 17:57:31 +00:00
they will still be transferred!
===========================================
This hint was reported by Marcus Wagner.
2006-01-17 10:43:56 +00:00
Re-using already created rsync-backups
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2006-01-17 12:08:05 +00:00
If you used `rsync` directly before you use `ccollect`, you can
use this old backup as initial backup for `ccollect`: You
simply move it into a subdirectory named "'interval'.0".
2006-01-17 12:08:05 +00:00
2006-02-05 23:46:50 +00:00
2006-01-17 12:08:05 +00:00
Example:
-------------------------------------------------------------------------------
backup:/home/backup/web1# ls
bin dev etc initrd lost+found mnt root srv usr vmlinuz
boot doc home lib media opt sbin tmp var vmlinuz.old
backup:/home/backup/web1# mkdir daily.0
# ignore error about copying to itself
backup:/home/backup/web1# mv * daily.0 2>/dev/null
backup:/home/backup/web1# ls
daily.0
-------------------------------------------------------------------------------
Now you could use /home/backup/web1 as the `destination` for the backup.
[NOTE]
===============================================================================
Do *not* name the first backup something like "daily.initial", but use
the "*0*" (or some very low number, at least lower than the current year)
as extension. `ccollect` uses `sort` to find the latest backup. `ccollect`
itself uses 'interval.YEAR-MONTH-DAY-HOUR:MINUTE.PID'. This notation will
2006-01-17 12:08:05 +00:00
*always* be before "daily.initial", as numbers are earlier in the list
which is produced by `sort`. So, if you have a directory named "daily.initial",
`ccollect` will always diff against this backup and transfer and delete
files which where deleted in previous backups. This means you simply
waste resources, but your backup will be complete.
===============================================================================
2006-01-22 12:27:12 +00:00
Using pre_/post_exec
~~~~~~~~~~~~~~~~~~~~
Your pre_/post_exec script does not need to be a script, you can also
use a link to
- an existing program
- an already written script
The only requirement is that it is executable.
2006-01-17 12:08:05 +00:00
F.A.Q.
------
What happens, if one backup is broken or empty?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Let us assume, that one backup failed (connection broke or the source
hard disk had some failures). So we've one backup in our history,
which is incomplete.
2006-01-17 12:08:05 +00:00
The next time you use `ccollect`, it will transfer the missing files.
This leads to
- more transferred files
- much greater disk space usage, as no hardlinks can be used
2006-01-17 12:08:05 +00:00
2006-01-17 10:43:56 +00:00
When backing up from localhost the destination is also included. Is this a bug?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
No. `ccollect` passes your source definition directly to `rsync`. It
does not try to analyze it. So it actually does not know if a source
comes from local harddisk or from a remote server. And it does not want
to. When you backup from the local harddisk (which is perhaps not
even a good idea when thinking of security) add the `destination`
to 'source/exclude'. (Daniel Aubry reported this problem)
2006-01-22 12:27:12 +00:00
Why does ccollect say "Permission denied" with my pre-/postexec script?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The most common error is to not give your script the correct
2006-01-27 22:07:35 +00:00
permissions. Try `chmod 0755 /etc/ccollect/sources/'yoursource'/*_exec``.
2006-01-22 12:27:12 +00:00
2006-01-13 11:36:20 +00:00
Examples
--------
A backup host configuration from scratch
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--------------------------------------------------------------------------------
srwali01:~# mkdir /etc/ccollect
srwali01:~# mkdir -p /etc/ccollect/defaults/intervals/
srwali01:~# echo 28 > /etc/ccollect/defaults/intervals/taeglich
srwali01:~# echo 52 > /etc/ccollect/defaults/intervals/woechentlich
2006-01-13 11:36:20 +00:00
srwali01:~# cd /etc/ccollect/
srwali01:/etc/ccollect# mkdir sources
srwali01:/etc/ccollect# cd sources/
srwali01:/etc/ccollect/sources# ls
srwali01:/etc/ccollect/sources# mkdir local-root
srwali01:/etc/ccollect/sources# cd local-root/
srwali01:/etc/ccollect/sources/local-root# echo / > source
srwali01:/etc/ccollect/sources/local-root# cat > exclude << EOF
> /proc
> /sys
> /mnt
> EOF
srwali01:/etc/ccollect/sources/local-root# ln -s /mnt/hdbackup/local-root destination
srwali01:/etc/ccollect/sources/local-root# mkdir /mnt/hdbackup/local-root
srwali01:/etc/ccollect/sources/local-root# ccollect.sh taeglich local-root
/o> ccollect.sh: Beginning backup using interval taeglich
2006-01-13 11:36:20 +00:00
/=> Beginning to backup "local-root" ...
|-> 0 backup(s) already exist, keeping 28 backup(s).
--------------------------------------------------------------------------------
2006-02-05 23:46:50 +00:00
2006-01-13 11:36:20 +00:00
After that, I added some more sources:
--------------------------------------------------------------------------------
srwali01:~# cd /etc/ccollect/sources
srwali01:/etc/ccollect/sources# mkdir windos-wl6
srwali01:/etc/ccollect/sources# cd windos-wl6/
srwali01:/etc/ccollect/sources/windos-wl6# echo /mnt/win/SYS/WL6 > source
srwali01:/etc/ccollect/sources/windos-wl6# ln -s /mnt/hdbackup/wl6 destination
srwali01:/etc/ccollect/sources/windos-wl6# mkdir /mnt/hdbackup/wl6
srwali01:/etc/ccollect/sources/windos-wl6# cd ..
srwali01:/etc/ccollect/sources# mkdir windos-daten
srwali01:/etc/ccollect/sources/windos-daten# echo /mnt/win/Daten > source
srwali01:/etc/ccollect/sources/windos-daten# ln -s /mnt/hdbackup/windos-daten destination
srwali01:/etc/ccollect/sources/windos-daten# mkdir /mnt/hdbackup/windos-daten
# Now add some remote source
srwali01:/etc/ccollect/sources/windos-daten# cd ..
srwali01:/etc/ccollect/sources# mkdir srwali03
srwali01:/etc/ccollect/sources# cd srwali03/
srwali01:/etc/ccollect/sources/srwali03# cat > exclude << EOF
> /proc
> /sys
> /mnt
> /home
> EOF
srwali01:/etc/ccollect/sources/srwali03# echo 'root@10.103.2.3:/' > source
srwali01:/etc/ccollect/sources/srwali03# ln -s /mnt/hdbackup/srwali03 destination
srwali01:/etc/ccollect/sources/srwali03# mkdir /mnt/hdbackup/srwali03
--------------------------------------------------------------------------------
2006-02-05 23:46:50 +00:00
2006-01-13 11:36:20 +00:00
Using hard-links requires less disk space
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-------------------------------------------------------------------------
# du (coreutils) 5.2.1
2006-01-13 11:36:20 +00:00
[10:53] srsyg01:sources% du -sh ~/backupdir
4.6M /home/nico/backupdir
[10:53] srsyg01:sources% du -sh ~/backupdir/*
4.1M /home/nico/backupdir/daily.2005-12-08-10:52.28456
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28484
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28507
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28531
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28554
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28577
srwali01:/etc/ccollect/sources# du -sh /mnt/hdbackup/wl6/
186M /mnt/hdbackup/wl6/
srwali01:/etc/ccollect/sources# du -sh /mnt/hdbackup/wl6/*
147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:42.312
147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:45.588
-------------------------------------------------------------------------
The backup of our main fileserver:
-------------------------------------------------------------------------
backup:~# df -h /home/backup/srsyg01/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/backup--01-srsyg01
591G 451G 111G 81% /home/backup/srsyg01
backup:~# du -sh /home/backup/srsyg01/*
432G /home/backup/srsyg01/daily.2006-01-24-01:00.15990
432G /home/backup/srsyg01/daily.2006-01-26-01:00.30152
434G /home/backup/srsyg01/daily.2006-01-27-01:00.4596
435G /home/backup/srsyg01/daily.2006-01-28-01:00.11998
437G /home/backup/srsyg01/daily.2006-01-29-01:00.19115
437G /home/backup/srsyg01/daily.2006-01-30-01:00.26405
438G /home/backup/srsyg01/daily.2006-01-31-01:00.1148
439G /home/backup/srsyg01/daily.2006-02-01-01:00.8321
439G /home/backup/srsyg01/daily.2006-02-02-01:00.15383
439G /home/backup/srsyg01/daily.2006-02-03-01:00.22567
16K /home/backup/srsyg01/lost+found
backup:~# du --version | head -n1
du (coreutils) 5.2.1
-------------------------------------------------------------------------
Newer versions of du also detect the hardlinks, so we can even compare
the sizes directly with du:
-------------------------------------------------------------------------
[8:16] eiche:~# du --version | head -n 1
du (GNU coreutils) 5.93
[8:17] eiche:schwarzesloch# du -slh hydrogenium/*
19G hydrogenium/durcheinander.0
18G hydrogenium/durcheinander.2006-01-17-00:27.13820
19G hydrogenium/durcheinander.2006-01-25-23:18.31328
19G hydrogenium/durcheinander.2006-01-26-00:11.3332
[8:22] eiche:schwarzesloch# du -sh hydrogenium/*
19G hydrogenium/durcheinander.0
12G hydrogenium/durcheinander.2006-01-17-00:27.13820
1.5G hydrogenium/durcheinander.2006-01-25-23:18.31328
200M hydrogenium/durcheinander.2006-01-26-00:11.3332
-------------------------------------------------------------------------
2006-02-05 23:46:50 +00:00
In the second report (without -l) the sizes include the space the inodes of
the hardlinks allocate.
A collection of backups on the backup server
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
All the data of my important hosts is backuped to eiche into
/mnt/schwarzesloch/backup:
-------------------------------------------------------------------------
[9:24] eiche:backup# ls *
creme:
woechentlich.2006-01-26-22:22.4153 woechentlich.2006-02-12-11:48.2461
woechentlich.2006-01-26-22:23.4180 woechentlich.2006-02-18-23:00.7898
woechentlich.2006-02-05-02:43.14281 woechentlich.2006-02-25-23:00.13480
woechentlich.2006-02-06-00:24.15509 woechentlich.2006-03-04-23:00.25439
hydrogenium:
durcheinander.2006-01-27-11:16.6391 durcheinander.2006-02-13-01:07.2895
durcheinander.2006-01-30-19:29.9505 durcheinander.2006-02-17-08:20.6707
durcheinander.2006-01-30-22:27.9623 durcheinander.2006-02-24-16:24.12461
durcheinander.2006-02-03-09:52.12885 durcheinander.2006-03-03-19:17.18075
durcheinander.2006-02-05-23:00.15068 durcheinander.2006-03-17-22:41.5007
scice:
woechentlich.2006-02-04-10:32.13766 woechentlich.2006-02-16-23:00.6185
woechentlich.2006-02-05-23:02.15093 woechentlich.2006-02-23-23:00.11783
woechentlich.2006-02-06-08:22.15994 woechentlich.2006-03-02-23:00.17346
woechentlich.2006-02-06-19:40.16321 woechentlich.2006-03-09-23:00.29317
woechentlich.2006-02-12-11:51.2514 woechentlich.2006-03-16-23:00.4218
-------------------------------------------------------------------------
And this incremental backup and the archive are copied to an external
usb harddisk:
-------------------------------------------------------------------------
[9:23] eiche:backup# df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 14G 8.2G 4.9G 63% /
/dev/root 14G 8.2G 4.9G 63% /
/dev/root 14G 8.2G 4.9G 63% /dev/.static/dev
tmpfs 10M 444K 9.6M 5% /dev
/dev/hdh 29G 3.7M 29G 1% /mnt/datenklo
tmpfs 110M 4.0K 110M 1% /dev/shm
/dev/mapper/nirvana 112G 90G 23G 81% /mnt/datennirvana
/dev/mapper/schwarzes-loch
230G 144G 86G 63% /mnt/schwarzesloch
/dev/mapper/archiv 38G 20G 19G 52% /mnt/archiv
/dev/mapper/usb-backup
280G 36M 280G 1% /mnt/usb/backup
[9:24] eiche:backup# cat ~/bin/sync-to-usb
DDIR=/mnt/usb/backup
rsync -av --delete /mnt/schwarzesloch/ "$DDIR/schwarzesloch/"
rsync -av --delete /mnt/archiv/ "$DDIR/archiv/"
-------------------------------------------------------------------------