ungleich-public
/
ccollect
6
0
Fork 1
Code Issues Pull Requests 1 Projects Releases Wiki Activity

English correction 

Many thanks to Lina -ettercat- Boelling
This commit is contained in:
Nico Schottelius 2006-06-15 23:46:30 +02:00
parent a524a9e7a3
commit 8fd52561c1
1 changed files with 85 additions and 89 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

174
doc/ccollect.text
View File

@ -15,34 +15,34 @@ It does not depend on a specific shell, only `/bin/sh` needs to be
bourne shell compatible (like 'dash', 'ksh', 'zsh', 'bash', ...). bourne shell compatible (like 'dash', 'ksh', 'zsh', 'bash', ...).
Why you cannot backup TO remote hosts (but FROM them!) Why you can only backup from remote hosts, not to them
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
While thinking about the design of ccollect, I thought about enabling While considering the design of ccollect, I thought about enabling
backup to *remote* hosts. Though this sounds like a nice feature backup to *remote* hosts. Though this sounds like a nice feature
('Backup my notebook to the server now.'), it is in my opinion a ('Backup my notebook to the server now.'), in my opinion it is a
bad idea to backup to a remote host. bad idea to backup to a remote host.
Reason Reason
^^^^^^ ^^^^^^
To backup *TO* a remote host, you have to open security on it. If you want to backup *TO* a remote host, you have to loosen security on it.
Think of the following situation: You backup your farm of webservers *TO* Imagine the following situation: You backup your farm of webservers *TO*
a backup host somewhere else. a backup host somewhere else.
Now, one of your webservers, which has access to your backup host, gets Now one of your webservers which has access to your backup host gets
compromised. compromised.
Then your backup server will be compromised, too. Your backup server will be compromised, too.
And all data from the other webservers are also know to the attacker. And the attacker will have access to all data on the other webservers.
Doing it the secure way Doing it securely
^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^
Think of it the other way round: The backup server (now behind a Think of it the other way round: The backup server (now behind a
firewall using NAT and strong firewall rules) connects to the firewall using NAT and strong firewall rules) connects to the
webservers and pulls the data *from* them. If someone gets access to one webservers and pulls the data *from* them. If someone gets access to one
of the webservers, the person will perhaps not even see your machine. If of the webservers, this person will perhaps not even see your machine. If
the attacker sees that there are connections from a host to the compromised the attacker does see connections from a host to the compromised
machine, he/she will not be able to login to the backup machine. machine, he/she will not be able to log in on the backup machine.
All other backups are still secure. All other backups are still secure.
@ -59,7 +59,7 @@ versions:
- `pax` (Posix) is now required, `cp -al` (GNU specific) is removed - `pax` (Posix) is now required, `cp -al` (GNU specific) is removed
- "interval" was written with two 'l' (ell), which is wrong in English - "interval" was written with two 'l' (ell), which is wrong in English
- Changed the name of backup directories, removed the colon in the interval - Changed the name of backup directories, removed the colon in the interval
- ccollect will now exit, when preexec returns non-zero - ccollect will now exit when preexec returns non-zero
- ccollect now reports when postexec returns non-zero - ccollect now reports when postexec returns non-zero
You can convert your old configuration directory using You can convert your old configuration directory using
@ -76,7 +76,7 @@ Requirements
Installing ccollect Installing ccollect
~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~
For the installation, you need at least For the installation you need at least
- the latest ccollect package (http://unix.schottelius.org/ccollect/) - the latest ccollect package (http://unix.schottelius.org/ccollect/)
- either `cp` and `chmod` or `install` - either `cp` and `chmod` or `install`
@ -86,7 +86,7 @@ For the installation, you need at least
Using ccollect Using ccollect
~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
.Running ccollect requires the following tools installed: .Running ccollect requires the following tools to be installed:
- `bc` - `bc`
- `pax` *NEW* (since ccollect 0.4, replaces previously used `cp -al`) - `pax` *NEW* (since ccollect 0.4, replaces previously used `cp -al`)
- `date` - `date`
@ -107,38 +107,35 @@ Runtime options
~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~
`ccollect` looks for its configuration in '/etc/ccollect' or, if set, in `ccollect` looks for its configuration in '/etc/ccollect' or, if set, in
the directory specified by the variable '$CCOLLECT_CONF' the directory specified by the variable '$CCOLLECT_CONF'
(use 'CCOLLECT_CONF=/your/config/dir ccollect.sh' on the shell). (set 'CCOLLECT_CONF=/your/config/dir ccollect.sh' on the shell).
When you start `ccollect`, you have to specify which interval When you start `ccollect`, you have to specify in which interval
to use for backup (daily, weekly, yearly; you can specify the names yourself, to backup (daily, weekly, yearly; you can specify the names yourself, see below) and which sources to backup (or -a to backup all sources).
see below) and which sources to backup or -a (to backup all source).
The interval is used to specify how many backups to keep. The interval specifies how many backups are kept.
There are also some self explaining parameters you can pass to ccollect, simply use There are also some self-explanatory parameters you can pass to ccollect, simply use
`ccollect.sh --help` for info. `ccollect.sh --help` for info.
General configuration General configuration
~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~
The general configuration can be found below $CCOLLECT_CONF/defaults or The general configuration can be found in $CCOLLECT_CONF/defaults or
/etc/ccollect/defaults. All options specified here are generally valid for /etc/ccollect/defaults. All options specified there are generally valid for
all source definitions. Though the values can be overwritten in the source all source definitions, although the values can be overwritten in the source
configuration. configuration.
All configuration entries are plain-text (use UTF-8 if you use All configuration entries are plain-text files (use UTF-8 for non-ascii characters).
non ASCII characters) files.
Interval definition Interval definition
^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^
The interval definition can be found below The interval definition can be found in
'$CCOLLECT_CONF/defaults/intervals/' or '/etc/ccollect/defaults/intervals'. '$CCOLLECT_CONF/defaults/intervals/' or '/etc/ccollect/defaults/intervals'.
Every file below this directory specifies an interval. The name of the file is the Each file in this directory specifies an interval. The name of the file is the same as the name of the interval: `intervals/'<interval name>'`.
name of the interval: `intervals/'<interval name>'`.
The content of this file should be a single line containing a number. The content of this file should be a single line containing a number.
This number defines how many versions of this interval to keep. This number defines how many versions of this interval are kept.
Example: Example:
------------------------------------------------------------------------- -------------------------------------------------------------------------
@ -176,12 +173,12 @@ human readable format before and after the whole backup process:
Source configuration Source configuration
~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
Each source configuration exists below '$CCOLLECT_CONF/sources/$name' or Each source configuration exists in '$CCOLLECT_CONF/sources/$name' or
'/etc/ccollect/sources/$name'. '/etc/ccollect/sources/$name'.
The name you choose for the subdirectory describes the source. The name you choose for the subdirectory describes the source.
Each source has at least the following files: Each source contains at least the following files:
- `source` (a text file containing the `rsync` compatible path to backup) - `source` (a text file containing the `rsync` compatible path to backup)
- `destination` (a link to the directory we should backup to) - `destination` (a link to the directory we should backup to)
@ -192,11 +189,11 @@ Additionally a source may have the following files:
- `very_verbose` be very verbose (-v also for `mkdir`, `pax`, `rm`) - `very_verbose` be very verbose (-v also for `mkdir`, `pax`, `rm`)
- `summary` create a transfer summary when `rsync` finished - `summary` create a transfer summary when `rsync` finished
- `exclude` exclude list for `rsync`. newline ('\n') seperated list. - `exclude` exclude list for `rsync`. newline ('\n') seperates list.
- `rsync_options` extra options to pass to `rsync` - `rsync_options` extra options to pass to `rsync`
- `pre_exec` program to execute before backuping *this* source - `pre_exec` program to execute before backing up *this* source
- `post_exec` program to execute after backuping *this* source - `post_exec` program to execute after backing up *this* source
Example: Example:
@ -221,8 +218,8 @@ Example:
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
Detailled description of "source" Detailed description of "source"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`source` describes a `rsync` compatible source (one line only). `source` describes a `rsync` compatible source (one line only).
For instance 'backup_user@foreign_host:/home/server/video'. For instance 'backup_user@foreign_host:/home/server/video'.
@ -231,22 +228,22 @@ To use the `rsync` protocol without the `ssh`-tunnel, use
of `rsync`(1). of `rsync`(1).
Detailled description of "verbose" Detailed description of "verbose"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`verbose` tells `ccollect` that the log should contain verbose messages. `verbose` tells `ccollect` that the log should contain verbose messages.
If this file exists in the source specification *-v* will be passed to `rsync`. If this file exists in the source specification *-v* will be passed to `rsync`.
`
Example: Example:
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
[11:35] zaphodbeeblebrox:ccollect-0.2% touch conf/sources/testsource1/verbose [11:35] zaphodbeeblebrox:ccollect-0.2% touch conf/sources/testsource1/verbose
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
Detailled description of "very_verbose" Detailed description of "very_verbose"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`very_verbose` tells `ccollect` that it should log very verbose. `very_verbose` tells `ccollect` that it should log very verbosely.
If this file exists in the source specification *-v* will be passed to If this file exists in the source specification *-v* will be passed to
`rsync`, `pax`, `rm` and `mkdir`. `rsync`, `pax`, `rm` and `mkdir`.
@ -258,11 +255,11 @@ Example:
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
Detailled description of "summary" Detailed description of "summary"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you create the file `summary` below the source definition, If you create the file `summary` in the source definition,
`ccollect` will present you with a nice summary at the end. `ccollect` will present you a nice summary at the end.
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
backup:~# touch /etc/ccollect/sources/root/summary backup:~# touch /etc/ccollect/sources/root/summary
@ -292,15 +289,14 @@ backup:~# ccollect.sh werktags root
==> Finished ccollect.sh <== ==> Finished ccollect.sh <==
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
You could also combine it with `verbose` or `very_verbose`, but they You could also combine it with `verbose` or `very_verbose`, but these
already print some statistics (but not all / the same as presented by already print some statistics (though not all / the same as presented by
`summary`). `summary`).
Detailled description of "exclude" Detailed description of "exclude"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`exclude` specifies a list of paths to exclude. The entries are new line (\n) `exclude` specifies a list of paths to exclude. The entries are seperated by a newline (\n).
seperated.
Example: Example:
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
@ -312,8 +308,8 @@ Example:
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
Detailled description of "destination" Detailed description of "destination"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`destination` must be a link to the destination directory. `destination` must be a link to the destination directory.
@ -323,16 +319,16 @@ Example:
lrwxrwxrwx 1 nico users 20 2005-11-17 16:44 conf/sources/testsource2/destination -> /home/nico/backupdir lrwxrwxrwx 1 nico users 20 2005-11-17 16:44 conf/sources/testsource2/destination -> /home/nico/backupdir
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
To speak truth, this is not fully correct. `ccollect` will also backup To tell the truth, this is not fully correct. `ccollect` will also backup
your data, if `destination` is a directory. But do you really want to have your data if `destination` is a directory. But do you really want to have
a backup below /etc? a backup in /etc?
Detailled description of "intervals/" Detailed description of "intervals/"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When you create a subdirectory `intervals/` within your source configuration When you create the subdirectory `intervals/` in your source configuration
directory, you can specify individiual intervals for this specific source. directory, you can specify individiual intervals for this specific source.
Each file below this directory describes an interval. Each file in this directory describes an interval.
Example: Example:
@ -349,10 +345,10 @@ Example:
Detailled description of "rsync_options" Detailled description of "rsync_options"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When you create the file `rsync_options` below your source configuration, When you create the file `rsync_options` in your source configuration,
all the parameters found in this file will be passed to rsync. This all the parameters in this file will be passed to rsync. This
way you can pass additional options to rsync. For instance you can tell rsync way you can pass additional options to rsync. For instance you can tell rsync
to show progress ("--progress") or which -password-file ("--password-file") to show progress ("--progress"), or which -password-file ("--password-file")
to use for automatic backup over the rsync-protocol. to use for automatic backup over the rsync-protocol.
@ -365,9 +361,9 @@ Example:
Detailled description of "pre_exec" and "post_exec" Detailled description of "pre_exec" and "post_exec"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When you create `pre_exec` and / or `post_exec` below your source When you create `pre_exec` and / or `post_exec` in your source
configuration, `ccollect` will execute this command before, configuration, `ccollect` will execute this command before and
respective after doing the backup for *this specific* source. respectively after doing the backup for *this specific* source.
If you want to have pre-/post-exec before and after *all* If you want to have pre-/post-exec before and after *all*
backups, see above for general configuration. backups, see above for general configuration.
@ -412,11 +408,11 @@ this_is_the_rsync_password
This hint was reported by Daniel Aubry. This hint was reported by Daniel Aubry.
Not-excluding top-level directories Not excluding top-level directories
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When you exclude "/proc" or "/mnt" from your backup, you may run into When you exclude "/proc" or "/mnt" from your backup, you may run into
trouble when you restore your backup. When you use "/proc/\*" or "/mnt/\*" trouble when you restore your backup. When you use "/proc/\*" or "/mnt/\*"
instead `ccollect` will backup empty directories. instead, `ccollect` will backup empty directories.
[NOTE] [NOTE]
=========================================== ===========================================
@ -448,14 +444,14 @@ backup:/home/backup/web1# mv * daily.0 2>/dev/null
backup:/home/backup/web1# ls backup:/home/backup/web1# ls
daily.0 daily.0
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
Now you could use /home/backup/web1 as the `destination` for the backup. Now you can use /home/backup/web1 as the `destination` for the backup.
[NOTE] [NOTE]
=============================================================================== ===============================================================================
Do *not* name the first backup something like "daily.initial", but use Do *not* name the first backup something like "daily.initial", but use
the "*0*" (or some very low number, at least lower than the current year) the "*0*" (or some number that is lower than the current year)
as extension. `ccollect` uses `sort` to find the latest backup. `ccollect` as extension. `ccollect` uses `sort` to find the latest backup. `ccollect`
itself uses 'interval.YEAR-MONTH-DAY-HOURMINUTE.PID'. This notation will itself uses 'interval.YEAR-MONTH-DAY-HOUR:MINUTE.PID'. This notation will
*always* be before "daily.initial", as numbers are earlier in the list *always* be before "daily.initial", as numbers are earlier in the list
which is produced by `sort`. So, if you have a directory named "daily.initial", which is produced by `sort`. So, if you have a directory named "daily.initial",
`ccollect` will always diff against this backup and transfer and delete `ccollect` will always diff against this backup and transfer and delete
@ -477,12 +473,12 @@ The only requirement is that it is executable.
F.A.Q. F.A.Q.
------ ------
What happens, if one backup is broken or empty? What happens if one backup is broken or empty?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Let us assume, that one backup failed (connection broke or the source Let us assume that one backup failed (connection broke or the source
hard disk had some failures). So we've an incomplete backup in our history. hard disk had some failures). Therefore we've got one incomplete backup in our history.
The next time you use `ccollect`, it will transfer the missing files. `ccollect` will transfer the missing files the next time you use it.
This leads to This leads to
- more transferred files - more transferred files
@ -495,18 +491,19 @@ No. `ccollect` passes your source definition directly to `rsync`. It
does not try to analyze it. So it actually does not know if a source does not try to analyze it. So it actually does not know if a source
comes from local harddisk or from a remote server. And it does not want comes from local harddisk or from a remote server. And it does not want
to. When you backup from the local harddisk (which is perhaps not to. When you backup from the local harddisk (which is perhaps not
even a good idea when thinking of security) add the `destination` even a good idea when thinking of security), add the `destination`
to 'source/exclude'. (Daniel Aubry reported this problem) to 'source/exclude'. (Daniel Aubry reported this problem)
Why does ccollect say "Permission denied" with my pre-/postexec script? Why does ccollect say "Permission denied" with my pre-/postexec script?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The most common error is to not give your script the correct The most common error is that you have not given your script the correct
permissions. Try `chmod 0755 /etc/ccollect/sources/'yoursource'/*_exec``. permissions. Try `chmod 0755 /etc/ccollect/sources/'yoursource'/*_exec``.
Why does the backup job fail, when part of the source is a link? Why does the backup job fail when part of the source is a link?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When a part of your path you specified in the source is a When a part of your path you specified in the source is a
(symbolic, hard links are not possible for directories) link, (symbolic, hard links are not possible for directories) link,
the backup *must* fail. the backup *must* fail.
@ -525,8 +522,7 @@ First of all, let us have a look at how it looks like:
[...] [...]
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
So what is the problem? It is very obvious, when you have a deeper look So what is the problem? It is very obvious when you look deeper into it:
into it:
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
% cat /etc/ccollect/sources/testsource/source % cat /etc/ccollect/sources/testsource/source
@ -537,13 +533,13 @@ lrwxrwxrwx 1 nico nico 29 2005-12-02 23:28 /home/user/nico/projekte -> oeffentli
lrwxrwxrwx 1 nico nico 29 2006-04-29 00:01 projekte -> oeffentlich/computer/projekte lrwxrwxrwx 1 nico nico 29 2006-04-29 00:01 projekte -> oeffentlich/computer/projekte
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
`rsync` creates the directory structure until it creates the symbolic link. `rsync` creates the directory structure before it creates the symbolic link.
This link now links to something not reachable (dead link). So it is This link now links to something not reachable (dead link). It is
impossible to create subdirectories below the broken link. impossible to create subdirectories under the broken link.
So, the conclusion is you cannot use paths with a linked part. In conclusion you cannot use paths with a linked part.
*BUT* you can backup directories containing symbolic links However, you can backup directories containing symbolic links
(in this case you could backup /home/user/nico, which contains (in this case you could backup /home/user/nico, which contains
/home/user/nico/projekte and oeffentlich/computer/projekte). /home/user/nico/projekte and oeffentlich/computer/projekte).