nico/www.nico.schottelius.org
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

raw import of ccollect from unix.schottelius.org

Browse source 
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
This commit is contained in:
Nico Schottelius 2009-05-26 23:45:19 +02:00
parent 0361d4a3d1
commit 7a4e82d87a
953 changed files with 53978 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

326
software/ccollect.mdwn Normal file
View file

@ -0,0 +1,326 @@
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<html>
<head>
<TITLE>unix.schottelius.org</title>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</HEAD>
<BODY>
<P><A HREF="http://nico.schotteli.us">Nico Schottelius</A>
<BR><A HREF="../index.html">unix.schottelius.org</A></P>
<HR>
<H1>ccollect</H1>
<h3>(pseudo) incremental backup with different exclude lists using hardlinks and rsync</h3>
<HR>
<!-- ######################################################### -->
<P ALIGN="center">
<a href="screenshots/"><IMG SRC="screenshots/ccollect-0.6-screenshot-20070816.png" ALT="ccollect 0.6"
border=0></a>
<P>ccollect backups data from local and remote hosts to your local harddisk.
Although ccollect creates full backups, it requires very less space on
the backup medium, because ccollect uses hardlinks to create an initial
copy of the last backup. Only the inodes used by the hardlinks and
the changed files need additional space.
<BR>ccollect uses rsync for synchronisation. Since ccollect-0.2 there is asciidoc
based documentation included in the distribution, see below.
You should also be able to configure ccollect with the help of the example
configuration in conf/.
<h3><A NAME="doc">Documentation</A></h3>
Latest documentation is available in the following formats and languages:
(English is the primary documentation, others may be outdated)
<ul>
<li>English:
<a href="doc/ccollect.text">Text</a>
<a href="doc/ccollect.html">HTML</a>
<a href="doc/ccollect.htm">HTML (docbook)</a>
<li>English (manpage):
<a href="doc/man/ccollect.text">Text</a>
<a href="doc/man/ccollect.1">Troff</a>
<a href="doc/man/ccollect.html">HTML</a>
<a href="doc/man/ccollect.htm">HTML (docbook)</a>
<li>German: <a href="doc/ccollect-DE.text">Text</a>
<a href="doc/ccollect-DE.html">HTML</a>
<a href="doc/ccollect-DE.texi">Texinfo</a>
(<b>OUT OF DATE</b>)
</ul>
There are now also some <a href="screenshots/index.html">screenshots</a>
available.
<h4><A NAME="portability">Portability</A></h4>
<!-- ######################################################### -->
As ccollect is written in simple sh-script, it should run on
any POSIX compatible OS. It is known or reported to run on:
<ul>
<li>FreeBSD: amd64/i386
<li>Linux: alpha/hppa/ppc/i386
<li>NetBSD: alpha/sparc64/i386/amd64
<li>OpenBSD: amd64
</ul>
Feel free to send me an e-mail if you've ccollect running on another
OS or architecture.
<!-- ######################################################### -->
<h4><A NAME="config">Sample configurations</A></h4>
<ul>
<li><a href="config/ccollect-buche-2006-06-16.tar.bz2">Configuration
for ccollect 0.4 or later from buche.intern.schottelius.org</a>
<li><a href="config/ccollect-eiche-2006-02-05.tar.bz2">Configuration
for ccollect 0.3.3 or earlier from eiche.intern.schottelius.org</a>
</ul>
<h3>Getting ccollect</h3>
<h4>via git</h4>
<pre>
git clone git://git.schottelius.org/cLinux/ccollect.git
or
git clone http://git.schottelius.org/cLinux/ccollect.git
</pre>
Alternatively, you can
<A HREF="http://git.schottelius.org/?p=cLinux/ccollect.git;a=summary">look
at the current status in gitweb</A>.
<!-- ######################################################### -->
<h4>Releases</h4>
<ul>
<li><A HREF="ccollect-0.7.1.tar.bz2">ccollect-0.7.1.tar.bz2</A>
(<A HREF="ccollect-0.7.1">browse source</A>)
(<A HREF="ccollect-0.7.1/doc/changes/0.7.1">Changes</A>)
<li><A HREF="ccollect-0.7.0.tar.bz2">ccollect-0.7.0.tar.bz2</A>
(<A HREF="ccollect-0.7.0">browse source</A>)
(<A HREF="ccollect-0.7.0/doc/CHANGES">Changes</A>)
<!-- ######################################################### -->
<li><A HREF="ccollect-0.6.2.tar.bz2">ccollect-0.6.2.tar.bz2</A>
(<A HREF="ccollect-0.6.2">browse source</A>)
(<A HREF="ccollect-0.6.2/doc/CHANGES">Changes</A>)
<ul>
<li>Manpage:
(<A HREF="ccollect-0.6.2/doc/man/ccollect.text">Text</A>)
(<A HREF="ccollect-0.6.2/doc/man/ccollect.1">Troff</A>)
(<A HREF="ccollect-0.6.2/doc/man/ccollect.html">HTML</A>)
<li>English documentation:
(<A HREF="ccollect-0.6.2/doc/ccollect.text">Text</A>)
(<A HREF="ccollect-0.6.2/doc/ccollect.html">HTML</A>)
(<A HREF="ccollect-0.6.2/doc/ccollect.htm">HTML (docbook)</A>)
<li>German documentation:
(<A HREF="ccollect-0.6.2/doc/ccollect-DE.text">Text</A>)
(<A HREF="ccollect-0.6.2/doc/ccollect-DE.html">HTML</A>)
(<A HREF="ccollect-0.6.2/doc/ccollect-DE.html">HTML (docbook)</A>)
(<b>EXTREMLY OUT OF DATE</b>)
</ul>
<!-- ######################################################### -->
<li><A HREF="ccollect-0.6.1.tar.bz2">ccollect-0.6.1.tar.bz2</A>
(<A HREF="ccollect-0.6.1">browse source</A>)
(<A HREF="ccollect-0.6.1/doc/CHANGES">Changes</A>)
<ul>
<li>Manpage:
(<A HREF="ccollect-0.6.1/doc/man/ccollect.text">Text</A>)
(<A HREF="ccollect-0.6.1/doc/man/ccollect.1">Troff</A>)
(<A HREF="ccollect-0.6.1/doc/man/ccollect.html">HTML</A>)
<li>English documentation:
(<A HREF="ccollect-0.6.1/doc/ccollect.text">Text</A>)
(<A HREF="ccollect-0.6.1/doc/ccollect.html">HTML</A>)
(<A HREF="ccollect-0.6.1/doc/ccollect.htm">HTML (docbook)</A>)
<li>German documentation:
(<A HREF="ccollect-0.6.1/doc/ccollect-DE.text">Text</A>)
(<A HREF="ccollect-0.6.1/doc/ccollect-DE.html">HTML</A>)
(<A HREF="ccollect-0.6.1/doc/ccollect-DE.html">HTML (docbook)</A>)
(<b>VERY OUT OF DATE</b>)
</ul>
<!-- -------------------------------------------------------- -->
<li><A HREF="ccollect-0.6.tar.bz2">ccollect-0.6.tar.bz2</A>
(<A HREF="ccollect-0.6">browse source</A>)
(<A HREF="ccollect-0.6/doc/CHANGES">Changes</A>)
<ul>
<li>Manpage:
(<A HREF="ccollect-0.6/doc/man/ccollect.text">Text</A>)
(<A HREF="ccollect-0.6/doc/man/ccollect.1">Troff</A>)
(<A HREF="ccollect-0.6/doc/man/ccollect.html">HTML</A>)
<li>English documentation:
(<A HREF="ccollect-0.6/doc/ccollect.text">Text</A>)
(<A HREF="ccollect-0.6/doc/ccollect.html">HTML</A>)
(<A HREF="ccollect-0.6/doc/ccollect.htm">HTML (docbook)</A>)
<li>German documentation:
(<A HREF="ccollect-0.6/doc/ccollect-DE.text">Text</A>)
(<A HREF="ccollect-0.6/doc/ccollect-DE.html">HTML</A>)
(<A HREF="ccollect-0.6/doc/ccollect-DE.html">HTML (docbook)</A>)
(<b>VERY OUT OF DATE</b>)
</ul>
<!-- -------------------------------------------------------- -->
<li><A HREF="ccollect-0.5.2.tar.bz2">ccollect-0.5.2.tar.bz2</A>
(<A HREF="ccollect-0.5.2">browse source</A>)
(<A HREF="ccollect-0.5.2/doc/CHANGES">Changes</A>)
<ul>
<li>Manpage:
(<A HREF="ccollect-0.5.2/doc/man/ccollect.text">Text</A>)
(<A HREF="ccollect-0.5.2/doc/man/ccollect.man">Troff</A>)
(<A HREF="ccollect-0.5.2/doc/man/ccollect.html">HTML</A>)
<li>English documentation:
(<A HREF="ccollect-0.5.2/doc/ccollect.text">Text</A>)
(<A HREF="ccollect-0.5.2/doc/ccollect.html">HTML</A>)
(<A HREF="ccollect-0.5.2/doc/ccollect.texi">Texinfo</A>)
<li>German documentation:
(<A HREF="ccollect-0.5.2/doc/ccollect-DE.text">Text</A>)
(<A HREF="ccollect-0.5.2/doc/ccollect-DE.html">HTML</A>)
(<A HREF="ccollect-0.5.2/doc/ccollect-DE.texi">Texinfo</A>)
(<b>OUT OF DATE</b>)
</ul>
<!-- -------------------------------------------------------- -->
<li><A HREF="ccollect-0.5.1.tar.bz2">ccollect-0.5.1.tar.bz2</A>
(<A HREF="ccollect-0.5.1">browse source</A>)
(<A HREF="ccollect-0.5.1/doc/CHANGES">Changes</A>)
<ul>
<li>Manpage:
(<A HREF="ccollect-0.5.1/doc/man/ccollect.text">Text</A>)
(<A HREF="ccollect-0.5.1/doc/man/ccollect.man">Troff</A>)
(<A HREF="ccollect-0.5.1/doc/man/ccollect.html">HTML</A>)
<li>English documentation:
(<A HREF="ccollect-0.5.1/doc/ccollect.text">Text</A>)
(<A HREF="ccollect-0.5.1/doc/ccollect.html">HTML</A>)
(<A HREF="ccollect-0.5.1/doc/ccollect.texi">Texinfo</A>)
<li>German documentation:
(<A HREF="ccollect-0.5.1/doc/ccollect-DE.text">Text</A>)
(<A HREF="ccollect-0.5.1/doc/ccollect-DE.html">HTML</A>)
(<A HREF="ccollect-0.5.1/doc/ccollect-DE.texi">Texinfo</A>)
(<b>OUT OF DATE</b>)
</ul>
<!-- -------------------------------------------------------- -->
<li><A HREF="ccollect-0.5.tar.bz2">ccollect-0.5.tar.bz2</A>
(<A HREF="ccollect-0.5">browse source</A>)
(<A HREF="ccollect-0.5/doc/CHANGES">Changes</A>)
<ul>
<li>Manpage:
(<A HREF="ccollect-0.5/doc/man/ccollect.text">Text</A>)
(<A HREF="ccollect-0.5/doc/man/ccollect.man">Troff</A>)
(<A HREF="ccollect-0.5/doc/man/ccollect.html">HTML</A>)
<li>English documentation:
(<A HREF="ccollect-0.5/doc/ccollect.text">Text</A>)
(<A HREF="ccollect-0.5/doc/ccollect.html">HTML</A>)
(<A HREF="ccollect-0.5/doc/ccollect.texi">Texinfo</A>)
<li>German documentation:
(<A HREF="ccollect-0.5/doc/ccollect-DE.text">Text</A>)
(<A HREF="ccollect-0.5/doc/ccollect-DE.html">HTML</A>)
(<A HREF="ccollect-0.5/doc/ccollect-DE.texi">Texinfo</A>)
(<b>OUT OF DATE</b>)
</ul>
<!-- -------------------------------------------------------- -->
<li><A HREF="ccollect-0.4.2.tar.bz2">ccollect-0.4.2.tar.bz2</A>
(<A HREF="ccollect-0.4.2">browse source</A>)
(<A HREF="ccollect-0.4.2/doc/CHANGES">Changes</A>)
<ul>
<li>Manpage:
(<A HREF="ccollect-0.4.2/doc/man/ccollect.text">Text</A>)
(<A HREF="ccollect-0.4.2/doc/man/ccollect.man">Troff</A>)
(<A HREF="ccollect-0.4.2/doc/man/ccollect.html">HTML</A>)
<li>English documentation:
(<A HREF="ccollect-0.4.2/doc/ccollect.text">Text</A>)
(<A HREF="ccollect-0.4.2/doc/ccollect.html">HTML</A>)
(<A HREF="ccollect-0.4.2/doc/ccollect.texi">Texinfo</A>)
<li>German documentation:
(<A HREF="ccollect-0.4.2/doc/ccollect-DE.text">Text</A>)
(<A HREF="ccollect-0.4.2/doc/ccollect-DE.html">HTML</A>)
(<A HREF="ccollect-0.4.2/doc/ccollect-DE.texi">Texinfo</A>)
</ul>
<!-- -------------------------------------------------------- -->
<li><A HREF="ccollect-0.4.1.tar.bz2">ccollect-0.4.1.tar.bz2</A>
(<A HREF="ccollect-0.4.1">browse source</A>)
(<A HREF="ccollect-0.4.1/doc/CHANGES">Changes</A>)
<ul>
<li>Manpage:
(<A HREF="ccollect-0.4.1/doc/man/ccollect.text">Text</A>)
(<A HREF="ccollect-0.4.1/doc/man/ccollect.man">Troff</A>)
(<A HREF="ccollect-0.4.1/doc/man/ccollect.html">HTML</A>)
<li>English documentation:
(<A HREF="ccollect-0.4.1/doc/ccollect.text">Text</A>)
(<A HREF="ccollect-0.4.1/doc/ccollect.html">HTML</A>)
(<A HREF="ccollect-0.4.1/doc/ccollect.texi">Texinfo</A>)
<li>German documentation:
(<A HREF="ccollect-0.4.1/doc/ccollect-DE.text">Text</A>)
(<A HREF="ccollect-0.4.1/doc/ccollect-DE.html">HTML</A>)
(<A HREF="ccollect-0.4.1/doc/ccollect-DE.texi">Texinfo</A>)
</ul>
<!-- -------------------------------------------------------- -->
<li><A HREF="ccollect-0.4.tar.bz2">ccollect-0.4.tar.bz2</A>
(<A HREF="ccollect-0.4">browse source</A>)
(<A HREF="ccollect-0.4/doc/CHANGES">Changes</A>)
(<A HREF="ccollect-0.4/doc/ccollect.html">HTML documentation</A>)
(<A HREF="ccollect-0.4/doc/ccollect.text">Text documentation</A>)
<li><A HREF="ccollect-0.3.3.tar.bz2">ccollect-0.3.3.tar.bz2</A>
(<A HREF="ccollect-0.3.3">browse source</A>)
(<A HREF="ccollect-0.3.3/doc/CHANGES">Changes</A>)
(<A HREF="ccollect-0.3.3/doc/ccollect.html">HTML documentation</A>)
(<A HREF="ccollect-0.3.3/doc/ccollect.text">Text documentation</A>)
<li><A HREF="ccollect-0.3.2.tar.bz2">ccollect-0.3.2.tar.bz2</A>
(<A HREF="ccollect-0.3.2">browse source</A>)
(<A HREF="ccollect-0.3.2/doc/CHANGES">Changes</A>)
(<A HREF="ccollect-0.3.2/doc/ccollect.html">HTML documentation</A>)
(<A HREF="ccollect-0.3.2/doc/ccollect.text">Text documentation</A>)
<li><A HREF="ccollect-0.3.1.tar.bz2">ccollect-0.3.1.tar.bz2</A>
(<A HREF="ccollect-0.3.1">browse source</A>)
(<A HREF="ccollect-0.3.1/doc/CHANGES">Changes</A>)
(<A HREF="ccollect-0.3.1/doc/ccollect.html">HTML documentation</A>)
(<A HREF="ccollect-0.3.1/doc/ccollect.text">Text documentation</A>)
<li><A HREF="ccollect-0.3.tar.bz2">ccollect-0.3.tar.bz2</A>
(<A HREF="ccollect-0.3">browse source</A>)
(<A HREF="ccollect-0.3/doc/CHANGES">Changes</A>)
(<A HREF="ccollect-0.3/doc/ccollect.html">HTML documentation</A>)
(<A HREF="ccollect-0.3/doc/ccollect.text">Text documentation</A>)
<li><A HREF="ccollect-0.2.tar.bz2">ccollect-0.2.tar.bz2</A>
(<A HREF="ccollect-0.2">browse source</A>)
(<A HREF="ccollect-0.2/doc/CHANGES">Changes</A>)
(<A HREF="ccollect-0.2/doc/ccollect.html">HTML documentation</A>)
(<A HREF="ccollect-0.2/doc/ccollect.text">Text documentation</A>)
<li><A HREF="ccollect-0.1.tar.bz2">ccollect-0.1.tar.bz2</A>
(<A HREF="ccollect-0.1">browse source</A>)
</ul>
<!-- ######################################################### -->
<h4><A NAME="gentoo">gentoo</A></h4>
<P>ccollect is included into gentoo portage.
Thanks to Ren&eacute; Nussbaumer.
<!-- ######################################################### -->
<h4><A NAME="debian">Debian packages</A></h4>
<P>Marcus Wagner makes ccollect available as Debian packages.
<BR>To get ccollect insert the following line into your /etc/apt/sources.list:
<PRE>
<A HREF="http://deb.notestc.de/clinux/">deb http://deb.notestc.de/ clinux/</A>
</PRE>
<HR>
Daniel Aubry also created a Debian package containing ccollect.
To use his archive, add the following line into your /etc/apt/sources.list:
<PRE>
<A HREF="http://debian.syhosting.ch/software/">deb http://debian.syhosting.ch/ software/</A>
</PRE>
<HR>
<P>
After adding one of the above sources.list entries do
<PRE>apt-get update && apt-get install ccollect</PRE>
<h3><A name="support">Support</a></h3>
<ul>
<li><a name="ml">Mailing list</a>
<ul>
<li>Subscribe: Send an e-mail to
<b>ccollect-subscribe at lists.schottelius.org</b>
<li>Sending: Send e-mails to ccollect [at] lists.schottelius.org
<li>Unsubscribe: Send an e-mail to
<b>ccollect-unsubscribe at lists.schottelius.org</b>
</ul>
<li><a name="irc">IRC</a>
<ul>
<li>Join
<a href="irc://irc.freenode.org/#cLinux">#cLinux on irc.freenode.org</a>
</ul>
</ul>
<h4><A NAME="links">Related websites</A></h4>
<ul>
<li><A HREF="http://0xf00.de/ccollect-config.html">ccollect-config</A>,
a configuration tool written by Frederic Jaeckel
<li><A HREF="http://www.juame.ch/index.php?go=ccollect">German ccollect
documentation</A> by Julian Meier
<li><A HREF="http://muasch.ch/article/show/ccollect-backup-software">German report about
ccollect by Benedikt K&ouml;ppel, which includes a description of reverse ssh tunnels</a>
</ul>
</BODY>
</HTML>

BIN
software/ccollect/ccollect-0.1.tar.bz2 Normal file
View file

Binary file not shown.

1
software/ccollect/ccollect-0.1/.gitignore vendored Normal file
View file

@ -0,0 +1 @@
conf/sources/*/destination/*

47
software/ccollect/ccollect-0.1/CONFIGURATION Normal file
View file

@ -0,0 +1,47 @@
--------------------------------------------------------------------------------
ccollect.sh, Nico Schottelius, 2005-12-07
--------------------------------------------------------------------------------
0. Runtime options
1. General configuration
2. Source configuration
--------------------------------------------------------------------------------
0. Runtime options
ccollect looks for its configuration in /etc/ccollect or, if set, in
the directory specified in $CCOLLECT_CONF.
There are also some self explaining parameters you can pass to ccollect, simply use
"ccollect.sh --help" for info.
The "intervall" is used to know how many backups to keep.
--------------------------------------------------------------------------------
1. General configuration
The general configuration can be found below $CCOLLECT_CONF/defaults or /etc/ccollect/defaults.
1.2. Intervall definition
Below this directory should be:
intervalls/<intervall name>
The number in this file tells ccollect how many versions to keep.
--------------------------------------------------------------------------------
2. Source configuration
Each source configuration exists below $CCOLLECT_CONF/sources/$name.
The name describes the source.
Each source has at least
- source (a text file containing the path to backup)
- destination (a link to the directory we should backup to)
Additionally a source may have the following files:
- verbose whether to be verbose
- exclude excludes for rsync. One exclude specification on each line

11
software/ccollect/ccollect-0.1/HOW_IT_LOOKS_LIKE_WHEN_YOU_SAVE_SPACE Normal file
View file

@ -0,0 +1,11 @@
Hard links are such a nice thing ;-)
[10:53] srsyg01:sources% du -sh ~/backupdir
4.6M /home/nico/backupdir
[10:53] srsyg01:sources% du -sh ~/backupdir/*
4.1M /home/nico/backupdir/daily.2005-12-08-10:52.28456
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28484
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28507
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28531
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28554
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28577

5
software/ccollect/ccollect-0.1/INSTALL Normal file
View file

@ -0,0 +1,5 @@
--------------------------------------------------------------------------------
ccollect.sh, Nico Schottelius, 2005-12-08
--------------------------------------------------------------------------------
Simply copy it to a directory in your $PATH.

73
software/ccollect/ccollect-0.1/README Normal file
View file

@ -0,0 +1,73 @@
--------------------------------------------------------------------------------
ccollect.sh, Nico Schottelius, 2005-12-06
--------------------------------------------------------------------------------
ccollect backups data from local or remote hosts to your local harddisk.
You can retriev the latest version of ccollect at [0].
ccollect was inspired by rsnapshot [1], which had some problems:
- configuration parameters had to be TAB seperated
- you could not specify exclude lists differently for every source
- no parallel execution
- I didn't like the configuration at all, so I used the cconfig style [2].
[0]: ccollect: http://linux.schottelius.org/ccollect/
[1]: rsnapshot: htt://www.rsnapshot.org/
[2]: cconfig: http://nico.schotteli.us/papers/linux/cconfig/
--------------------------------------------------------------------------------
$CCOLLECT_CONF/
-> Directories, which are so called 'backup-definitions'
$dir/
source -> file with the source
destination -> link to the destination
exclude -> \n seperated
--------------------------------------------------------------------------------
source - a rsync compatible source (one liner)
For instance:
backup_user@foreign_host:/home/server/video
or
rsync://[USER@]HOST[:PORT]/SRC
Have a look at rsync(1).
--------------------------------------------------------------------------------
verbose - should we log verbose or silent
If this file exists in the source specification -v will be passed to rsync.
--------------------------------------------------------------------------------
exclude - a new line seperated list of paths to exclude
--------------------------------------------------------------------------------
destination - a link to the destination directory
--------------------------------------------------------------------------------
intervalls/ - subdirectory of source or defaults
Each file below this directory describe an intervalls.
--------------------------------------------------------------------------------
log - link to file we should log to
If a backup source exists (the cconfig dir exists) all logs for this
source will be written to this file. General errors and errors of
non existent or broken configuration will be logged to stderr.
I do not think it is senseful to have one logfile for all sources, as
the sources can be backuped in parallel and you would not be able to
distinguish the different log processes very good then.
If you REALLY REALLY REALLY want to have all in one logfile, simply
link all "log" entries to the same file, output will be appended.
--------------------------------------------------------------------------------

12
software/ccollect/ccollect-0.1/REQUIREMENTS Normal file
View file

@ -0,0 +1,12 @@
--------------------------------------------------------------------------------
ccollect.sh, Nico Schottelius, 2005-12-06 (updated: 2005-12-07)
Requirements
--------------------------------------------------------------------------------
ccollect needs the following packages:
- a bourne shell (/bin/sh) compatible shell
- bc
- cp with support for hard links (cp -al is used)
- rsync
- ssh if you want to use rsync over ssh

11
software/ccollect/ccollect-0.1/TODO Normal file
View file

@ -0,0 +1,11 @@
- write a manpage
- write a simple manual
- implement verbosity
o general, very verbose (set -x)
- implement general log
- implement source specific log
- implement parallel execution
DONE
- implement verbosity
o per source (-v to rsync)

288
software/ccollect/ccollect-0.1/ccollect.sh Normal file
View file

@ -0,0 +1,288 @@
#!/bin/sh
# Nico Schottelius
# written for SyGroup (www.sygroup.ch)
# Date: Mon Nov 14 11:45:11 CET 2005
# Last Modified:
#
# temporary as long as inofficial
#
CCOLLECT_CONF=$HOME/crsnapshot/conf
#
# where to find our configuration and temporary file
#
CCOLLECT_CONF=${CCOLLECT_CONF:-/etc/ccollect}
CSOURCES=$CCOLLECT_CONF/sources
CDEFAULTS=$CCOLLECT_CONF/defaults
TMP=$(mktemp /tmp/$(basename $0).XXXXXX)
WE=$(basename $0)
#
# catch signals
#
trap "rm -f \"$TMP\"" 1 2 15
#
# errors!
#
errecho()
{
echo "|E> Error: $@" >&2
}
#
# Tell how to use us
#
usage()
{
echo "$WE: <intervall name> [args] <sources to backup>"
echo ""
echo " Nico Schottelius (nico-linux-ccollect schottelius.org) - 2005-12-06"
echo ""
echo " Backup data pseudo incremental"
echo ""
echo " -h, --help: Show this help screen"
echo " -p, --parallel: Parellize backup process"
echo " -a, --all: Backup all sources specified in $CSOURCES"
echo ""
echo " Retrieve latest ccollect at http://linux.schottelius.org/ccollect/."
echo ""
exit 0
}
#
# need at least intervall and one source or --all
#
if [ $# -lt 2 ]; then
usage
fi
#
# Filter arguments
#
INTERVALL=$1; shift
i=1
no_shares=0
while [ $i -le $# ]; do
eval arg=\$$i
if [ "$NO_MORE_ARGS" = 1 ]; then
eval share_${no_shares}=\"$arg\"
no_shares=$[$no_shares+1]
else
case $arg in
-a|--all)
ALL=1
;;
-p|--parallel)
PARALLEL=1
;;
-h|--help)
usage
;;
--)
NO_MORE_ARGS=1
;;
*)
eval share_${no_shares}=\"$arg\"
no_shares=$[$no_shares+1]
;;
esac
fi
i=$[$i+1]
done
#
# Look, if we should take ALL sources
#
if [ "$ALL" = 1 ]; then
# reset everything specified before
no_shares=0
#
# get entries from sources
#
cwd=$(pwd)
cd $CSOURCES;
ls > "$TMP"
while read tmp; do
eval share_${no_shares}=\"$tmp\"
no_shares=$[$no_shares+1]
done < "$TMP"
fi
#
# Need at least ONE source to backup
#
if [ "$no_shares" -lt 1 ]; then
usage
else
echo "/o> $WE: Beginning backup using intervall $INTERVALL"
fi
#
# check default configuration
#
D_FILE_INTERVALL="$CDEFAULTS/intervalls/$INTERVALL"
D_INTERVALL=$(cat $D_FILE_INTERVALL 2>/dev/null)
#
# Let's do the backup
#
i=0
while [ "$i" -lt "$no_shares" ]; do
#
# Standard locations
#
eval name=\$share_${i}
backup="$CSOURCES/$name"
c_source="$backup/source"
c_dest="$backup/destination"
c_exclude="$backup/exclude"
c_verbose="$backup/verbose"
echo "/=> Beginning to backup \"$name\" ..."
i=$[$i+1]
#
# Standard configuration checks
#
if [ ! -e "$backup" ]; then
errecho "Source \"$name\" does not exist."
continue
fi
if [ ! -d "$backup" ]; then
errecho "\"$name\" is not a cconfig-directory. Skipping."
continue
fi
#
# intervall definiition: First try source specific, fallback to default
#
c_intervall="$(cat "$backup/intervalls/$INTERVALL" 2>/dev/null)"
if [ -z "$c_intervall" ]; then
c_intervall=$D_INTERVALL
if [ -z "$c_intervall" ]; then
errecho "Default and source specific intervall missing. Skipping."
continue
fi
fi
#
# standard rsync options
#
VERBOSE=""
EXCLUDE=""
#
# next configuration checks
#
if [ ! -f "$c_source" ]; then
echo "|-> Source description $c_source is not a file. Skipping."
continue
else
source=$(cat "$c_source")
if [ $? -ne 0 ]; then
echo "|-> Skipping: Source $c_source is not readable"
continue
fi
fi
if [ ! -d "$c_dest" ]; then
errecho "Destination $c_dest does not link to a directory. Skipping"
continue
fi
# exclude
if [ -f "$c_exclude" ]; then
EXCLUDE="--exclude-from=$c_exclude"
fi
# verbose
if [ -f "$c_verbose" ]; then
VERBOSE="-v"
fi
#
# check if maximum number of backups is reached, if so remove
#
# the created directories are named $INTERVALL.$DATE
count=$(ls -d "$c_dest/${INTERVALL}."?* 2>/dev/null | wc -l)
echo "|-> $count backup(s) already exist, keeping $c_intervall backup(s)."
if [ "$count" -ge "$c_intervall" ]; then
substract=$(echo $c_intervall - 1 | bc)
remove=$(echo $count - $substract | bc)
echo "|-> Removing $remove backup(s)..."
ls -d "$c_dest/${INTERVALL}."?* | sort -n | head -n $remove > "$TMP"
while read to_remove; do
dir="$to_remove"
echo "|-> Removing $dir ..."
rm -rf "$dir"
done < "$TMP"
fi
#
# clone the old directory with hardlinks
#
destination_date=$(date +%Y-%m-%d-%H:%M)
destination_dir="$c_dest/${INTERVALL}.${destination_date}.$$"
last_dir=$(ls -d "$c_dest/${INTERVALL}."?* 2>/dev/null | sort -n | tail -n 1)
# only copy if a directory exists
if [ "$last_dir" ]; then
# echo cp -al "$last_dir" "$destination_dir"
cp $VERBOSE -al "$last_dir" "$destination_dir"
else
mkdir "$destination_dir"
fi
if [ $? -ne 0 ]; then
errecho "Creating/cloning backup directory failed. Skipping backup."
continue
fi
#
# the rsync part
# options stolen shameless from rsnapshot
#
rsync -a $VERBOSE --delete --numeric-ids --relative --delete-excluded \
"$EXCLUDE" $EXCLUDE "$source" "$destination_dir"
if [ $? -ne 0 ]; then
errecho "rsync failed, backup most likely broken"
continue
fi
echo "\=> Successfully finished backup of \"$name\"."
done
#
# Be a good parent and wait for our children, if they are running wild parallel
#
if [ "$PARALLEL" = 1 ]; then
wait
fi
rm -f "$TMP"
echo "\o> Finished $WE."

1
software/ccollect/ccollect-0.1/conf/defaults/intervalls/daily Normal file
View file

@ -0,0 +1 @@
28

1
software/ccollect/ccollect-0.1/conf/defaults/intervalls/monthly Normal file
View file

@ -0,0 +1 @@
12

1
software/ccollect/ccollect-0.1/conf/defaults/intervalls/weekly Normal file
View file

@ -0,0 +1 @@
4

1
software/ccollect/ccollect-0.1/conf/sources/bla blu/destination Symbolic link
View file

@ -0,0 +1 @@
/home/nico/backupdir

1
software/ccollect/ccollect-0.1/conf/sources/bla blu/source Normal file
View file

@ -0,0 +1 @@
/home/nico/vpn

0
software/ccollect/ccollect-0.1/conf/sources/bla blu/verbose Normal file
View file

0
software/ccollect/ccollect-0.1/conf/sources/fehler Normal file
View file

3
software/ccollect/ccollect-0.1/conf/sources/share2/exclude Normal file
View file

@ -0,0 +1,3 @@
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli

1
software/ccollect/ccollect-0.1/conf/sources/share2/intervalls/daily Normal file
View file

@ -0,0 +1 @@
2

1
software/ccollect/ccollect-0.1/conf/sources/share2/source Normal file
View file

@ -0,0 +1 @@
/home/nico/vpn

1
software/ccollect/ccollect-0.1/conf/sources/source-without-intervall/exclude Normal file
View file

@ -0,0 +1 @@
manage

1
software/ccollect/ccollect-0.1/conf/sources/source-without-intervall/source Normal file
View file

@ -0,0 +1 @@
nico@creme.schottelius.org:bin

1
software/ccollect/ccollect-0.1/conf/sources/testshare/destination Symbolic link
View file

@ -0,0 +1 @@
/home/nico/backupdir

3
software/ccollect/ccollect-0.1/conf/sources/testshare/exclude Normal file
View file

@ -0,0 +1,3 @@
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli

1
software/ccollect/ccollect-0.1/conf/sources/testshare/intervalls/daily Normal file
View file

@ -0,0 +1 @@
5

1
software/ccollect/ccollect-0.1/conf/sources/testshare/source Normal file
View file

@ -0,0 +1 @@
/home/nico/vpn

1
software/ccollect/ccollect-0.1/release.sh Normal file
View file

@ -0,0 +1 @@
tar cvfj ccollect-0.1.tar.bz2 --exclude=.git --exclude="conf/sources/*/destination/*" ccollect-0.1

BIN
software/ccollect/ccollect-0.2.tar.bz2 Normal file
View file

Binary file not shown.

2
software/ccollect/ccollect-0.2/.gitignore vendored Normal file
View file

@ -0,0 +1,2 @@
conf/sources/*/destination/*
doc/old

25
software/ccollect/ccollect-0.2/Makefile Normal file
View file

@ -0,0 +1,25 @@
#
# ccollect
# Nico Schottelius, Fri Jan 13 12:13:08 CET 2006
#
INSTALL=install
CCOLLECT=ccollect.sh
LN=ln -sf
prefix=/usr/packages/ccollect-0.2
bindir=$(prefix)/bin
destination=$(bindir)/$(CCOLLECT)
path_dir=/usr/local/bin
path_destination=$(path_dir)/$(CCOLLECT)
all:
@echo "Nothing to make, make install."
documentation:
@asciidoc -n -o doc/ccollect.html doc/ccollect.text
install:
$(INSTALL) -D -m 0755 -s $(CCOLLECT) $(destination)
$(LN) $(destination) $(path_destination)

22
software/ccollect/ccollect-0.2/README Normal file
View file

@ -0,0 +1,22 @@
--------------------------------------------------------------------------------
ccollect.sh, Nico Schottelius, 2005-12-06
--------------------------------------------------------------------------------
ccollect backups data from local or remote hosts to your local harddisk.
You can retriev the latest version of ccollect at [0].
doc/ccollect.text Manual in text format
doc/ccollect.html Manual in xhtml
ccollect was inspired by rsnapshot [1], which had some problems:
- configuration parameters had to be TAB seperated
- you could not specify exclude lists differently for every source
- no parallel execution
- I didn't like the configuration at all, so I used the cconfig style [2].
[0]: ccollect: http://linux.schottelius.org/ccollect/
[1]: rsnapshot: htt://www.rsnapshot.org/
[2]: cconfig: http://nico.schotteli.us/papers/linux/cconfig/

344
software/ccollect/ccollect-0.2/ccollect.sh Normal file
View file

@ -0,0 +1,344 @@
#!/bin/sh
# Nico Schottelius
# written for SyGroup (www.sygroup.ch)
# Date: Mon Nov 14 11:45:11 CET 2005
# Last Modified: (See ls -l or git)
#
# where to find our configuration and temporary file
#
CCOLLECT_CONF=${CCOLLECT_CONF:-/etc/ccollect}
CSOURCES=$CCOLLECT_CONF/sources
CDEFAULTS=$CCOLLECT_CONF/defaults
TMP=$(mktemp /tmp/$(basename $0).XXXXXX)
WE=$(basename $0)
VERSION=0.2
RELEASE="2006-01-13"
#
# unset parallel execution
#
PARALLEL=""
#
# catch signals
#
trap "rm -f \"$TMP\"" 1 2 15
add_name()
{
sed "s/^/\[$name\] /"
}
#
# Tell how to use us
#
usage()
{
echo "$WE: <intervall name> [args] <sources to backup>"
echo ""
echo " Nico Schottelius (nico-linux-ccollect schottelius.org) - 2005-12-06"
echo ""
echo " Backup data pseudo incremental"
echo ""
echo " -h, --help: Show this help screen"
echo " -p, --parallel: Parellize backup process"
echo " -a, --all: Backup all sources specified in $CSOURCES"
echo " -v, --verbose: Be very verbose."
echo ""
echo " Retrieve latest ccollect at http://linux.schottelius.org/ccollect/."
echo ""
echo " Version: $VERSION ($RELEASE, Black Friday Release)"
exit 0
}
#
# need at least intervall and one source or --all
#
if [ $# -lt 2 ]; then
usage
fi
#
# check for configuraton directory
#
if [ ! -d "$CCOLLECT_CONF" ]; then
echo "Configuration \"$CCOLLECT_CONF\" not found."
exit 1
fi
#
# Filter arguments
#
INTERVALL=$1; shift
i=1
no_shares=0
while [ $i -le $# ]; do
eval arg=\$$i
if [ "$NO_MORE_ARGS" = 1 ]; then
eval share_${no_shares}=\"$arg\"
no_shares=$[$no_shares+1]
else
case $arg in
-a|--all)
ALL=1
;;
-v|--verbose)
VERBOSE=1
;;
-p|--parallel)
PARALLEL="1"
;;
-h|--help)
usage
;;
--)
NO_MORE_ARGS=1
;;
*)
eval share_${no_shares}=\"$arg\"
no_shares=$[$no_shares+1]
;;
esac
fi
i=$[$i+1]
done
#
# be really really really verbose
#
if [ "$VERBOSE" = 1 ]; then
set -x
fi
#
# Look, if we should take ALL sources
#
if [ "$ALL" = 1 ]; then
# reset everything specified before
no_shares=0
#
# get entries from sources
#
cwd=$(pwd)
cd "$CSOURCES";
ls > "$TMP"
while read tmp; do
eval share_${no_shares}=\"$tmp\"
no_shares=$[$no_shares+1]
done < "$TMP"
fi
#
# Need at least ONE source to backup
#
if [ "$no_shares" -lt 1 ]; then
usage
else
echo "==> $WE: Beginning backup using intervall $INTERVALL <=="
fi
#
# check default configuration
#
D_FILE_INTERVALL="$CDEFAULTS/intervalls/$INTERVALL"
D_INTERVALL=$(cat $D_FILE_INTERVALL 2>/dev/null)
#
# Let's do the backup
#
i=0
while [ "$i" -lt "$no_shares" ]; do
#
# Get current share
#
eval name=\$share_${i}
i=$[$i+1]
export name
#
# start ourself, if we want parallel execution
#
if [ "$PARALLEL" ]; then
$0 "$INTERVALL" "$name" &
continue
fi
#
# Start subshell for easy log editing
#
(
#
# Stderr to stdout, so we can produce nice logs
#
exec 2>&1
#
# Standard locations
#
backup="$CSOURCES/$name"
c_source="$backup/source"
c_dest="$backup/destination"
c_exclude="$backup/exclude"
c_verbose="$backup/verbose"
c_rsync_extra="$backup/rsync_options"
echo "Beginning to backup this source ..."
#
# Standard configuration checks
#
if [ ! -e "$backup" ]; then
echo "Source does not exist."
exit 1
fi
if [ ! -d "$backup" ]; then
echo "\"$name\" is not a cconfig-directory. Skipping."
exit 1
fi
#
# intervall definition: First try source specific, fallback to default
#
c_intervall="$(cat "$backup/intervalls/$INTERVALL" 2>/dev/null)"
if [ -z "$c_intervall" ]; then
c_intervall=$D_INTERVALL
if [ -z "$c_intervall" ]; then
echo "Default and source specific intervall missing. Skipping."
exit 1
fi
fi
#
# standard rsync options
#
VERBOSE=""
EXCLUDE=""
RSYNC_EXTRA=""
#
# next configuration checks
#
if [ ! -f "$c_source" ]; then
echo "Source description $c_source is not a file. Skipping."
exit 1
else
source=$(cat "$c_source")
if [ $? -ne 0 ]; then
echo "Skipping: Source $c_source is not readable"
exit 1
fi
fi
if [ ! -d "$c_dest" ]; then
echo "Destination $c_dest does not link to a directory. Skipping"
exit 1
fi
# exclude
if [ -f "$c_exclude" ]; then
EXCLUDE="--exclude-from=$c_exclude"
fi
# extra options for rsync
if [ -f "$c_rsync_extra" ]; then
RSYNC_EXTRA="$(cat "$c_rsync_extra")"
fi
# verbose
if [ -f "$c_verbose" ]; then
VERBOSE="-v"
fi
#
# check if maximum number of backups is reached, if so remove
#
# the created directories are named $INTERVALL.$DA
count=$(ls -d "$c_dest/${INTERVALL}."?* 2>/dev/null | wc -l)
echo "Currently $count backup(s) exist, total keeping $c_intervall backup(s)."
if [ "$count" -ge "$c_intervall" ]; then
substract=$(echo $c_intervall - 1 | bc)
remove=$(echo $count - $substract | bc)
echo "Removing $remove backup(s)..."
ls -d "$c_dest/${INTERVALL}."?* | sort -n | head -n $remove > "$TMP"
while read to_remove; do
dir="$to_remove"
echo "Removing $dir ..."
rm -rf "$dir"
done < "$TMP"
fi
#
# clone the old directory with hardlinks
#
destination_date=$(date +%Y-%m-%d-%H:%M)
destination_dir="$c_dest/${INTERVALL}.${destination_date}.$$"
last_dir=$(ls -d "$c_dest/${INTERVALL}."?* 2>/dev/null | sort -n | tail -n 1)
# give some info
echo "Beginning to backup, this may take some time..."
# only copy if a directory exists
if [ "$last_dir" ]; then
echo "Hard linking..."
cp -al $VERBOSE "$last_dir" "$destination_dir"
else
echo "Creating $destination_dir"
mkdir "$destination_dir"
fi
if [ $? -ne 0 ]; then
echo "Creating/cloning backup directory failed. Skipping backup."
exit 1
fi
#
# the rsync part
# options partly stolen from rsnapshot
#
echo "Transferring files..."
rsync -a $VERBOSE $RSYNC_EXTRA $EXCLUDE \
--delete --numeric-ids --relative --delete-excluded \
"$source" "$destination_dir"
if [ "$?" -ne 0 ]; then
echo "rsync failed, backup may be broken (see rsync errors)"
exit 1
fi
echo "Successfully finished backup."
) | add_name
done
#
# Be a good parent and wait for our children, if they are running wild parallel
#
if [ "$PARALLEL" ]; then
echo "Waiting for child jobs to complete..."
wait
fi
rm -f "$TMP"
echo "==> Finished $WE <=="

1
software/ccollect/ccollect-0.2/conf/defaults/intervalls/daily Normal file
View file

@ -0,0 +1 @@
28

1
software/ccollect/ccollect-0.2/conf/defaults/intervalls/monthly Normal file
View file

@ -0,0 +1 @@
12

1
software/ccollect/ccollect-0.2/conf/defaults/intervalls/weekly Normal file
View file

@ -0,0 +1 @@
4

1
software/ccollect/ccollect-0.2/conf/sources/bla blu/destination Symbolic link
View file

@ -0,0 +1 @@
/home/nico/backupdir

1
software/ccollect/ccollect-0.2/conf/sources/bla blu/source Normal file
View file

@ -0,0 +1 @@
/home/nico/vpn

0
software/ccollect/ccollect-0.2/conf/sources/bla blu/verbose Normal file
View file

0
software/ccollect/ccollect-0.2/conf/sources/fehler Normal file
View file

1
software/ccollect/ccollect-0.2/conf/sources/source-without-intervall/exclude Normal file
View file

@ -0,0 +1 @@
manage

1
software/ccollect/ccollect-0.2/conf/sources/source-without-intervall/source Normal file
View file

@ -0,0 +1 @@
nico@creme.schottelius.org:bin

3
software/ccollect/ccollect-0.2/conf/sources/testsource1/exclude Normal file
View file

@ -0,0 +1,3 @@
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli

1
software/ccollect/ccollect-0.2/conf/sources/testsource1/intervalls/daily Normal file
View file

@ -0,0 +1 @@
2

1
software/ccollect/ccollect-0.2/conf/sources/testsource1/source Normal file
View file

@ -0,0 +1 @@
/home/nico/bilder

1
software/ccollect/ccollect-0.2/conf/sources/testsource2/destination Symbolic link
View file

@ -0,0 +1 @@
/home/nico/backupdir

3
software/ccollect/ccollect-0.2/conf/sources/testsource2/exclude Normal file
View file

@ -0,0 +1,3 @@
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli

1
software/ccollect/ccollect-0.2/conf/sources/testsource2/intervalls/daily Normal file
View file

@ -0,0 +1 @@
5

1
software/ccollect/ccollect-0.2/conf/sources/testsource2/intervalls/yearly Normal file
View file

@ -0,0 +1 @@
20

1
software/ccollect/ccollect-0.2/conf/sources/testsource2/source Normal file
View file

@ -0,0 +1 @@
/home/nico/vpn

1
software/ccollect/ccollect-0.2/conf/sources/vpn/destination Symbolic link
View file

@ -0,0 +1 @@
/home/nico/backupdir/vpn

1
software/ccollect/ccollect-0.2/conf/sources/vpn/source Normal file
View file

@ -0,0 +1 @@
/home/nico/vpn/

7
software/ccollect/ccollect-0.2/doc/CHANGES Normal file
View file

@ -0,0 +1,7 @@
0.1 to 0.2:
* Added plausibility check
* Updated and made documentation readable
* implemented verbose option
* Fixed double exclude parameter bug
* Added much better documentation (asciidoc)
* added rsync extra parameter option

11
software/ccollect/ccollect-0.2/doc/TODO Normal file
View file

@ -0,0 +1,11 @@
- write a manpage
DONE
- implement verbosity
o per source (-v to rsync)
o general, very verbose (set -x)
- write a simple manual
- implement parallel execution
- implement general log
- implement source specific log (canceled)
- the name prefix is currently somehow inconsistent

571
software/ccollect/ccollect-0.2/doc/ccollect.html Normal file
View file

@ -0,0 +1,571 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta name="generator" content="AsciiDoc 7.0.2" />
<style type="text/css">
/* Debug borders */
p, li, dt, dd, div, pre, h1, h2, h3, h4, h5, h6 {
/*
border: 1px solid red;
*/
}
body {
margin: 1em 5% 1em 5%;
}
a { color: blue; }
a:visited { color: fuchsia; }
em {
font-style: italic;
}
strong {
font-weight: bold;
}
tt {
color: navy;
}
h1, h2, h3, h4, h5, h6 {
color: #527bbd;
font-family: sans-serif;
margin-top: 1.2em;
margin-bottom: 0.5em;
line-height: 1.3;
}
h1 {
border-bottom: 2px solid silver;
}
h2 {
border-bottom: 2px solid silver;
padding-top: 0.5em;
}
div.sectionbody {
font-family: serif;
margin-left: 0;
}
hr {
border: 1px solid silver;
}
p {
margin-top: 0.5em;
margin-bottom: 0.5em;
}
pre {
padding: 0;
margin: 0;
}
span#author {
color: #527bbd;
font-family: sans-serif;
font-weight: bold;
font-size: 1.2em;
}
span#email {
}
span#revision {
font-family: sans-serif;
}
div#footer {
font-family: sans-serif;
font-size: small;
border-top: 2px solid silver;
padding-top: 0.5em;
margin-top: 4.0em;
}
div#footer-text {
float: left;
padding-bottom: 0.5em;
}
div#footer-badges {
float: right;
padding-bottom: 0.5em;
}
div#preamble,
div.tableblock, div.imageblock, div.exampleblock, div.verseblock,
div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
div.admonitionblock {
margin-right: 10%;
margin-top: 1.5em;
margin-bottom: 1.5em;
}
div.admonitionblock {
margin-top: 2.5em;
margin-bottom: 2.5em;
}
div.content { /* Block element content. */
padding: 0;
}
/* Block element titles. */
div.title, caption.title {
font-family: sans-serif;
font-weight: bold;
text-align: left;
margin-top: 1.0em;
margin-bottom: 0.5em;
}
div.title + * {
margin-top: 0;
}
td div.title:first-child {
margin-top: 0.0em;
}
div.content div.title:first-child {
margin-top: 0.0em;
}
div.content + div.title {
margin-top: 0.0em;
}
div.sidebarblock > div.content {
background: #ffffee;
border: 1px solid silver;
padding: 0.5em;
}
div.listingblock > div.content {
border: 1px solid silver;
background: #f4f4f4;
padding: 0.5em;
}
div.quoteblock > div.content {
padding-left: 2.0em;
}
div.quoteblock .attribution {
text-align: right;
}
div.admonitionblock .icon {
vertical-align: top;
font-size: 1.1em;
font-weight: bold;
text-decoration: underline;
color: #527bbd;
padding-right: 0.5em;
}
div.admonitionblock td.content {
padding-left: 0.5em;
border-left: 2px solid silver;
}
div.exampleblock > div.content {
border-left: 2px solid silver;
padding: 0.5em;
}
div.verseblock div.content {
white-space: pre;
}
div.imageblock div.content { padding-left: 0; }
div.imageblock img { border: 1px solid silver; }
span.image img { border-style: none; }
dl {
margin-top: 0.8em;
margin-bottom: 0.8em;
}
dt {
margin-top: 0.5em;
margin-bottom: 0;
font-style: italic;
}
dd > *:first-child {
margin-top: 0;
}
ul, ol {
list-style-position: outside;
}
ol.olist2 {
list-style-type: lower-alpha;
}
div.tableblock > table {
border-color: #527bbd;
border-width: 3px;
}
thead {
font-family: sans-serif;
font-weight: bold;
}
tfoot {
font-weight: bold;
}
div.hlist {
margin-top: 0.8em;
margin-bottom: 0.8em;
}
td.hlist1 {
vertical-align: top;
font-style: italic;
padding-right: 0.8em;
}
td.hlist2 {
vertical-align: top;
}
@media print {
div#footer-badges { display: none; }
}
/* Workarounds for IE6's broken and incomplete CSS2. */
div.sidebar-content {
background: #ffffee;
border: 1px solid silver;
padding: 0.5em;
}
div.sidebar-title, div.image-title {
font-family: sans-serif;
font-weight: bold;
margin-top: 0.0em;
margin-bottom: 0.5em;
}
div.listingblock div.content {
border: 1px solid silver;
background: #f4f4f4;
padding: 0.5em;
}
div.quoteblock-content {
padding-left: 2.0em;
}
div.exampleblock-content {
border-left: 2px solid silver;
padding-left: 0.5em;
}
</style>
<title>ccollect - Installing, Configuring and Using</title>
</head>
<body>
<div id="header">
<h1>ccollect - Installing, Configuring and Using</h1>
<span id="author">Nico Schottelius</span><br />
<span id="email"><tt>&lt;<a href="mailto:nico-linux-ccollect__@__schottelius.org">nico-linux-ccollect__@__schottelius.org</a>&gt;</tt></span><br />
<span id="revision">version 0.2,</span>
2005-01-13
</div>
<div id="preamble">
<div class="sectionbody">
<p>(pseudo) incremental backup
with different exclude lists
using hardlinks and <tt>rsync</tt></p>
</div>
</div>
<h2>1. Introduction</h2>
<div class="sectionbody">
<p>ccollect is a backup utitily written in the sh-scripting language.
It does not depend on a specific shell, only <tt>/bin/sh</tt> needs to be
bourne shell compatibel (like <em>dash</em>, <em>zsh</em> or <em>bash</em>).</p>
<h3>1.1. Why you can only backup TO localhost</h3>
<p>While thinking about the design of ccollect, I thought about enabling
backup to <strong>remote</strong> hosts. Though this sounds like a nice feature
(<em>Backup my notebook to the server now.</em>), it is in my opinion a
bad idea to backup to a remote host, because you have to open
security at your backup host. Think of the following situation: You backup
your farm of webservers <strong>to</strong> a backup host somewhere else. One of
your webservers gets compromised, then your backup server will be compromised,
too. Think of it the other way round: The backup server (now behind a
firewall using NAT and strong firewall rules) connects to the
webservers and pulls the data to it. If someone gets access to the
webserver, the person will perhaps not even see your machine. If
he/she sees that there are connections from a host to the compromised
machine, he/she will not be able to login to the backup machine.
All other backups are still secure.</p>
</div>
<h2>2. Requirements</h2>
<div class="sectionbody">
<h3>2.1. Installing ccollect</h3>
<p>For the installation, you need at least
- either <em>cp</em> and <em>chmod</em> or <em>install
- for more comfort: 'make</em>
- for rebuilding the generated documentation: additionally <em>asciidoc</em></p>
<h3>2.2. Using ccollect</h3>
<div class="title">When running ccollect, it requires the following tools installed:</div><ul>
<li>
<p>
<tt>bc</tt>
</p>
</li>
<li>
<p>
<tt>cp</tt> with support for hard links (<em>cp -al</em>)
</p>
</li>
<li>
<p>
<tt>rsync</tt>
</p>
</li>
<li>
<p>
<tt>ssh</tt> (if you want to use rsync over ssh, which is recommened for security)
</p>
</li>
</ul>
</div>
<h2>3. Installing</h2>
<div class="sectionbody">
<p>Either type <em>make install</em> or simply copy it to a directory in your
$PATH and execute <em>chmod <strong>0755</strong> /path/to/ccollect.sh</em>.</p>
</div>
<h2>4. Configuring</h2>
<div class="sectionbody">
<h3>4.1. Runtime options</h3>
<p><tt>ccollect</tt> looks for its configuration in <em>/etc/ccollect</em> or, if set, in
the directory specified by the variable <em>$CCOLLECT_CONF</em>
(use <em>CCOLLECT_CONF=/your/config/dir ccollect.sh</em> on the shell).</p>
<p>When you start <tt>ccollect</tt>, you have either to specify which intervall
to backup (daily, weekly, yearly; you can specify the names yourself, see below).</p>
<p>The intervall is used to specify how many backups to keep.</p>
<p>There are also some self explaining parameters you can pass to ccollect, simply use
"ccollect.sh &#8212;help" for info.</p>
<h3>4.2. General configuration</h3>
<p>The general configuration can be found below $CCOLLECT_CONF/defaults or
/etc/ccollect/defaults. All options specified here are generally valid for
all source definitions. Though the values can be overwritten in the source
configuration.</p>
<p>All configuration entries are plain-text (use UTF-8 if you use
non ASCII characters) files.</p>
<h4>4.2.1. Intervall definition</h4>
<p>The intervall definition can be found below
<em>$CCOLLECT_CONF/defaults/intervalls/</em> or <em>/etc/ccollect/defaults/intervalls</em>.
Every file below this directory specifies an intervall. The name of the file is the
name of the intervall: <tt>intervalls/<em>&lt;intervall name&gt;</em></tt>.</p>
<p>The content of this file should be a single line containing a number.
This number defines how many versions of this intervall to keep.</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt> [10:23] zaphodbeeblebrox:ccollect-0.2% ls -l conf/defaults/intervalls/
insgesamt 12
-rw-r--r-- 1 nico users 3 2005-12-08 10:24 daily
-rw-r--r-- 1 nico users 3 2005-12-08 11:36 monthly
-rw-r--r-- 1 nico users 2 2005-12-08 11:36 weekly
[10:23] zaphodbeeblebrox:ccollect-0.2% cat conf/defaults/intervalls/*
28
12
4</tt></pre>
</div></div>
<p>This means to keep 28 daily backups, 12 monthly backups and 4 weekly.</p>
<h3>4.3. Source configuration</h3>
<p>Each source configuration exists below <em>$CCOLLECT_CONF/sources/$name</em> or
<em>/etc/ccollect/sources/$name</em>.</p>
<p>The name you choose for the subdirectory describes the source.</p>
<p>Each source has at least the following files:</p>
<ul>
<li>
<p>
<tt>source</tt> (a text file containing the rsync compatible path to backup)
</p>
</li>
<li>
<p>
<tt>destination</tt> (a link to the directory we should backup to)
</p>
</li>
</ul>
<p>Additionally a source may have the following files:</p>
<ul>
<li>
<p>
<tt>verbose</tt> whether to be verbose (passes -v to rsync)
</p>
</li>
<li>
<p>
<tt>exclude</tt> exclude list for rsync. One exclude specification on each line.
</p>
</li>
<li>
<p>
`rsync_options' extra options to pass to rsync
</p>
</li>
</ul>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt> [10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2
insgesamt 12
lrwxrwxrwx 1 nico users 20 2005-11-17 16:44 destination -&gt; /home/nico/backupdir
-rw-r--r-- 1 nico users 62 2005-12-07 17:43 exclude
drwxr-xr-x 2 nico users 4096 2005-12-07 17:38 intervalls
-rw-r--r-- 1 nico users 15 2005-11-17 16:44 source
[10:47] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli
[10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervalls
insgesamt 4
-rw-r--r-- 1 nico users 2 2005-12-07 17:38 daily
[10:48] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/intervalls/daily
5
[10:48] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/source
/home/nico/vpn</tt></pre>
</div></div>
<h4>4.3.1. Detailled description of "source"</h4>
<p><tt>source</tt> describes a rsync compatible source (one line only).</p>
<p>For instance <em>backup_user@foreign_host:/home/server/video</em>.
To use the rsync protocol without the ssh-tunnel, use
<em>rsync::USER@HOST/SRC</em>. For more information have a look at rsync(1).</p>
<h4>4.3.2. Detailled description of "verbose"</h4>
<p><tt>verbose</tt> tells <tt>ccollect</tt> that the log should contain verbose messages.</p>
<p>If this file exists in the source specification <strong>-v</strong> will be passed to <tt>rsync</tt>.</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt> [11:35] zaphodbeeblebrox:ccollect-0.2% touch conf/sources/testsource1/verbose</tt></pre>
</div></div>
<h4>4.3.3. Detailled description of "exclude"</h4>
<p><tt>exclude</tt> specifies a list of paths to exclude. The entries are new line (\n)
seperated.</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt> [11:35] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli
something with spaces is not a problem</tt></pre>
</div></div>
<h4>4.3.4. Detailled description of "destination"</h4>
<p><tt>destination</tt> must be a link to the destination directory.</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt> [11:36] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/destination
lrwxrwxrwx 1 nico users 20 2005-11-17 16:44 conf/sources/testsource2/destination -&gt; /home/nico/backupdir</tt></pre>
</div></div>
<h4>4.3.5. Detailled description of "intervalls/"</h4>
<p>When you create a subdirectory <tt>intervalls/</tt> within your source configuration
directory, you can specify individiual intervalls for this specific source.
Each file below this directory describes an intervall.</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt> [11:37] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervalls/
insgesamt 8
-rw-r--r-- 1 nico users 2 2005-12-07 17:38 daily
-rw-r--r-- 1 nico users 3 2005-12-14 11:33 yearly
[11:37] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/intervalls/*
5
20</tt></pre>
</div></div>
<h4>4.3.6. Detailled description of "rsync_options"</h4>
<p>When you create the file "rsync_options" below your source configuration,
all the parameters found in this file will be passed to rsync. This
way you may specify a rsync-passwordfile for automatic backup over
the rsync-protocoll.</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt> [23:42] hydrogenium:ccollect-0.2% cat conf/sources/test_rsync/rsync_options
--password-file=/home/user/backup/protected_password_file</tt></pre>
</div></div>
</div>
<h2>5. Examples</h2>
<div class="sectionbody">
<h3>5.1. A backup host configuration from scratch</h3>
<div class="listingblock">
<div class="content">
<pre><tt>srwali01:~# mkdir /etc/ccollect
srwali01:~# mkdir -p /etc/ccollect/defaults/intervalls/
srwali01:~# echo 28 &gt; /etc/ccollect/defaults/intervalls/taeglich
srwali01:~# echo 52 &gt; /etc/ccollect/defaults/intervalls/woechentlich
srwali01:~# cd /etc/ccollect/
srwali01:/etc/ccollect# mkdir sources
srwali01:/etc/ccollect# cd sources/
srwali01:/etc/ccollect/sources# ls
srwali01:/etc/ccollect/sources# mkdir local-root
srwali01:/etc/ccollect/sources# cd local-root/
srwali01:/etc/ccollect/sources/local-root# echo / &gt; source
srwali01:/etc/ccollect/sources/local-root# cat &gt; exclude &lt;&lt; EOF
&gt; /proc
&gt; /sys
&gt; /mnt
&gt; EOF
srwali01:/etc/ccollect/sources/local-root# ln -s /mnt/hdbackup/local-root destination
srwali01:/etc/ccollect/sources/local-root# mkdir /mnt/hdbackup/local-root
srwali01:/etc/ccollect/sources/local-root# ccollect.sh taeglich local-root
/o&gt; ccollect.sh: Beginning backup using intervall taeglich
/=&gt; Beginning to backup "local-root" ...
|-&gt; 0 backup(s) already exist, keeping 28 backup(s).</tt></pre>
</div></div>
<p>After that, I added some more sources:</p>
<div class="listingblock">
<div class="content">
<pre><tt>srwali01:~# cd /etc/ccollect/sources
srwali01:/etc/ccollect/sources# mkdir windos-wl6
srwali01:/etc/ccollect/sources# cd windos-wl6/
srwali01:/etc/ccollect/sources/windos-wl6# echo /mnt/win/SYS/WL6 &gt; source
srwali01:/etc/ccollect/sources/windos-wl6# ln -s /mnt/hdbackup/wl6 destination
srwali01:/etc/ccollect/sources/windos-wl6# mkdir /mnt/hdbackup/wl6
srwali01:/etc/ccollect/sources/windos-wl6# cd ..
srwali01:/etc/ccollect/sources# mkdir windos-daten
srwali01:/etc/ccollect/sources/windos-daten# echo /mnt/win/Daten &gt; source
srwali01:/etc/ccollect/sources/windos-daten# ln -s /mnt/hdbackup/windos-daten destination
srwali01:/etc/ccollect/sources/windos-daten# mkdir /mnt/hdbackup/windos-daten
# Now add some remote source
srwali01:/etc/ccollect/sources/windos-daten# cd ..
srwali01:/etc/ccollect/sources# mkdir srwali03
srwali01:/etc/ccollect/sources# cd srwali03/
srwali01:/etc/ccollect/sources/srwali03# cat &gt; exclude &lt;&lt; EOF
&gt; /proc
&gt; /sys
&gt; /mnt
&gt; /home
&gt; EOF
srwali01:/etc/ccollect/sources/srwali03# echo 'root@10.103.2.3:/' &gt; source
srwali01:/etc/ccollect/sources/srwali03# ln -s /mnt/hdbackup/srwali03 destination
srwali01:/etc/ccollect/sources/srwali03# mkdir /mnt/hdbackup/srwali03</tt></pre>
</div></div>
<h3>5.2. Using hard-links requires less disk space</h3>
<div class="listingblock">
<div class="content">
<pre><tt>[10:53] srsyg01:sources% du -sh ~/backupdir
4.6M /home/nico/backupdir
[10:53] srsyg01:sources% du -sh ~/backupdir/*
4.1M /home/nico/backupdir/daily.2005-12-08-10:52.28456
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28484
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28507
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28531
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28554
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28577
srwali01:/etc/ccollect/sources# du -sh /mnt/hdbackup/wl6/
186M /mnt/hdbackup/wl6/
srwali01:/etc/ccollect/sources# du -sh /mnt/hdbackup/wl6/*
147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:42.312
147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:45.588</tt></pre>
</div></div>
</div>
<div id="footer">
<div id="footer-text">
Version 0.2<br />
Last updated 13-Jan-2006 23:47:12 CEST
</div>
</div>
</body>
</html>

312
software/ccollect/ccollect-0.2/doc/ccollect.text Normal file
View file

@ -0,0 +1,312 @@
ccollect - Installing, Configuring and Using
============================================
Nico Schottelius <nico-linux-ccollect__@__schottelius.org>
v0.2, 2005-01-13
:Author Initials: NS
(pseudo) incremental backup
with different exclude lists
using hardlinks and `rsync`
Introduction
------------
ccollect is a backup utitily written in the sh-scripting language.
It does not depend on a specific shell, only `/bin/sh` needs to be
bourne shell compatibel (like 'dash', 'zsh' or 'bash').
Why you can only backup TO localhost
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
While thinking about the design of ccollect, I thought about enabling
backup to *remote* hosts. Though this sounds like a nice feature
('Backup my notebook to the server now.'), it is in my opinion a
bad idea to backup to a remote host, because you have to open
security at your backup host. Think of the following situation: You backup
your farm of webservers *to* a backup host somewhere else. One of
your webservers gets compromised, then your backup server will be compromised,
too. Think of it the other way round: The backup server (now behind a
firewall using NAT and strong firewall rules) connects to the
webservers and pulls the data to it. If someone gets access to the
webserver, the person will perhaps not even see your machine. If
he/she sees that there are connections from a host to the compromised
machine, he/she will not be able to login to the backup machine.
All other backups are still secure.
Requirements
------------
Installing ccollect
~~~~~~~~~~~~~~~~~~~
For the installation, you need at least
- either 'cp' and 'chmod' or 'install
- for more comfort: 'make'
- for rebuilding the generated documentation: additionally 'asciidoc'
Using ccollect
~~~~~~~~~~~~~~
.When running ccollect, it requires the following tools installed:
- `bc`
- `cp` with support for hard links ('cp -al')
- `rsync`
- `ssh` (if you want to use rsync over ssh, which is recommened for security)
Installing
----------
Either type 'make install' or simply copy it to a directory in your
$PATH and execute 'chmod *0755* /path/to/ccollect.sh'.
Configuring
-----------
Runtime options
~~~~~~~~~~~~~~~
`ccollect` looks for its configuration in '/etc/ccollect' or, if set, in
the directory specified by the variable '$CCOLLECT_CONF'
(use 'CCOLLECT_CONF=/your/config/dir ccollect.sh' on the shell).
When you start `ccollect`, you have either to specify which intervall
to backup (daily, weekly, yearly; you can specify the names yourself, see below).
The intervall is used to specify how many backups to keep.
There are also some self explaining parameters you can pass to ccollect, simply use
"ccollect.sh --help" for info.
General configuration
~~~~~~~~~~~~~~~~~~~~~
The general configuration can be found below $CCOLLECT_CONF/defaults or
/etc/ccollect/defaults. All options specified here are generally valid for
all source definitions. Though the values can be overwritten in the source
configuration.
All configuration entries are plain-text (use UTF-8 if you use
non ASCII characters) files.
Intervall definition
^^^^^^^^^^^^^^^^^^^^
The intervall definition can be found below
'$CCOLLECT_CONF/defaults/intervalls/' or '/etc/ccollect/defaults/intervalls'.
Every file below this directory specifies an intervall. The name of the file is the
name of the intervall: `intervalls/'<intervall name>'`.
The content of this file should be a single line containing a number.
This number defines how many versions of this intervall to keep.
Example:
-------------------------------------------------------------------------
[10:23] zaphodbeeblebrox:ccollect-0.2% ls -l conf/defaults/intervalls/
insgesamt 12
-rw-r--r-- 1 nico users 3 2005-12-08 10:24 daily
-rw-r--r-- 1 nico users 3 2005-12-08 11:36 monthly
-rw-r--r-- 1 nico users 2 2005-12-08 11:36 weekly
[10:23] zaphodbeeblebrox:ccollect-0.2% cat conf/defaults/intervalls/*
28
12
4
--------------------------------------------------------------------------------
This means to keep 28 daily backups, 12 monthly backups and 4 weekly.
Source configuration
~~~~~~~~~~~~~~~~~~~~
Each source configuration exists below '$CCOLLECT_CONF/sources/$name' or
'/etc/ccollect/sources/$name'.
The name you choose for the subdirectory describes the source.
Each source has at least the following files:
- `source` (a text file containing the rsync compatible path to backup)
- `destination` (a link to the directory we should backup to)
Additionally a source may have the following files:
- `verbose` whether to be verbose (passes -v to rsync)
- `exclude` exclude list for rsync. One exclude specification on each line.
- `rsync_options' extra options to pass to rsync
Example:
--------------------------------------------------------------------------------
[10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2
insgesamt 12
lrwxrwxrwx 1 nico users 20 2005-11-17 16:44 destination -> /home/nico/backupdir
-rw-r--r-- 1 nico users 62 2005-12-07 17:43 exclude
drwxr-xr-x 2 nico users 4096 2005-12-07 17:38 intervalls
-rw-r--r-- 1 nico users 15 2005-11-17 16:44 source
[10:47] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli
[10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervalls
insgesamt 4
-rw-r--r-- 1 nico users 2 2005-12-07 17:38 daily
[10:48] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/intervalls/daily
5
[10:48] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/source
/home/nico/vpn
--------------------------------------------------------------------------------
Detailled description of "source"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`source` describes a rsync compatible source (one line only).
For instance 'backup_user@foreign_host:/home/server/video'.
To use the rsync protocol without the ssh-tunnel, use
'rsync::USER@HOST/SRC'. For more information have a look at rsync(1).
Detailled description of "verbose"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`verbose` tells `ccollect` that the log should contain verbose messages.
If this file exists in the source specification *-v* will be passed to `rsync`.
Example:
--------------------------------------------------------------------------------
[11:35] zaphodbeeblebrox:ccollect-0.2% touch conf/sources/testsource1/verbose
--------------------------------------------------------------------------------
Detailled description of "exclude"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`exclude` specifies a list of paths to exclude. The entries are new line (\n)
seperated.
Example:
--------------------------------------------------------------------------------
[11:35] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli
something with spaces is not a problem
--------------------------------------------------------------------------------
Detailled description of "destination"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`destination` must be a link to the destination directory.
Example:
--------------------------------------------------------------------------------
[11:36] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/destination
lrwxrwxrwx 1 nico users 20 2005-11-17 16:44 conf/sources/testsource2/destination -> /home/nico/backupdir
--------------------------------------------------------------------------------
Detailled description of "intervalls/"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When you create a subdirectory `intervalls/` within your source configuration
directory, you can specify individiual intervalls for this specific source.
Each file below this directory describes an intervall.
Example:
--------------------------------------------------------------------------------
[11:37] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervalls/
insgesamt 8
-rw-r--r-- 1 nico users 2 2005-12-07 17:38 daily
-rw-r--r-- 1 nico users 3 2005-12-14 11:33 yearly
[11:37] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/intervalls/*
5
20
--------------------------------------------------------------------------------
Detailled description of "rsync_options"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When you create the file "rsync_options" below your source configuration,
all the parameters found in this file will be passed to rsync. This
way you may specify a rsync-passwordfile for automatic backup over
the rsync-protocoll.
Example:
--------------------------------------------------------------------------------
[23:42] hydrogenium:ccollect-0.2% cat conf/sources/test_rsync/rsync_options
--password-file=/home/user/backup/protected_password_file
--------------------------------------------------------------------------------
Examples
--------
A backup host configuration from scratch
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--------------------------------------------------------------------------------
srwali01:~# mkdir /etc/ccollect
srwali01:~# mkdir -p /etc/ccollect/defaults/intervalls/
srwali01:~# echo 28 > /etc/ccollect/defaults/intervalls/taeglich
srwali01:~# echo 52 > /etc/ccollect/defaults/intervalls/woechentlich
srwali01:~# cd /etc/ccollect/
srwali01:/etc/ccollect# mkdir sources
srwali01:/etc/ccollect# cd sources/
srwali01:/etc/ccollect/sources# ls
srwali01:/etc/ccollect/sources# mkdir local-root
srwali01:/etc/ccollect/sources# cd local-root/
srwali01:/etc/ccollect/sources/local-root# echo / > source
srwali01:/etc/ccollect/sources/local-root# cat > exclude << EOF
> /proc
> /sys
> /mnt
> EOF
srwali01:/etc/ccollect/sources/local-root# ln -s /mnt/hdbackup/local-root destination
srwali01:/etc/ccollect/sources/local-root# mkdir /mnt/hdbackup/local-root
srwali01:/etc/ccollect/sources/local-root# ccollect.sh taeglich local-root
/o> ccollect.sh: Beginning backup using intervall taeglich
/=> Beginning to backup "local-root" ...
|-> 0 backup(s) already exist, keeping 28 backup(s).
--------------------------------------------------------------------------------
After that, I added some more sources:
--------------------------------------------------------------------------------
srwali01:~# cd /etc/ccollect/sources
srwali01:/etc/ccollect/sources# mkdir windos-wl6
srwali01:/etc/ccollect/sources# cd windos-wl6/
srwali01:/etc/ccollect/sources/windos-wl6# echo /mnt/win/SYS/WL6 > source
srwali01:/etc/ccollect/sources/windos-wl6# ln -s /mnt/hdbackup/wl6 destination
srwali01:/etc/ccollect/sources/windos-wl6# mkdir /mnt/hdbackup/wl6
srwali01:/etc/ccollect/sources/windos-wl6# cd ..
srwali01:/etc/ccollect/sources# mkdir windos-daten
srwali01:/etc/ccollect/sources/windos-daten# echo /mnt/win/Daten > source
srwali01:/etc/ccollect/sources/windos-daten# ln -s /mnt/hdbackup/windos-daten destination
srwali01:/etc/ccollect/sources/windos-daten# mkdir /mnt/hdbackup/windos-daten
# Now add some remote source
srwali01:/etc/ccollect/sources/windos-daten# cd ..
srwali01:/etc/ccollect/sources# mkdir srwali03
srwali01:/etc/ccollect/sources# cd srwali03/
srwali01:/etc/ccollect/sources/srwali03# cat > exclude << EOF
> /proc
> /sys
> /mnt
> /home
> EOF
srwali01:/etc/ccollect/sources/srwali03# echo 'root@10.103.2.3:/' > source
srwali01:/etc/ccollect/sources/srwali03# ln -s /mnt/hdbackup/srwali03 destination
srwali01:/etc/ccollect/sources/srwali03# mkdir /mnt/hdbackup/srwali03
--------------------------------------------------------------------------------
Using hard-links requires less disk space
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-------------------------------------------------------------------------
[10:53] srsyg01:sources% du -sh ~/backupdir
4.6M /home/nico/backupdir
[10:53] srsyg01:sources% du -sh ~/backupdir/*
4.1M /home/nico/backupdir/daily.2005-12-08-10:52.28456
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28484
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28507
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28531
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28554
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28577
srwali01:/etc/ccollect/sources# du -sh /mnt/hdbackup/wl6/
186M /mnt/hdbackup/wl6/
srwali01:/etc/ccollect/sources# du -sh /mnt/hdbackup/wl6/*
147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:42.312
147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:45.588
-------------------------------------------------------------------------

19
software/ccollect/ccollect-0.2/release.sh Normal file
View file

@ -0,0 +1,19 @@
if [ $# -ne 1 ]; then
echo "$0: ccollect dir"
exit 23
fi
NAME=$1
TARNAME=${NAME}.tar.bz2
DHOST=nico@creme.schottelius.org
DDIR=www/org/schottelius/linux/ccollect/
DESTINATION="$DHOST:$DDIR"
tar cvfj "$TARNAME" \
--exclude=.git \
--exclude="conf/sources/*/destination/*" "$NAME"
scp "${TARNAME}" "$DESTINATION"
ssh "$DHOST" "( cd $DDIR; tar xfj \"$TARNAME\" )"

BIN
software/ccollect/ccollect-0.3.1.tar.bz2 Normal file
View file

Binary file not shown.

2
software/ccollect/ccollect-0.3.1/.gitignore vendored Normal file
View file

@ -0,0 +1,2 @@
conf/sources/*/destination/*
doc/old

340
software/ccollect/ccollect-0.3.1/COPYING Normal file
View file

@ -0,0 +1,340 @@
GNU GENERAL PUBLIC LICENSE
Version 2, June 1991
Copyright (C) 1989, 1991 Free Software Foundation, Inc.
51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users. This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it. (Some other Free Software Foundation software is covered by
the GNU Library General Public License instead.) You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.
To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have. You must make sure that they, too, receive or can get the
source code. And you must show them these terms so they know their
rights.
We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.
Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software. If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.
Finally, any free program is threatened constantly by software
patents. We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary. To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.
The precise terms and conditions for copying, distribution and
modification follow.
GNU GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License. The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language. (Hereinafter, translation is included without limitation in
the term "modification".) Each licensee is addressed as "you".
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.
1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.
You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.
2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) You must cause the modified files to carry prominent notices
stating that you changed the files and the date of any change.
b) You must cause any work that you distribute or publish, that in
whole or in part contains or is derived from the Program or any
part thereof, to be licensed as a whole at no charge to all third
parties under the terms of this License.
c) If the modified program normally reads commands interactively
when run, you must cause it, when started running for such
interactive use in the most ordinary way, to print or display an
announcement including an appropriate copyright notice and a
notice that there is no warranty (or else, saying that you provide
a warranty) and that users may redistribute the program under
these conditions, and telling the user how to view a copy of this
License. (Exception: if the Program itself is interactive but
does not normally print such an announcement, your work based on
the Program is not required to print an announcement.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.
In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:
a) Accompany it with the complete corresponding machine-readable
source code, which must be distributed under the terms of Sections
1 and 2 above on a medium customarily used for software interchange; or,
b) Accompany it with a written offer, valid for at least three
years, to give any third party, for a charge no more than your
cost of physically performing source distribution, a complete
machine-readable copy of the corresponding source code, to be
distributed under the terms of Sections 1 and 2 above on a medium
customarily used for software interchange; or,
c) Accompany it with the information you received as to the offer
to distribute corresponding source code. (This alternative is
allowed only for noncommercial distribution and only if you
received the program in object code or executable form with such
an offer, in accord with Subsection b above.)
The source code for a work means the preferred form of the work for
making modifications to it. For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable. However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.
If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.
4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License. Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.
5. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Program or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.
6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.
7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all. For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.
If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded. In such case, this License incorporates
the limitation as if written in the body of this License.
9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation. If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.
10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission. For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this. Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.
NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
Also add information on how to contact you by electronic and paper mail.
If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:
Gnomovision version 69, Copyright (C) year name of author
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
`Gnomovision' (which makes passes at compilers) written by James Hacker.
<signature of Ty Coon>, 1 April 1989
Ty Coon, President of Vice
This General Public License does not permit incorporating your program into
proprietary programs. If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library. If this is what you want to do, use the GNU Library General
Public License instead of this License.

53
software/ccollect/ccollect-0.3.1/Makefile Normal file
View file

@ -0,0 +1,53 @@
#
# ccollect
# Nico Schottelius, Fri Jan 13 12:13:08 CET 2006
#
INSTALL=install
CCOLLECT=ccollect.sh
LN=ln -sf
prefix=/usr/packages/ccollect-0.2
bindir=$(prefix)/bin
destination=$(bindir)/$(CCOLLECT)
path_dir=/usr/local/bin
path_destination=$(path_dir)/$(CCOLLECT)
# where to publish
host=creme.schottelius.org
dir=www/org/schottelius/linux/ccollect
docdir=$(dir)/doc
#
# End user targets
#
all:
@echo "Nothing to make, make install."
install: install-script install-link
install-link: install-script
$(LN) $(destination) $(path_destination)
install-script:
$(INSTALL) -D -m 0755 -s $(CCOLLECT) $(destination)
documentation:
@echo "Generating HTML-documentation"
@asciidoc -n -o doc/ccollect.html doc/ccollect.text
#
# Developer targets
#
update:
@cg-update creme
push-work:
@cg-push creme
@cg-push main
publish-doc: documentation
@chmod a+r doc/ccollect.html
@scp doc/ccollect.html doc/ccollect.text $(host):$(docdir)

23
software/ccollect/ccollect-0.3.1/README Normal file
View file

@ -0,0 +1,23 @@
--------------------------------------------------------------------------------
ccollect.sh, Nico Schottelius, 2005-12-06
--------------------------------------------------------------------------------
ccollect backups data from local or remote hosts to your local harddisk.
You can retriev the latest version of ccollect at [0].
doc/ccollect.text Manual in text format
doc/ccollect.html Manual in xhtml
ccollect was inspired by rsnapshot [1], which had some problems:
- configuration parameters had to be TAB seperated
- you could not specify exclude lists differently for every source
- no per source pre/post execution support
- no parallel execution
- I didn't like the configuration at all, so I used the cconfig style [2].
[0]: ccollect: http://linux.schottelius.org/ccollect/
[1]: rsnapshot: http://www.rsnapshot.org/
[2]: cconfig: http://nico.schotteli.us/papers/linux/cconfig/

402
software/ccollect/ccollect-0.3.1/ccollect.sh Normal file
View file

@ -0,0 +1,402 @@
#!/bin/sh
# Nico Schottelius
# written for SyGroup (www.sygroup.ch)
# Date: Mon Nov 14 11:45:11 CET 2005
# Last Modified: (See ls -l or git)
#
# where to find our configuration and temporary file
#
CCOLLECT_CONF=${CCOLLECT_CONF:-/etc/ccollect}
CSOURCES=$CCOLLECT_CONF/sources
CDEFAULTS=$CCOLLECT_CONF/defaults
CPREEXEC="$CDEFAULTS/pre_exec"
CPOSTEXEC="$CDEFAULTS/post_exec"
TMP=$(mktemp /tmp/$(basename $0).XXXXXX)
VERSION=0.3.1
RELEASE="2006-XX-XX"
HALF_VERSION="ccollect $VERSION"
FULL_VERSION="ccollect $VERSION ($RELEASE)"
#
# unset parallel execution
#
PARALLEL=""
#
# catch signals
#
trap "rm -f \"$TMP\"" 1 2 15
add_name()
{
sed "s/^/\[$name\] /"
}
#
# Tell how to use us
#
usage()
{
echo "$(basename $0): <intervall name> [args] <sources to backup>"
echo ""
echo " ccollect creates (pseudo) incremental backups"
echo ""
echo " -h, --help: Show this help screen"
echo " -p, --parallel: Parallelise backup processes"
echo " -a, --all: Backup all sources specified in $CSOURCES"
echo " -v, --verbose: Be very verbose (uses set -x)."
echo ""
echo ""
echo " On 2005-12-05 ccollect was written by Nico Schottelius."
echo ""
echo " This is version $VERSION, released at ${RELEASE}."
echo ""
echo " Retrieve latest ccollect at http://linux.schottelius.org/ccollect/"
exit 0
}
#
# need at least intervall and one source or --all
#
if [ $# -lt 2 ]; then
usage
fi
#
# check for configuraton directory
#
if [ ! -d "$CCOLLECT_CONF" ]; then
echo "No configuration found in \"$CCOLLECT_CONF\"" \
" (set \$CCOLLECT_CONF corectly?)"
exit 1
fi
#
# Filter arguments
#
INTERVALL=$1; shift
i=1
no_shares=0
while [ $i -le $# ]; do
eval arg=\$$i
if [ "$NO_MORE_ARGS" = 1 ]; then
eval share_${no_shares}=\"$arg\"
no_shares=$(($no_shares+1))
else
case $arg in
-a|--all)
ALL=1
;;
-v|--verbose)
VERBOSE=1
;;
-p|--parallel)
PARALLEL="1"
;;
-h|--help)
usage
;;
--)
NO_MORE_ARGS=1
;;
*)
eval share_${no_shares}=\"$arg\"
no_shares=$(($no_shares+1))
;;
esac
fi
i=$(($i+1))
done
#
# be really really really verbose
#
if [ "$VERBOSE" = 1 ]; then
set -x
fi
#
# Look, if we should take ALL sources
#
if [ "$ALL" = 1 ]; then
# reset everything specified before
no_shares=0
#
# get entries from sources
#
cwd=$(pwd)
cd "$CSOURCES";
ls > "$TMP"
while read tmp; do
eval share_${no_shares}=\"$tmp\"
no_shares=$(($no_shares+1))
done < "$TMP"
fi
#
# Need at least ONE source to backup
#
if [ "$no_shares" -lt 1 ]; then
usage
else
echo "==> $HALF_VERSION: Beginning backup using intervall $INTERVALL <=="
fi
#
# check default configuration
#
D_FILE_INTERVALL="$CDEFAULTS/intervalls/$INTERVALL"
D_INTERVALL=$(cat $D_FILE_INTERVALL 2>/dev/null)
#
# Look for pre-exec command (general)
#
if [ -x "$CPREEXEC" ]; then
echo "Executing $CPREEXEC ..."
"$CPREEXEC"
echo "Finished ${CPREEXEC}."
fi
#
# Let's do the backup
#
i=0
while [ "$i" -lt "$no_shares" ]; do
#
# Get current share
#
eval name=\$share_${i}
i=$(($i+1))
export name
#
# start ourself, if we want parallel execution
#
if [ "$PARALLEL" ]; then
$0 "$INTERVALL" "$name" &
continue
fi
#
# Start subshell for easy log editing
#
(
#
# Stderr to stdout, so we can produce nice logs
#
exec 2>&1
#
# Standard locations
#
backup="$CSOURCES/$name"
c_source="$backup/source"
c_dest="$backup/destination"
c_exclude="$backup/exclude"
c_verbose="$backup/verbose"
c_vverbose="$backup/very_verbose"
c_rsync_extra="$backup/rsync_options"
c_summary="$backup/summary"
c_pre_exec="$backup/pre_exec"
c_post_exec="$backup/post_exec"
echo "Beginning to backup this source ..."
#
# Standard configuration checks
#
if [ ! -e "$backup" ]; then
echo "Source does not exist."
exit 1
fi
if [ ! -d "$backup" ]; then
echo "\"$name\" is not a cconfig-directory. Skipping."
exit 1
fi
#
# intervall definition: First try source specific, fallback to default
#
c_intervall="$(cat "$backup/intervalls/$INTERVALL" 2>/dev/null)"
if [ -z "$c_intervall" ]; then
c_intervall=$D_INTERVALL
if [ -z "$c_intervall" ]; then
echo "Default and source specific intervall missing. Skipping."
exit 1
fi
fi
#
# unset possible options
#
EXCLUDE=""
RSYNC_EXTRA=""
SUMMARY=""
VERBOSE=""
VVERBOSE=""
#
# next configuration checks
#
if [ ! -f "$c_source" ]; then
echo "Source description $c_source is not a file. Skipping."
exit 1
else
source=$(cat "$c_source")
if [ $? -ne 0 ]; then
echo "Skipping: Source $c_source is not readable"
exit 1
fi
fi
if [ ! -d "$c_dest" ]; then
echo "Destination $c_dest does not link to a directory. Skipping"
exit 1
fi
#
# pre_exec
#
if [ -x "$c_pre_exec" ]; then
echo "Executing $c_pre_exec ..."
$c_pre_exec
echo "Finished ${c_pre_exec}."
fi
# exclude
if [ -f "$c_exclude" ]; then
EXCLUDE="--exclude-from=$c_exclude"
fi
# extra options for rsync
if [ -f "$c_rsync_extra" ]; then
RSYNC_EXTRA="$(cat "$c_rsync_extra")"
fi
# verbosity for rsync
if [ -f "$c_verbose" ]; then
VERBOSE="-v"
fi
# Output a summary
if [ -f "$c_summary" ]; then
SUMMARY="--stats"
fi
# MORE verbosity, includes standard verbosity
if [ -f "$c_vverbose" ]; then
VERBOSE="-v"
VVERBOSE="-v"
fi
#
# check if maximum number of backups is reached, if so remove
#
# the created directories are named $INTERVALL.$DA
count=$(ls -d "$c_dest/${INTERVALL}."?* 2>/dev/null | wc -l)
echo "Currently $count backup(s) exist, total keeping $c_intervall backup(s)."
if [ "$count" -ge "$c_intervall" ]; then
substract=$(echo $c_intervall - 1 | bc)
remove=$(echo $count - $substract | bc)
echo "Removing $remove backup(s)..."
ls -d "$c_dest/${INTERVALL}."?* | sort -n | head -n $remove > "$TMP"
while read to_remove; do
dir="$to_remove"
echo "Removing $dir ..."
rm $VVERBOSE -rf "$dir"
done < "$TMP"
fi
#
# clone the old directory with hardlinks
#
destination_date=$(date +%Y-%m-%d-%H:%M)
destination_dir="$c_dest/${INTERVALL}.${destination_date}.$$"
last_dir=$(ls -d "$c_dest/${INTERVALL}."?* 2>/dev/null | sort -n | tail -n 1)
# give some info
echo "Beginning to backup, this may take some time..."
# only copy if a directory exists
if [ "$last_dir" ]; then
echo "Hard linking..."
cp -al $VVERBOSE "$last_dir" "$destination_dir"
else
echo "Creating $destination_dir"
mkdir $VVERBOSE "$destination_dir"
fi
if [ $? -ne 0 ]; then
echo "Creating/cloning backup directory failed. Skipping backup."
exit 1
fi
#
# the rsync part
# options partly stolen from rsnapshot
#
echo "Transferring files..."
rsync -a $VERBOSE $RSYNC_EXTRA $EXCLUDE $SUMMARY \
--delete --numeric-ids --relative --delete-excluded \
"$source" "$destination_dir"
if [ "$?" -ne 0 ]; then
echo "rsync failed, backup may be broken (see rsync errors)"
exit 1
fi
echo "Successfully finished backup."
#
# post_exec
#
if [ -x "$c_post_exec" ]; then
echo "Executing $c_post_exec ..."
"$c_post_exec"
echo "Finished ${c_post_exec}."
fi
) | add_name
done
#
# Be a good parent and wait for our children, if they are running wild parallel
#
if [ "$PARALLEL" ]; then
echo "Waiting for child jobs to complete..."
wait
fi
#
# Look for post-exec command (general)
#
if [ -x "$CPOSTEXEC" ]; then
echo "Executing $CPOSTEXEC ..."
"$CPOSTEXEC"
echo "Finished ${CPOSTEXEC}."
fi
rm -f "$TMP"
echo "==> Finished $WE <=="

1
software/ccollect/ccollect-0.3.1/conf/defaults/intervalls/daily Normal file
View file

@ -0,0 +1 @@
28

1
software/ccollect/ccollect-0.3.1/conf/defaults/intervalls/monthly Normal file
View file

@ -0,0 +1 @@
12

1
software/ccollect/ccollect-0.3.1/conf/defaults/intervalls/weekly Normal file
View file

@ -0,0 +1 @@
4

5
software/ccollect/ccollect-0.3.1/conf/defaults/post_exec Normal file
View file

@ -0,0 +1,5 @@
#!/bin/cat
######################################################################
If you see this content, post_exec was executed.
######################################################################

3
software/ccollect/ccollect-0.3.1/conf/defaults/pre_exec Normal file
View file

@ -0,0 +1,3 @@
#!/bin/cat
If you see this content, pre_exec was executed.

1
software/ccollect/ccollect-0.3.1/conf/sources/bla blu/destination Symbolic link
View file

@ -0,0 +1 @@
/home/nico/backupdir

1
software/ccollect/ccollect-0.3.1/conf/sources/bla blu/source Normal file
View file

@ -0,0 +1 @@
/home/nico/vpn

0
software/ccollect/ccollect-0.3.1/conf/sources/bla blu/verbose Normal file
View file

0
software/ccollect/ccollect-0.3.1/conf/sources/fehler Normal file
View file

1
software/ccollect/ccollect-0.3.1/conf/sources/source-without-intervall/exclude Normal file
View file

@ -0,0 +1 @@
manage

1
software/ccollect/ccollect-0.3.1/conf/sources/source-without-intervall/source Normal file
View file

@ -0,0 +1 @@
nico@creme.schottelius.org:bin

1
software/ccollect/ccollect-0.3.1/conf/sources/testsource1/destination Symbolic link
View file

@ -0,0 +1 @@
/home/nico/backupdir/testsource1

3
software/ccollect/ccollect-0.3.1/conf/sources/testsource1/exclude Normal file
View file

@ -0,0 +1,3 @@
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli

1
software/ccollect/ccollect-0.3.1/conf/sources/testsource1/intervalls/daily Normal file
View file

@ -0,0 +1 @@
2

1
software/ccollect/ccollect-0.3.1/conf/sources/testsource1/source Normal file
View file

@ -0,0 +1 @@
/home/nico/bilder

1
software/ccollect/ccollect-0.3.1/conf/sources/testsource2/destination Symbolic link
View file

@ -0,0 +1 @@
/home/nico/backupdir

3
software/ccollect/ccollect-0.3.1/conf/sources/testsource2/exclude Normal file
View file

@ -0,0 +1,3 @@
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli

1
software/ccollect/ccollect-0.3.1/conf/sources/testsource2/intervalls/daily Normal file
View file

@ -0,0 +1 @@
5

1
software/ccollect/ccollect-0.3.1/conf/sources/testsource2/intervalls/yearly Normal file
View file

@ -0,0 +1 @@
20

1
software/ccollect/ccollect-0.3.1/conf/sources/testsource2/source Normal file
View file

@ -0,0 +1 @@
/home/nico/vpn

1
software/ccollect/ccollect-0.3.1/conf/sources/vpn/destination Symbolic link
View file

@ -0,0 +1 @@
/home/nico/backupdir/vpn

1
software/ccollect/ccollect-0.3.1/conf/sources/vpn/source Normal file
View file

@ -0,0 +1 @@
/home/nico/vpn/

1
software/ccollect/ccollect-0.3.1/conf/sources/with_exec/destination Symbolic link
View file

@ -0,0 +1 @@
/tmp

5
software/ccollect/ccollect-0.3.1/conf/sources/with_exec/post_exec Normal file
View file

@ -0,0 +1,5 @@
#!/bin/sh
# Show whats free after
df -h

5
software/ccollect/ccollect-0.3.1/conf/sources/with_exec/pre_exec Normal file
View file

@ -0,0 +1,5 @@
#!/bin/sh
# Show whats free before
df -h

1
software/ccollect/ccollect-0.3.1/conf/sources/with_exec/source Normal file
View file

@ -0,0 +1 @@
/home/user/nico/oeffentlich/computer/projekte/ccollect-0.3/

17
software/ccollect/ccollect-0.3.1/doc/CHANGES Normal file
View file

@ -0,0 +1,17 @@
0.3 to 0.3.1:
* added support for printing a summary
* some cosmetic changes
0.2 to 0.3:
* added "very_verbose"
* normal "verbose" is now less verbose
* added general 'pre_exec' and 'post_exec' support
* added source specfifc 'pre_exec' and 'post_exec' support
0.1 to 0.2:
* Added plausibility check
* Updated and made documentation readable
* implemented verbose option
* Fixed double exclude parameter bug
* Added much better documentation (asciidoc)
* added rsync extra parameter option

34
software/ccollect/ccollect-0.3.1/doc/TODO Normal file
View file

@ -0,0 +1,34 @@
- update documentation:
- exit pre/post exec -> error codes (after implementation!)
- summary support -> release 0.3.1
- write mkccollectconfig
o create source configuration
o another script for changing defaults
x intervalls
x pre-/post exec
o dialog based?
- implement pre- and post-exec commands
o what to do with return values?
- Documentation
- write/generate a manpage
- rsync_options \n seperated
DONE
- implement verbosity
o per source (-v to rsync)
o general, very verbose (set -x)
- implement parallel execution
- implement general log
- implement source specific log (canceled)
- the name prefix is currently somehow inconsistent
- Documentation
- rsync_options, example paswd
- write a simple manual
- Hints section
- add note to --exclude=/proc/ vs. --exclude=/proc/*
- implement pre- and post-exec commands
o For the general backup process
o source specific

795
software/ccollect/ccollect-0.3.1/doc/ccollect.html Normal file
View file

@ -0,0 +1,795 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta name="generator" content="AsciiDoc 7.0.2" />
<style type="text/css">
/* Debug borders */
p, li, dt, dd, div, pre, h1, h2, h3, h4, h5, h6 {
/*
border: 1px solid red;
*/
}
body {
margin: 1em 5% 1em 5%;
}
a { color: blue; }
a:visited { color: fuchsia; }
em {
font-style: italic;
}
strong {
font-weight: bold;
}
tt {
color: navy;
}
h1, h2, h3, h4, h5, h6 {
color: #527bbd;
font-family: sans-serif;
margin-top: 1.2em;
margin-bottom: 0.5em;
line-height: 1.3;
}
h1 {
border-bottom: 2px solid silver;
}
h2 {
border-bottom: 2px solid silver;
padding-top: 0.5em;
}
div.sectionbody {
font-family: serif;
margin-left: 0;
}
hr {
border: 1px solid silver;
}
p {
margin-top: 0.5em;
margin-bottom: 0.5em;
}
pre {
padding: 0;
margin: 0;
}
span#author {
color: #527bbd;
font-family: sans-serif;
font-weight: bold;
font-size: 1.2em;
}
span#email {
}
span#revision {
font-family: sans-serif;
}
div#footer {
font-family: sans-serif;
font-size: small;
border-top: 2px solid silver;
padding-top: 0.5em;
margin-top: 4.0em;
}
div#footer-text {
float: left;
padding-bottom: 0.5em;
}
div#footer-badges {
float: right;
padding-bottom: 0.5em;
}
div#preamble,
div.tableblock, div.imageblock, div.exampleblock, div.verseblock,
div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
div.admonitionblock {
margin-right: 10%;
margin-top: 1.5em;
margin-bottom: 1.5em;
}
div.admonitionblock {
margin-top: 2.5em;
margin-bottom: 2.5em;
}
div.content { /* Block element content. */
padding: 0;
}
/* Block element titles. */
div.title, caption.title {
font-family: sans-serif;
font-weight: bold;
text-align: left;
margin-top: 1.0em;
margin-bottom: 0.5em;
}
div.title + * {
margin-top: 0;
}
td div.title:first-child {
margin-top: 0.0em;
}
div.content div.title:first-child {
margin-top: 0.0em;
}
div.content + div.title {
margin-top: 0.0em;
}
div.sidebarblock > div.content {
background: #ffffee;
border: 1px solid silver;
padding: 0.5em;
}
div.listingblock > div.content {
border: 1px solid silver;
background: #f4f4f4;
padding: 0.5em;
}
div.quoteblock > div.content {
padding-left: 2.0em;
}
div.quoteblock .attribution {
text-align: right;
}
div.admonitionblock .icon {
vertical-align: top;
font-size: 1.1em;
font-weight: bold;
text-decoration: underline;
color: #527bbd;
padding-right: 0.5em;
}
div.admonitionblock td.content {
padding-left: 0.5em;
border-left: 2px solid silver;
}
div.exampleblock > div.content {
border-left: 2px solid silver;
padding: 0.5em;
}
div.verseblock div.content {
white-space: pre;
}
div.imageblock div.content { padding-left: 0; }
div.imageblock img { border: 1px solid silver; }
span.image img { border-style: none; }
dl {
margin-top: 0.8em;
margin-bottom: 0.8em;
}
dt {
margin-top: 0.5em;
margin-bottom: 0;
font-style: italic;
}
dd > *:first-child {
margin-top: 0;
}
ul, ol {
list-style-position: outside;
}
ol.olist2 {
list-style-type: lower-alpha;
}
div.tableblock > table {
border-color: #527bbd;
border-width: 3px;
}
thead {
font-family: sans-serif;
font-weight: bold;
}
tfoot {
font-weight: bold;
}
div.hlist {
margin-top: 0.8em;
margin-bottom: 0.8em;
}
td.hlist1 {
vertical-align: top;
font-style: italic;
padding-right: 0.8em;
}
td.hlist2 {
vertical-align: top;
}
@media print {
div#footer-badges { display: none; }
}
/* Workarounds for IE6's broken and incomplete CSS2. */
div.sidebar-content {
background: #ffffee;
border: 1px solid silver;
padding: 0.5em;
}
div.sidebar-title, div.image-title {
font-family: sans-serif;
font-weight: bold;
margin-top: 0.0em;
margin-bottom: 0.5em;
}
div.listingblock div.content {
border: 1px solid silver;
background: #f4f4f4;
padding: 0.5em;
}
div.quoteblock-content {
padding-left: 2.0em;
}
div.exampleblock-content {
border-left: 2px solid silver;
padding-left: 0.5em;
}
</style>
<title>ccollect - Installing, Configuring and Using</title>
</head>
<body>
<div id="header">
<h1>ccollect - Installing, Configuring and Using</h1>
<span id="author">Nico Schottelius</span><br />
<span id="email"><tt>&lt;<a href="mailto:nico-linux-ccollect__@__schottelius.org">nico-linux-ccollect__@__schottelius.org</a>&gt;</tt></span><br />
<span id="revision">version 0.3.1,</span>
for ccollect 0.3.1, Initial Version from 2005-01-13
</div>
<div id="preamble">
<div class="sectionbody">
<p>(pseudo) incremental backup
with different exclude lists
using hardlinks and <tt>rsync</tt></p>
</div>
</div>
<h2>1. Introduction</h2>
<div class="sectionbody">
<p>ccollect is a backup utitily written in the sh-scripting language.
It does not depend on a specific shell, only <tt>/bin/sh</tt> needs to be
bourne shell compatibel (like <em>dash</em>, <em>ksh</em>, <em>zsh</em>, <em>bash</em>, &#8230;).</p>
<h3>1.1. Why you can only backup TO localhost</h3>
<p>While thinking about the design of ccollect, I thought about enabling
backup to <strong>remote</strong> hosts. Though this sounds like a nice feature
(<em>Backup my notebook to the server now.</em>), it is in my opinion a
bad idea to backup to a remote host, because you have to open
security at your backup host. Think of the following situation: You backup
your farm of webservers <strong>to</strong> a backup host somewhere else. One of
your webservers gets compromised, then your backup server will be compromised,
too. Think of it the other way round: The backup server (now behind a
firewall using NAT and strong firewall rules) connects to the
webservers and pulls the data to it. If someone gets access to the
webserver, the person will perhaps not even see your machine. If
he/she sees that there are connections from a host to the compromised
machine, he/she will not be able to login to the backup machine.
All other backups are still secure.</p>
</div>
<h2>2. Requirements</h2>
<div class="sectionbody">
<h3>2.1. Installing ccollect</h3>
<p>For the installation, you need at least</p>
<ul>
<li>
<p>
either <tt>cp</tt> and <tt>chmod</tt> or <tt>install</tt>
</p>
</li>
<li>
<p>
for more comfort: <tt>make</tt>
</p>
</li>
<li>
<p>
for rebuilding the generated documentation: additionally <tt>asciidoc</tt>
</p>
</li>
</ul>
<h3>2.2. Using ccollect</h3>
<div class="title">Running ccollect requires the following tools installed:</div><ul>
<li>
<p>
<tt>bc</tt>
</p>
</li>
<li>
<p>
<tt>cp</tt> with support for hard links (<em>cp -al</em>)
</p>
</li>
<li>
<p>
<tt>rsync</tt>
</p>
</li>
<li>
<p>
<tt>ssh</tt> (if you want to use rsync over ssh, which is recommened for security)
</p>
</li>
</ul>
</div>
<h2>3. Installing</h2>
<div class="sectionbody">
<p>Either type <em>make install</em> or simply copy it to a directory in your
$PATH and execute <em>chmod <strong>0755</strong> /path/to/ccollect.sh</em>.</p>
</div>
<h2>4. Configuring</h2>
<div class="sectionbody">
<h3>4.1. Runtime options</h3>
<p><tt>ccollect</tt> looks for its configuration in <em>/etc/ccollect</em> or, if set, in
the directory specified by the variable <em>$CCOLLECT_CONF</em>
(use <em>CCOLLECT_CONF=/your/config/dir ccollect.sh</em> on the shell).</p>
<p>When you start <tt>ccollect</tt>, you have either to specify which intervall
to backup (daily, weekly, yearly; you can specify the names yourself, see below).</p>
<p>The intervall is used to specify how many backups to keep.</p>
<p>There are also some self explaining parameters you can pass to ccollect, simply use
"ccollect.sh &#8212;help" for info.</p>
<h3>4.2. General configuration</h3>
<p>The general configuration can be found below $CCOLLECT_CONF/defaults or
/etc/ccollect/defaults. All options specified here are generally valid for
all source definitions. Though the values can be overwritten in the source
configuration.</p>
<p>All configuration entries are plain-text (use UTF-8 if you use
non ASCII characters) files.</p>
<h4>4.2.1. Intervall definition</h4>
<p>The intervall definition can be found below
<em>$CCOLLECT_CONF/defaults/intervalls/</em> or <em>/etc/ccollect/defaults/intervalls</em>.
Every file below this directory specifies an intervall. The name of the file is the
name of the intervall: <tt>intervalls/<em>&lt;intervall name&gt;</em></tt>.</p>
<p>The content of this file should be a single line containing a number.
This number defines how many versions of this intervall to keep.</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt> [10:23] zaphodbeeblebrox:ccollect-0.2% ls -l conf/defaults/intervalls/
insgesamt 12
-rw-r--r-- 1 nico users 3 2005-12-08 10:24 daily
-rw-r--r-- 1 nico users 3 2005-12-08 11:36 monthly
-rw-r--r-- 1 nico users 2 2005-12-08 11:36 weekly
[10:23] zaphodbeeblebrox:ccollect-0.2% cat conf/defaults/intervalls/*
28
12
4</tt></pre>
</div></div>
<p>This means to keep 28 daily backups, 12 monthly backups and 4 weekly.</p>
<h4>4.2.2. General pre- and post-execution</h4>
<p>If you add <em>$CCOLLECT_CONF/defaults/<tt>pre_exec</tt></em> or
<em>/etc/ccollect/defaults/<tt>pre_exec</tt></em> (same with <tt>post_exec</tt>), <tt>ccollect</tt>
will start <tt>pre_exec</tt> before the whole backup process and
<tt>post_exec</tt> after backup of all sources is done.</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt>[13:00] hydrogenium:~# mkdir -p /etc/ccollect/defaults/
[13:00] hydrogenium:~# echo '!/bin/sh' &gt; /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# echo '' &gt; /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# echo '!/bin/sh' &gt; /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# echo '' &gt;&gt; /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# echo 'df -h' &gt;&gt; /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# chmod 0755 /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# cp /etc/ccollect/defaults/pre_exec /etc/ccollect/defaults/post_exec</tt></pre>
</div></div>
<h3>4.3. Source configuration</h3>
<p>Each source configuration exists below <em>$CCOLLECT_CONF/sources/$name</em> or
<em>/etc/ccollect/sources/$name</em>.</p>
<p>The name you choose for the subdirectory describes the source.</p>
<p>Each source has at least the following files:</p>
<ul>
<li>
<p>
<tt>source</tt> (a text file containing the <tt>rsync</tt> compatible path to backup)
</p>
</li>
<li>
<p>
<tt>destination</tt> (a link to the directory we should backup to)
</p>
</li>
</ul>
<p>Additionally a source may have the following files:</p>
<ul>
<li>
<p>
<tt>verbose</tt> whether to be verbose (passes -v to <tt>rsync</tt>)
</p>
</li>
<li>
<p>
<tt>very_verbose</tt> be very verbose (-v also for <tt>mkdir</tt>, <tt>cp</tt>, <tt>rm</tt>)
</p>
</li>
<li>
<p>
<tt>summary</tt> create a transfer summary when <tt>rsync</tt> finished
</p>
</li>
<li>
<p>
<tt>exclude</tt> exclude list for <tt>rsync</tt>. newline (<em>\n</em>) seperated list.
</p>
</li>
<li>
<p>
<tt>rsync_options' extra options to pass to `rsync</tt>
</p>
</li>
<li>
<p>
<tt>pre_exec</tt> program to execute before backuping this source
</p>
</li>
<li>
<p>
<tt>post_exec</tt> program to execute after backuping this source
</p>
</li>
</ul>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt> [10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2
insgesamt 12
lrwxrwxrwx 1 nico users 20 2005-11-17 16:44 destination -&gt; /home/nico/backupdir
-rw-r--r-- 1 nico users 62 2005-12-07 17:43 exclude
drwxr-xr-x 2 nico users 4096 2005-12-07 17:38 intervalls
-rw-r--r-- 1 nico users 15 2005-11-17 16:44 source
[10:47] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli
[10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervalls
insgesamt 4
-rw-r--r-- 1 nico users 2 2005-12-07 17:38 daily
[10:48] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/intervalls/daily
5
[10:48] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/source
/home/nico/vpn</tt></pre>
</div></div>
<h4>4.3.1. Detailled description of "source"</h4>
<p><tt>source</tt> describes a <tt>rsync</tt> compatible source (one line only).</p>
<p>For instance <em>backup_user@foreign_host:/home/server/video</em>.
To use the <tt>rsync</tt> protocol without the <tt>ssh</tt>-tunnel, use
<em>rsync::USER@HOST/SRC</em>. For more information have a look at <tt>rsync</tt>(1).</p>
<h4>4.3.2. Detailled description of "verbose"</h4>
<p><tt>verbose</tt> tells <tt>ccollect</tt> that the log should contain verbose messages.</p>
<p>If this file exists in the source specification <strong>-v</strong> will be passed to <tt>rsync</tt>.</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt> [11:35] zaphodbeeblebrox:ccollect-0.2% touch conf/sources/testsource1/verbose</tt></pre>
</div></div>
<h4>4.3.3. Detailled description of "very_verbose"</h4>
<p><tt>very_verbose</tt> tells <tt>ccollect</tt> that it should log very verbose.</p>
<p>If this file exists in the source specification <strong>-v</strong> will be passed to
<tt>rsync</tt>, <tt>cp</tt>, <tt>rm</tt> and <tt>mkdir</tt>.</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt> [23:67] nohost:~% touch conf/sources/testsource1/very_verbose</tt></pre>
</div></div>
<h4>4.3.4. Detailled description of "summary"</h4>
<p>If you create the file <tt>summary</tt> below the source definition,
<tt>ccollect</tt> will present you with a nice summary at the end.</p>
<div class="listingblock">
<div class="content">
<pre><tt>backup:~# touch /etc/ccollect/sources/root/summary
backup:~# ccollect.sh werktags root
==&gt; ccollect.sh: Beginning backup using intervall werktags &lt;==
[root] Beginning to backup this source ...
[root] Currently 3 backup(s) exist, total keeping 50 backup(s).
[root] Beginning to backup, this may take some time...
[root] Hard linking...
[root] Transferring files...
[root]
[root] Number of files: 84183
[root] Number of files transferred: 32
[root] Total file size: 26234080536 bytes
[root] Total transferred file size: 9988252 bytes
[root] Literal data: 9988252 bytes
[root] Matched data: 0 bytes
[root] File list size: 3016771
[root] File list generation time: 1.786 seconds
[root] File list transfer time: 0.000 seconds
[root] Total bytes sent: 13009119
[root] Total bytes received: 2152
[root]
[root] sent 13009119 bytes received 2152 bytes 2891393.56 bytes/sec
[root] total size is 26234080536 speedup is 2016.26
[root] Successfully finished backup.
==&gt; Finished ccollect.sh &lt;==</tt></pre>
</div></div>
<p>You could also combine it with <tt>verbose</tt> or <tt>very_verbose</tt>, but they
already print some statistics (but not all / the same as presented by
<tt>summary</tt>).</p>
<h4>4.3.5. Detailled description of "exclude"</h4>
<p><tt>exclude</tt> specifies a list of paths to exclude. The entries are new line (\n)
seperated.</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt> [11:35] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli
something with spaces is not a problem</tt></pre>
</div></div>
<h4>4.3.6. Detailled description of "destination"</h4>
<p><tt>destination</tt> must be a link to the destination directory.</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt> [11:36] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/destination
lrwxrwxrwx 1 nico users 20 2005-11-17 16:44 conf/sources/testsource2/destination -&gt; /home/nico/backupdir</tt></pre>
</div></div>
<h4>4.3.7. Detailled description of "intervalls/"</h4>
<p>When you create a subdirectory <tt>intervalls/</tt> within your source configuration
directory, you can specify individiual intervalls for this specific source.
Each file below this directory describes an intervall.</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt> [11:37] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervalls/
insgesamt 8
-rw-r--r-- 1 nico users 2 2005-12-07 17:38 daily
-rw-r--r-- 1 nico users 3 2005-12-14 11:33 yearly
[11:37] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/intervalls/*
5
20</tt></pre>
</div></div>
<h4>4.3.8. Detailled description of "rsync_options"</h4>
<p>When you create the file <tt>rsync_options</tt> below your source configuration,
all the parameters found in this file will be passed to rsync. This
way you can pass additional options to rsync. For instance you can tell rsync
to show progress ("&#8212;progress") or which -password-file ("&#8212;password-file")
to use for automatic backup over the rsync-protocol.</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt> [23:42] hydrogenium:ccollect-0.2% cat conf/sources/test_rsync/rsync_options
--password-file=/home/user/backup/protected_password_file</tt></pre>
</div></div>
<h4>4.3.9. Detailled description of "pre_exec" and "post_exec"</h4>
<p>When you create <tt>pre_exec</tt> and / or <tt>post_exec</tt> below your source
configuration, <tt>ccollect</tt> will execute this command before,
respective after doing the backup for <strong>this specific</strong> source.
If you want to have pre-/post-exec before and after <strong>all</strong>
backups, see above for general configuration.</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt>[13:09] hydrogenium:ccollect-0.3% cat conf/sources/with_exec/pre_exec
#!/bin/sh
# Show whats free before
df -h
[13:09] hydrogenium:ccollect-0.3% cat conf/sources/with_exec/post_exec
#!/bin/sh
# Show whats free after
df -h</tt></pre>
</div></div>
</div>
<h2>5. Hints</h2>
<div class="sectionbody">
<h3>5.1. Using rsync protocol without ssh</h3>
<p>When you have a computer with little computing power, it may be useful to use
rsync without ssh, directly using the rsync protocol
(specify <em>user@host::share</em> in <tt>source</tt>). You may wish to use
<tt>rsync_options</tt> to specify a password file to use for automatic backup.</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt>backup:~# cat /etc/ccollect/sources/sample.backup.host.org/source
backup@webserver::backup-share
backup:~# cat /etc/ccollect/sources/sample.backup.host.org/rsync_options
--password-file=/etc/ccollect/sources/sample.backup.host.org/rsync_password
backup:~# cat /etc/ccollect/sources/sample.backup.host.org/rsync_password
this_is_the_rsync_password</tt></pre>
</div></div>
<p>This hint was reported by Daniel Aubry.</p>
<h3>5.2. Not-excluding top-level directories</h3>
<p>When you exclude "/proc" or "/mnt" from your backup, you may run into
trouble when you restore your backup. When you use "/proc/*" or "/mnt/*"
instead <tt>ccollect</tt> will backup empty directories.</p>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">
<p>When those directories contain hidden files
(those beginning with a dot (<strong>.</strong>)),
they will still be transferred!</p>
</td>
</tr></table>
</div>
<p>This hint was reported by Marcus Wagner.</p>
<h3>5.3. Re-using already created rsync-backups</h3>
<p>If you used <tt>rsync</tt> directly before you use <tt>ccollect</tt>, you can
use this old backup as initial backup for <tt>ccollect</tt>: You
simply move it into a subdirectory named "<em>intervall</em>.0".</p>
<p>Example:</p>
<div class="listingblock">
<div class="content">
<pre><tt>backup:/home/backup/web1# ls
bin dev etc initrd lost+found mnt root srv usr vmlinuz
boot doc home lib media opt sbin tmp var vmlinuz.old
backup:/home/backup/web1# mkdir daily.0
# ignore error about copying to itself
backup:/home/backup/web1# mv * daily.0 2&gt;/dev/null
backup:/home/backup/web1# ls
daily.0</tt></pre>
</div></div>
<p>Now you could use /home/backup/web1 as the <tt>destination</tt> for the backup.</p>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">
<p>Do <strong>not</strong> name the first backup something like "daily.initial", but use
the "<strong>0</strong>" (or some very low number, at least lower than the current year)
as extension. <tt>ccollect</tt> uses <tt>sort</tt> to find the latest backup. <tt>ccollect</tt>
itself uses <em>intervall.YEAR-MONTH-DAY-HOUR:MINUTE.PID</em>. This notation will
<strong>always</strong> be before "daily.initial", as numbers are earlier in the list
which is produced by <tt>sort</tt>. So, if you have a directory named "daily.initial",
<tt>ccollect</tt> will always diff against this backup and transfer and delete
files which where deleted in previous backups. This means you simply
waste resources, but your backup will be complete.</p>
</td>
</tr></table>
</div>
<h3>5.4. Using pre_/post_exec</h3>
<p>Your pre_/post_exec script does not need to be a script, you can also
use a link to</p>
<ul>
<li>
<p>
an existing program
</p>
</li>
<li>
<p>
an already written script
</p>
</li>
</ul>
<p>The only requirement is that it is executable.</p>
</div>
<h2>6. F.A.Q.</h2>
<div class="sectionbody">
<h3>6.1. What happens, if one backup is broken or empty?</h3>
<p>Let us assume, that one backup failed (connection broke or hard disk had
some failures). So we've one backup in our history, which is incomplete.</p>
<p>The next time you use <tt>ccollect</tt>, it will transfer the missing files</p>
<h3>6.2. When backing up from localhost the destination is also included. Is this a bug?</h3>
<p>No. <tt>ccollect</tt> passes your source definition directly to <tt>rsync</tt>. It
does not try to analyze it. So it actually does not know if a source
comes from local harddisk or from a remote server. And it does not want
to. When you backup from the local harddisk (which is perhaps not
even a good idea when thinking of security) add the <tt>destination</tt>
to <em>source/exclude</em>. (Daniel Aubry reported this problem)</p>
<h3>6.3. Why does ccollect say "Permission denied" with my pre-/postexec script?</h3>
<p>The most common error is to not give your script the correct
permissions. Try <tt>chmod 0755 <em>/etc/ccollect/sources/yoursource/*_exec</em></tt>.</p>
</div>
<h2>7. Examples</h2>
<div class="sectionbody">
<h3>7.1. A backup host configuration from scratch</h3>
<div class="listingblock">
<div class="content">
<pre><tt>srwali01:~# mkdir /etc/ccollect
srwali01:~# mkdir -p /etc/ccollect/defaults/intervalls/
srwali01:~# echo 28 &gt; /etc/ccollect/defaults/intervalls/taeglich
srwali01:~# echo 52 &gt; /etc/ccollect/defaults/intervalls/woechentlich
srwali01:~# cd /etc/ccollect/
srwali01:/etc/ccollect# mkdir sources
srwali01:/etc/ccollect# cd sources/
srwali01:/etc/ccollect/sources# ls
srwali01:/etc/ccollect/sources# mkdir local-root
srwali01:/etc/ccollect/sources# cd local-root/
srwali01:/etc/ccollect/sources/local-root# echo / &gt; source
srwali01:/etc/ccollect/sources/local-root# cat &gt; exclude &lt;&lt; EOF
&gt; /proc
&gt; /sys
&gt; /mnt
&gt; EOF
srwali01:/etc/ccollect/sources/local-root# ln -s /mnt/hdbackup/local-root destination
srwali01:/etc/ccollect/sources/local-root# mkdir /mnt/hdbackup/local-root
srwali01:/etc/ccollect/sources/local-root# ccollect.sh taeglich local-root
/o&gt; ccollect.sh: Beginning backup using intervall taeglich
/=&gt; Beginning to backup "local-root" ...
|-&gt; 0 backup(s) already exist, keeping 28 backup(s).</tt></pre>
</div></div>
<p>After that, I added some more sources:</p>
<div class="listingblock">
<div class="content">
<pre><tt>srwali01:~# cd /etc/ccollect/sources
srwali01:/etc/ccollect/sources# mkdir windos-wl6
srwali01:/etc/ccollect/sources# cd windos-wl6/
srwali01:/etc/ccollect/sources/windos-wl6# echo /mnt/win/SYS/WL6 &gt; source
srwali01:/etc/ccollect/sources/windos-wl6# ln -s /mnt/hdbackup/wl6 destination
srwali01:/etc/ccollect/sources/windos-wl6# mkdir /mnt/hdbackup/wl6
srwali01:/etc/ccollect/sources/windos-wl6# cd ..
srwali01:/etc/ccollect/sources# mkdir windos-daten
srwali01:/etc/ccollect/sources/windos-daten# echo /mnt/win/Daten &gt; source
srwali01:/etc/ccollect/sources/windos-daten# ln -s /mnt/hdbackup/windos-daten destination
srwali01:/etc/ccollect/sources/windos-daten# mkdir /mnt/hdbackup/windos-daten
# Now add some remote source
srwali01:/etc/ccollect/sources/windos-daten# cd ..
srwali01:/etc/ccollect/sources# mkdir srwali03
srwali01:/etc/ccollect/sources# cd srwali03/
srwali01:/etc/ccollect/sources/srwali03# cat &gt; exclude &lt;&lt; EOF
&gt; /proc
&gt; /sys
&gt; /mnt
&gt; /home
&gt; EOF
srwali01:/etc/ccollect/sources/srwali03# echo 'root@10.103.2.3:/' &gt; source
srwali01:/etc/ccollect/sources/srwali03# ln -s /mnt/hdbackup/srwali03 destination
srwali01:/etc/ccollect/sources/srwali03# mkdir /mnt/hdbackup/srwali03</tt></pre>
</div></div>
<h3>7.2. Using hard-links requires less disk space</h3>
<div class="listingblock">
<div class="content">
<pre><tt>[10:53] srsyg01:sources% du -sh ~/backupdir
4.6M /home/nico/backupdir
[10:53] srsyg01:sources% du -sh ~/backupdir/*
4.1M /home/nico/backupdir/daily.2005-12-08-10:52.28456
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28484
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28507
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28531
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28554
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28577
srwali01:/etc/ccollect/sources# du -sh /mnt/hdbackup/wl6/
186M /mnt/hdbackup/wl6/
srwali01:/etc/ccollect/sources# du -sh /mnt/hdbackup/wl6/*
147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:42.312
147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:45.588</tt></pre>
</div></div>
</div>
<div id="footer">
<div id="footer-text">
Version 0.3.1<br />
Last updated 25-Jan-2006 00:12:06 CEST
</div>
</div>
</body>
</html>

530
software/ccollect/ccollect-0.3.1/doc/ccollect.text Normal file
View file

@ -0,0 +1,530 @@
ccollect - Installing, Configuring and Using
============================================
Nico Schottelius <nico-linux-ccollect__@__schottelius.org>
0.3.1, for ccollect 0.3.1, Initial Version from 2005-01-13
:Author Initials: NS
(pseudo) incremental backup
with different exclude lists
using hardlinks and `rsync`
Introduction
------------
ccollect is a backup utitily written in the sh-scripting language.
It does not depend on a specific shell, only `/bin/sh` needs to be
bourne shell compatibel (like 'dash', 'ksh', 'zsh', 'bash', ...).
Why you can only backup TO localhost
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
While thinking about the design of ccollect, I thought about enabling
backup to *remote* hosts. Though this sounds like a nice feature
('Backup my notebook to the server now.'), it is in my opinion a
bad idea to backup to a remote host, because you have to open
security at your backup host. Think of the following situation: You backup
your farm of webservers *to* a backup host somewhere else. One of
your webservers gets compromised, then your backup server will be compromised,
too. Think of it the other way round: The backup server (now behind a
firewall using NAT and strong firewall rules) connects to the
webservers and pulls the data to it. If someone gets access to the
webserver, the person will perhaps not even see your machine. If
he/she sees that there are connections from a host to the compromised
machine, he/she will not be able to login to the backup machine.
All other backups are still secure.
Requirements
------------
Installing ccollect
~~~~~~~~~~~~~~~~~~~
For the installation, you need at least
- either `cp` and `chmod` or `install`
- for more comfort: `make`
- for rebuilding the generated documentation: additionally `asciidoc`
Using ccollect
~~~~~~~~~~~~~~
.Running ccollect requires the following tools installed:
- `bc`
- `cp` with support for hard links ('cp -al')
- `rsync`
- `ssh` (if you want to use rsync over ssh, which is recommened for security)
Installing
----------
Either type 'make install' or simply copy it to a directory in your
$PATH and execute 'chmod *0755* /path/to/ccollect.sh'.
Configuring
-----------
Runtime options
~~~~~~~~~~~~~~~
`ccollect` looks for its configuration in '/etc/ccollect' or, if set, in
the directory specified by the variable '$CCOLLECT_CONF'
(use 'CCOLLECT_CONF=/your/config/dir ccollect.sh' on the shell).
When you start `ccollect`, you have either to specify which intervall
to backup (daily, weekly, yearly; you can specify the names yourself, see below).
The intervall is used to specify how many backups to keep.
There are also some self explaining parameters you can pass to ccollect, simply use
"ccollect.sh --help" for info.
General configuration
~~~~~~~~~~~~~~~~~~~~~
The general configuration can be found below $CCOLLECT_CONF/defaults or
/etc/ccollect/defaults. All options specified here are generally valid for
all source definitions. Though the values can be overwritten in the source
configuration.
All configuration entries are plain-text (use UTF-8 if you use
non ASCII characters) files.
Intervall definition
^^^^^^^^^^^^^^^^^^^^
The intervall definition can be found below
'$CCOLLECT_CONF/defaults/intervalls/' or '/etc/ccollect/defaults/intervalls'.
Every file below this directory specifies an intervall. The name of the file is the
name of the intervall: `intervalls/'<intervall name>'`.
The content of this file should be a single line containing a number.
This number defines how many versions of this intervall to keep.
Example:
-------------------------------------------------------------------------
[10:23] zaphodbeeblebrox:ccollect-0.2% ls -l conf/defaults/intervalls/
insgesamt 12
-rw-r--r-- 1 nico users 3 2005-12-08 10:24 daily
-rw-r--r-- 1 nico users 3 2005-12-08 11:36 monthly
-rw-r--r-- 1 nico users 2 2005-12-08 11:36 weekly
[10:23] zaphodbeeblebrox:ccollect-0.2% cat conf/defaults/intervalls/*
28
12
4
--------------------------------------------------------------------------------
This means to keep 28 daily backups, 12 monthly backups and 4 weekly.
General pre- and post-execution
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you add '$CCOLLECT_CONF/defaults/`pre_exec`' or
'/etc/ccollect/defaults/`pre_exec`' (same with `post_exec`), `ccollect`
will start `pre_exec` before the whole backup process and
`post_exec` after backup of all sources is done.
Example:
-------------------------------------------------------------------------
[13:00] hydrogenium:~# mkdir -p /etc/ccollect/defaults/
[13:00] hydrogenium:~# echo '!/bin/sh' > /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# echo '' > /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# echo '!/bin/sh' > /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# echo '' >> /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# echo 'df -h' >> /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# chmod 0755 /etc/ccollect/defaults/pre_exec
[13:01] hydrogenium:~# cp /etc/ccollect/defaults/pre_exec /etc/ccollect/defaults/post_exec
-------------------------------------------------------------------------
Source configuration
~~~~~~~~~~~~~~~~~~~~
Each source configuration exists below '$CCOLLECT_CONF/sources/$name' or
'/etc/ccollect/sources/$name'.
The name you choose for the subdirectory describes the source.
Each source has at least the following files:
- `source` (a text file containing the `rsync` compatible path to backup)
- `destination` (a link to the directory we should backup to)
Additionally a source may have the following files:
- `verbose` whether to be verbose (passes -v to `rsync`)
- `very_verbose` be very verbose (-v also for `mkdir`, `cp`, `rm`)
- `summary` create a transfer summary when `rsync` finished
- `exclude` exclude list for `rsync`. newline ('\n') seperated list.
- `rsync_options' extra options to pass to `rsync`
- `pre_exec` program to execute before backuping this source
- `post_exec` program to execute after backuping this source
Example:
--------------------------------------------------------------------------------
[10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2
insgesamt 12
lrwxrwxrwx 1 nico users 20 2005-11-17 16:44 destination -> /home/nico/backupdir
-rw-r--r-- 1 nico users 62 2005-12-07 17:43 exclude
drwxr-xr-x 2 nico users 4096 2005-12-07 17:38 intervalls
-rw-r--r-- 1 nico users 15 2005-11-17 16:44 source
[10:47] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli
[10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervalls
insgesamt 4
-rw-r--r-- 1 nico users 2 2005-12-07 17:38 daily
[10:48] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/intervalls/daily
5
[10:48] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/source
/home/nico/vpn
--------------------------------------------------------------------------------
Detailled description of "source"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`source` describes a `rsync` compatible source (one line only).
For instance 'backup_user@foreign_host:/home/server/video'.
To use the `rsync` protocol without the `ssh`-tunnel, use
'rsync::USER@HOST/SRC'. For more information have a look at `rsync`(1).
Detailled description of "verbose"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`verbose` tells `ccollect` that the log should contain verbose messages.
If this file exists in the source specification *-v* will be passed to `rsync`.
Example:
--------------------------------------------------------------------------------
[11:35] zaphodbeeblebrox:ccollect-0.2% touch conf/sources/testsource1/verbose
--------------------------------------------------------------------------------
Detailled description of "very_verbose"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`very_verbose` tells `ccollect` that it should log very verbose.
If this file exists in the source specification *-v* will be passed to
`rsync`, `cp`, `rm` and `mkdir`.
Example:
--------------------------------------------------------------------------------
[23:67] nohost:~% touch conf/sources/testsource1/very_verbose
--------------------------------------------------------------------------------
Detailled description of "summary"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you create the file `summary` below the source definition,
`ccollect` will present you with a nice summary at the end.
-------------------------------------------------------------------------------
backup:~# touch /etc/ccollect/sources/root/summary
backup:~# ccollect.sh werktags root
==> ccollect.sh: Beginning backup using intervall werktags <==
[root] Beginning to backup this source ...
[root] Currently 3 backup(s) exist, total keeping 50 backup(s).
[root] Beginning to backup, this may take some time...
[root] Hard linking...
[root] Transferring files...
[root]
[root] Number of files: 84183
[root] Number of files transferred: 32
[root] Total file size: 26234080536 bytes
[root] Total transferred file size: 9988252 bytes
[root] Literal data: 9988252 bytes
[root] Matched data: 0 bytes
[root] File list size: 3016771
[root] File list generation time: 1.786 seconds
[root] File list transfer time: 0.000 seconds
[root] Total bytes sent: 13009119
[root] Total bytes received: 2152
[root]
[root] sent 13009119 bytes received 2152 bytes 2891393.56 bytes/sec
[root] total size is 26234080536 speedup is 2016.26
[root] Successfully finished backup.
==> Finished ccollect.sh <==
-------------------------------------------------------------------------------
You could also combine it with `verbose` or `very_verbose`, but they
already print some statistics (but not all / the same as presented by
`summary`).
Detailled description of "exclude"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`exclude` specifies a list of paths to exclude. The entries are new line (\n)
seperated.
Example:
--------------------------------------------------------------------------------
[11:35] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude
openvpn-2.0.1.tar.gz
nicht_reinnehmen
etwas mit leerzeichenli
something with spaces is not a problem
--------------------------------------------------------------------------------
Detailled description of "destination"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`destination` must be a link to the destination directory.
Example:
--------------------------------------------------------------------------------
[11:36] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/destination
lrwxrwxrwx 1 nico users 20 2005-11-17 16:44 conf/sources/testsource2/destination -> /home/nico/backupdir
--------------------------------------------------------------------------------
Detailled description of "intervalls/"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When you create a subdirectory `intervalls/` within your source configuration
directory, you can specify individiual intervalls for this specific source.
Each file below this directory describes an intervall.
Example:
--------------------------------------------------------------------------------
[11:37] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervalls/
insgesamt 8
-rw-r--r-- 1 nico users 2 2005-12-07 17:38 daily
-rw-r--r-- 1 nico users 3 2005-12-14 11:33 yearly
[11:37] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/intervalls/*
5
20
--------------------------------------------------------------------------------
Detailled description of "rsync_options"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When you create the file `rsync_options` below your source configuration,
all the parameters found in this file will be passed to rsync. This
way you can pass additional options to rsync. For instance you can tell rsync
to show progress ("--progress") or which -password-file ("--password-file")
to use for automatic backup over the rsync-protocol.
Example:
--------------------------------------------------------------------------------
[23:42] hydrogenium:ccollect-0.2% cat conf/sources/test_rsync/rsync_options
--password-file=/home/user/backup/protected_password_file
--------------------------------------------------------------------------------
Detailled description of "pre_exec" and "post_exec"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When you create `pre_exec` and / or `post_exec` below your source
configuration, `ccollect` will execute this command before,
respective after doing the backup for *this specific* source.
If you want to have pre-/post-exec before and after *all*
backups, see above for general configuration.
Example:
--------------------------------------------------------------------------------
[13:09] hydrogenium:ccollect-0.3% cat conf/sources/with_exec/pre_exec
#!/bin/sh
# Show whats free before
df -h
[13:09] hydrogenium:ccollect-0.3% cat conf/sources/with_exec/post_exec
#!/bin/sh
# Show whats free after
df -h
--------------------------------------------------------------------------------
Hints
-----
Using rsync protocol without ssh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When you have a computer with little computing power, it may be useful to use
rsync without ssh, directly using the rsync protocol
(specify 'user@host::share' in `source`). You may wish to use
`rsync_options` to specify a password file to use for automatic backup.
Example:
--------------------------------------------------------------------------------
backup:~# cat /etc/ccollect/sources/sample.backup.host.org/source
backup@webserver::backup-share
backup:~# cat /etc/ccollect/sources/sample.backup.host.org/rsync_options
--password-file=/etc/ccollect/sources/sample.backup.host.org/rsync_password
backup:~# cat /etc/ccollect/sources/sample.backup.host.org/rsync_password
this_is_the_rsync_password
--------------------------------------------------------------------------------
This hint was reported by Daniel Aubry.
Not-excluding top-level directories
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When you exclude "/proc" or "/mnt" from your backup, you may run into
trouble when you restore your backup. When you use "/proc/\*" or "/mnt/\*"
instead `ccollect` will backup empty directories.
[NOTE]
===========================================
When those directories contain hidden files
(those beginning with a dot (*.*)),
they will still be transferred!
===========================================
This hint was reported by Marcus Wagner.
Re-using already created rsync-backups
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you used `rsync` directly before you use `ccollect`, you can
use this old backup as initial backup for `ccollect`: You
simply move it into a subdirectory named "'intervall'.0".
Example:
-------------------------------------------------------------------------------
backup:/home/backup/web1# ls
bin dev etc initrd lost+found mnt root srv usr vmlinuz
boot doc home lib media opt sbin tmp var vmlinuz.old
backup:/home/backup/web1# mkdir daily.0
# ignore error about copying to itself
backup:/home/backup/web1# mv * daily.0 2>/dev/null
backup:/home/backup/web1# ls
daily.0
-------------------------------------------------------------------------------
Now you could use /home/backup/web1 as the `destination` for the backup.
[NOTE]
===============================================================================
Do *not* name the first backup something like "daily.initial", but use
the "*0*" (or some very low number, at least lower than the current year)
as extension. `ccollect` uses `sort` to find the latest backup. `ccollect`
itself uses 'intervall.YEAR-MONTH-DAY-HOUR:MINUTE.PID'. This notation will
*always* be before "daily.initial", as numbers are earlier in the list
which is produced by `sort`. So, if you have a directory named "daily.initial",
`ccollect` will always diff against this backup and transfer and delete
files which where deleted in previous backups. This means you simply
waste resources, but your backup will be complete.
===============================================================================
Using pre_/post_exec
~~~~~~~~~~~~~~~~~~~~
Your pre_/post_exec script does not need to be a script, you can also
use a link to
- an existing program
- an already written script
The only requirement is that it is executable.
F.A.Q.
------
What happens, if one backup is broken or empty?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Let us assume, that one backup failed (connection broke or hard disk had
some failures). So we've one backup in our history, which is incomplete.
The next time you use `ccollect`, it will transfer the missing files
When backing up from localhost the destination is also included. Is this a bug?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
No. `ccollect` passes your source definition directly to `rsync`. It
does not try to analyze it. So it actually does not know if a source
comes from local harddisk or from a remote server. And it does not want
to. When you backup from the local harddisk (which is perhaps not
even a good idea when thinking of security) add the `destination`
to 'source/exclude'. (Daniel Aubry reported this problem)
Why does ccollect say "Permission denied" with my pre-/postexec script?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The most common error is to not give your script the correct
permissions. Try `chmod 0755 '/etc/ccollect/sources/yoursource/*_exec'`.
Examples
--------
A backup host configuration from scratch
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--------------------------------------------------------------------------------
srwali01:~# mkdir /etc/ccollect
srwali01:~# mkdir -p /etc/ccollect/defaults/intervalls/
srwali01:~# echo 28 > /etc/ccollect/defaults/intervalls/taeglich
srwali01:~# echo 52 > /etc/ccollect/defaults/intervalls/woechentlich
srwali01:~# cd /etc/ccollect/
srwali01:/etc/ccollect# mkdir sources
srwali01:/etc/ccollect# cd sources/
srwali01:/etc/ccollect/sources# ls
srwali01:/etc/ccollect/sources# mkdir local-root
srwali01:/etc/ccollect/sources# cd local-root/
srwali01:/etc/ccollect/sources/local-root# echo / > source
srwali01:/etc/ccollect/sources/local-root# cat > exclude << EOF
> /proc
> /sys
> /mnt
> EOF
srwali01:/etc/ccollect/sources/local-root# ln -s /mnt/hdbackup/local-root destination
srwali01:/etc/ccollect/sources/local-root# mkdir /mnt/hdbackup/local-root
srwali01:/etc/ccollect/sources/local-root# ccollect.sh taeglich local-root
/o> ccollect.sh: Beginning backup using intervall taeglich
/=> Beginning to backup "local-root" ...
|-> 0 backup(s) already exist, keeping 28 backup(s).
--------------------------------------------------------------------------------
After that, I added some more sources:
--------------------------------------------------------------------------------
srwali01:~# cd /etc/ccollect/sources
srwali01:/etc/ccollect/sources# mkdir windos-wl6
srwali01:/etc/ccollect/sources# cd windos-wl6/
srwali01:/etc/ccollect/sources/windos-wl6# echo /mnt/win/SYS/WL6 > source
srwali01:/etc/ccollect/sources/windos-wl6# ln -s /mnt/hdbackup/wl6 destination
srwali01:/etc/ccollect/sources/windos-wl6# mkdir /mnt/hdbackup/wl6
srwali01:/etc/ccollect/sources/windos-wl6# cd ..
srwali01:/etc/ccollect/sources# mkdir windos-daten
srwali01:/etc/ccollect/sources/windos-daten# echo /mnt/win/Daten > source
srwali01:/etc/ccollect/sources/windos-daten# ln -s /mnt/hdbackup/windos-daten destination
srwali01:/etc/ccollect/sources/windos-daten# mkdir /mnt/hdbackup/windos-daten
# Now add some remote source
srwali01:/etc/ccollect/sources/windos-daten# cd ..
srwali01:/etc/ccollect/sources# mkdir srwali03
srwali01:/etc/ccollect/sources# cd srwali03/
srwali01:/etc/ccollect/sources/srwali03# cat > exclude << EOF
> /proc
> /sys
> /mnt
> /home
> EOF
srwali01:/etc/ccollect/sources/srwali03# echo 'root@10.103.2.3:/' > source
srwali01:/etc/ccollect/sources/srwali03# ln -s /mnt/hdbackup/srwali03 destination
srwali01:/etc/ccollect/sources/srwali03# mkdir /mnt/hdbackup/srwali03
--------------------------------------------------------------------------------
Using hard-links requires less disk space
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-------------------------------------------------------------------------
[10:53] srsyg01:sources% du -sh ~/backupdir
4.6M /home/nico/backupdir
[10:53] srsyg01:sources% du -sh ~/backupdir/*
4.1M /home/nico/backupdir/daily.2005-12-08-10:52.28456
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28484
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28507
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28531
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28554
4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28577
srwali01:/etc/ccollect/sources# du -sh /mnt/hdbackup/wl6/
186M /mnt/hdbackup/wl6/
srwali01:/etc/ccollect/sources# du -sh /mnt/hdbackup/wl6/*
147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:42.312
147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:45.588
-------------------------------------------------------------------------

23
software/ccollect/ccollect-0.3.1/release.sh Normal file
View file

@ -0,0 +1,23 @@
if [ $# -ne 1 ]; then
echo "$0: ccollect dir"
exit 23
fi
NAME=$1
TARNAME=${NAME}.tar.bz2
DHOST=nico@creme.schottelius.org
DDIR=www/org/schottelius/linux/ccollect/
DESTINATION="$DHOST:$DDIR"
tar cvfj "$TARNAME" \
--exclude=.git \
--exclude="conf/sources/*/destination/*" "$NAME"
scp "${TARNAME}" "$DESTINATION"
ssh "$DHOST" "( cd $DDIR; tar xfj \"$TARNAME\" )"
echo "setting paranoid permissions to public..."
ssh "$DHOST" "( cd $DDIR; find -type d -exec chmod 0755 {} \; )"
ssh "$DHOST" "( cd $DDIR; find -type f -exec chmod 0644 {} \; )"

BIN
software/ccollect/ccollect-0.3.2.tar.bz2 Normal file
View file

Binary file not shown.

2
software/ccollect/ccollect-0.3.2/.gitignore vendored Normal file
View file

@ -0,0 +1,2 @@
conf/sources/*/destination/*
doc/old

340
software/ccollect/ccollect-0.3.2/COPYING Normal file
View file

@ -0,0 +1,340 @@
GNU GENERAL PUBLIC LICENSE
Version 2, June 1991
Copyright (C) 1989, 1991 Free Software Foundation, Inc.
51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users. This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it. (Some other Free Software Foundation software is covered by
the GNU Library General Public License instead.) You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.
To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have. You must make sure that they, too, receive or can get the
source code. And you must show them these terms so they know their
rights.
We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.
Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software. If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.
Finally, any free program is threatened constantly by software
patents. We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary. To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.
The precise terms and conditions for copying, distribution and
modification follow.
GNU GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License. The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language. (Hereinafter, translation is included without limitation in
the term "modification".) Each licensee is addressed as "you".
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.
1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.
You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.
2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) You must cause the modified files to carry prominent notices
stating that you changed the files and the date of any change.
b) You must cause any work that you distribute or publish, that in
whole or in part contains or is derived from the Program or any
part thereof, to be licensed as a whole at no charge to all third
parties under the terms of this License.
c) If the modified program normally reads commands interactively
when run, you must cause it, when started running for such
interactive use in the most ordinary way, to print or display an
announcement including an appropriate copyright notice and a
notice that there is no warranty (or else, saying that you provide
a warranty) and that users may redistribute the program under
these conditions, and telling the user how to view a copy of this
License. (Exception: if the Program itself is interactive but
does not normally print such an announcement, your work based on
the Program is not required to print an announcement.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.
In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:
a) Accompany it with the complete corresponding machine-readable
source code, which must be distributed under the terms of Sections
1 and 2 above on a medium customarily used for software interchange; or,
b) Accompany it with a written offer, valid for at least three
years, to give any third party, for a charge no more than your
cost of physically performing source distribution, a complete
machine-readable copy of the corresponding source code, to be
distributed under the terms of Sections 1 and 2 above on a medium
customarily used for software interchange; or,
c) Accompany it with the information you received as to the offer
to distribute corresponding source code. (This alternative is
allowed only for noncommercial distribution and only if you
received the program in object code or executable form with such
an offer, in accord with Subsection b above.)
The source code for a work means the preferred form of the work for
making modifications to it. For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable. However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.
If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.
4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License. Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.
5. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Program or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.
6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.
7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all. For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.
If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded. In such case, this License incorporates
the limitation as if written in the body of this License.
9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation. If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.
10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission. For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this. Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.
NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
Also add information on how to contact you by electronic and paper mail.
If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:
Gnomovision version 69, Copyright (C) year name of author
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
`Gnomovision' (which makes passes at compilers) written by James Hacker.
<signature of Ty Coon>, 1 April 1989
Ty Coon, President of Vice
This General Public License does not permit incorporating your program into
proprietary programs. If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library. If this is what you want to do, use the GNU Library General
Public License instead of this License.

53
software/ccollect/ccollect-0.3.2/Makefile Normal file
View file

@ -0,0 +1,53 @@
#
# ccollect
# Nico Schottelius, Fri Jan 13 12:13:08 CET 2006
#
INSTALL=install
CCOLLECT=ccollect.sh
LN=ln -sf
prefix=/usr/packages/ccollect-0.2
bindir=$(prefix)/bin
destination=$(bindir)/$(CCOLLECT)
path_dir=/usr/local/bin
path_destination=$(path_dir)/$(CCOLLECT)
# where to publish
host=creme.schottelius.org
dir=www/org/schottelius/linux/ccollect
docdir=$(dir)/doc
#
# End user targets
#
all:
@echo "Nothing to make, make install."
install: install-script install-link
install-link: install-script
$(LN) $(destination) $(path_destination)
install-script:
$(INSTALL) -D -m 0755 -s $(CCOLLECT) $(destination)
documentation:
@echo "Generating HTML-documentation"
@asciidoc -n -o doc/ccollect.html doc/ccollect.text
#
# Developer targets
#
update:
@cg-update creme
push-work:
@cg-push creme
@cg-push main
publish-doc: documentation
@chmod a+r doc/ccollect.html
@scp doc/ccollect.html doc/ccollect.text $(host):$(docdir)

23
software/ccollect/ccollect-0.3.2/README Normal file
View file

@ -0,0 +1,23 @@
--------------------------------------------------------------------------------
ccollect.sh, Nico Schottelius, 2005-12-06
--------------------------------------------------------------------------------
ccollect backups data from local or remote hosts to your local harddisk.
You can retriev the latest version of ccollect at [0].
doc/ccollect.text Manual in text format
doc/ccollect.html Manual in xhtml
ccollect was inspired by rsnapshot [1], which had some problems:
- configuration parameters had to be TAB seperated
- you could not specify exclude lists differently for every source
- no per source pre/post execution support
- no parallel execution
- I didn't like the configuration at all, so I used the cconfig style [2].
[0]: ccollect: http://linux.schottelius.org/ccollect/
[1]: rsnapshot: http://www.rsnapshot.org/
[2]: cconfig: http://nico.schotteli.us/papers/linux/cconfig/

413
software/ccollect/ccollect-0.3.2/ccollect.sh Normal file
View file

@ -0,0 +1,413 @@
#!/bin/sh
# Nico Schottelius
# written for SyGroup (www.sygroup.ch)
# Date: Mon Nov 14 11:45:11 CET 2005
# Last Modified: (See ls -l or git)
#
# where to find our configuration and temporary file
#
CCOLLECT_CONF=${CCOLLECT_CONF:-/etc/ccollect}
CSOURCES=$CCOLLECT_CONF/sources
CDEFAULTS=$CCOLLECT_CONF/defaults
CPREEXEC="$CDEFAULTS/pre_exec"
CPOSTEXEC="$CDEFAULTS/post_exec"
TMP=$(mktemp /tmp/$(basename $0).XXXXXX)
VERSION=0.3.2
RELEASE="2006-02-05"
HALF_VERSION="ccollect $VERSION"
FULL_VERSION="ccollect $VERSION ($RELEASE)"
#
# unset parallel execution
#
PARALLEL=""
#
# catch signals
#
trap "rm -f \"$TMP\"" 1 2 15
add_name()
{
sed "s/^/\[$name\] /"
}
#
# Tell how to use us
#
usage()
{
echo "$(basename $0): <intervall name> [args] <sources to backup>"
echo ""
echo " ccollect creates (pseudo) incremental backups"
echo ""
echo " -h, --help: Show this help screen"
echo " -p, --parallel: Parallelise backup processes"
echo " -a, --all: Backup all sources specified in $CSOURCES"
echo " -v, --verbose: Be very verbose (uses set -x)."
echo ""
echo " This is version $VERSION, released on ${RELEASE}"
echo " (the first version was written on 2005-12-05 by Nico Schottelius)."
echo ""
echo " Retrieve latest ccollect at http://linux.schottelius.org/ccollect/"
exit 0
}
#
# need at least intervall and one source or --all
#
if [ $# -lt 2 ]; then
usage
fi
#
# check for configuraton directory
#
if [ ! -d "$CCOLLECT_CONF" ]; then
echo "No configuration found in \"$CCOLLECT_CONF\"" \
" (set \$CCOLLECT_CONF corectly?)"
exit 1
fi
#
# Filter arguments
#
INTERVALL=$1; shift
i=1
no_shares=0
while [ $i -le $# ]; do
eval arg=\$$i
if [ "$NO_MORE_ARGS" = 1 ]; then
eval share_${no_shares}=\"$arg\"
no_shares=$(($no_shares+1))
else
case $arg in
-a|--all)
ALL=1
;;
-v|--verbose)
VERBOSE=1
;;
-p|--parallel)
PARALLEL="1"
;;
-h|--help)
usage
;;
--)
NO_MORE_ARGS=1
;;
*)
eval share_${no_shares}=\"$arg\"
no_shares=$(($no_shares+1))
;;
esac
fi
i=$(($i+1))
done
#
# be really really really verbose
#
if [ "$VERBOSE" = 1 ]; then
set -x
fi
#
# Look, if we should take ALL sources
#
if [ "$ALL" = 1 ]; then
# reset everything specified before
no_shares=0
#
# get entries from sources
#
cwd=$(pwd)
cd "$CSOURCES";
ls > "$TMP"
while read tmp; do
eval share_${no_shares}=\"$tmp\"
no_shares=$(($no_shares+1))
done < "$TMP"
fi
#
# Need at least ONE source to backup
#
if [ "$no_shares" -lt 1 ]; then
usage
else
echo "==> $HALF_VERSION: Beginning backup using intervall $INTERVALL <=="
fi
#
# check default configuration
#
D_FILE_INTERVALL="$CDEFAULTS/intervalls/$INTERVALL"
D_INTERVALL=$(cat $D_FILE_INTERVALL 2>/dev/null)
#
# Look for pre-exec command (general)
#
if [ -x "$CPREEXEC" ]; then
echo "Executing $CPREEXEC ..."
"$CPREEXEC"
echo "Finished ${CPREEXEC}."
fi
#
# Let's do the backup
#
i=0
while [ "$i" -lt "$no_shares" ]; do
#
# Get current share
#
eval name=\$share_${i}
i=$(($i+1))
export name
#
# start ourself, if we want parallel execution
#
if [ "$PARALLEL" ]; then
$0 "$INTERVALL" "$name" &
continue
fi
#
# Start subshell for easy log editing
#
(
#
# Stderr to stdout, so we can produce nice logs
#
exec 2>&1
#
# Standard locations
#
backup="$CSOURCES/$name"
c_source="$backup/source"
c_dest="$backup/destination"
c_exclude="$backup/exclude"
c_verbose="$backup/verbose"
c_vverbose="$backup/very_verbose"
c_rsync_extra="$backup/rsync_options"
c_summary="$backup/summary"
c_pre_exec="$backup/pre_exec"
c_post_exec="$backup/post_exec"
begin=$(date)
begin_s=$(date +%s)
echo "$begin Beginning to backup"
#
# Standard configuration checks
#
if [ ! -e "$backup" ]; then
echo "Source does not exist."
exit 1
fi
if [ ! -d "$backup" ]; then
echo "\"$name\" is not a cconfig-directory. Skipping."
exit 1
fi
#
# intervall definition: First try source specific, fallback to default
#
c_intervall="$(cat "$backup/intervalls/$INTERVALL" 2>/dev/null)"
if [ -z "$c_intervall" ]; then
c_intervall=$D_INTERVALL
if [ -z "$c_intervall" ]; then
echo "Default and source specific intervall missing. Skipping."
exit 1
fi
fi
#
# unset possible options
#
EXCLUDE=""
RSYNC_EXTRA=""
SUMMARY=""
VERBOSE=""
VVERBOSE=""
#
# next configuration checks
#
if [ ! -f "$c_source" ]; then
echo "Source description $c_source is not a file. Skipping."
exit 1
else
source=$(cat "$c_source")
if [ $? -ne 0 ]; then
echo "Skipping: Source $c_source is not readable"
exit 1
fi
fi
if [ ! -d "$c_dest" ]; then
echo "Destination $c_dest does not link to a directory. Skipping"
exit 1
fi
#
# pre_exec
#
if [ -x "$c_pre_exec" ]; then
echo "Executing $c_pre_exec ..."
$c_pre_exec
echo "Finished ${c_pre_exec}."
fi
# exclude
if [ -f "$c_exclude" ]; then
EXCLUDE="--exclude-from=$c_exclude"
fi
# extra options for rsync
if [ -f "$c_rsync_extra" ]; then
RSYNC_EXTRA="$(cat "$c_rsync_extra")"
fi
# verbosity for rsync
if [ -f "$c_verbose" ]; then
VERBOSE="-v"
fi
# Output a summary
if [ -f "$c_summary" ]; then
SUMMARY="--stats"
fi
# MORE verbosity, includes standard verbosity
if [ -f "$c_vverbose" ]; then
VERBOSE="-v"
VVERBOSE="-v"
fi
#
# check if maximum number of backups is reached, if so remove
#
# the created directories are named $INTERVALL.$DA
count=$(ls -d "$c_dest/${INTERVALL}."?* 2>/dev/null | wc -l)
echo "Currently $count backup(s) exist, total keeping $c_intervall backup(s)."
if [ "$count" -ge "$c_intervall" ]; then
substract=$(echo $c_intervall - 1 | bc)
remove=$(echo $count - $substract | bc)
echo "Removing $remove backup(s)..."
ls -d "$c_dest/${INTERVALL}."?* | sort -n | head -n $remove > "$TMP"
while read to_remove; do
dir="$to_remove"
echo "Removing $dir ..."
rm $VVERBOSE -rf "$dir"
done < "$TMP"
fi
#
# clone the old directory with hardlinks
#
destination_date=$(date +%Y-%m-%d-%H:%M)
destination_dir="$c_dest/${INTERVALL}.${destination_date}.$$"
last_dir=$(ls -d "$c_dest/${INTERVALL}."?* 2>/dev/null | sort -n | tail -n 1)
# give some info
echo "Beginning to backup, this may take some time..."
# only copy if a directory exists
if [ "$last_dir" ]; then
echo "Hard linking..."
cp -al $VVERBOSE "$last_dir" "$destination_dir"
else
echo "Creating $destination_dir"
mkdir $VVERBOSE "$destination_dir"
fi
if [ $? -ne 0 ]; then
echo "Creating/cloning backup directory failed. Skipping backup."
exit 1
fi
#
# the rsync part
# options partly stolen from rsnapshot
#
echo "Transferring files..."
rsync -a $VERBOSE $RSYNC_EXTRA $EXCLUDE $SUMMARY \
--delete --numeric-ids --relative --delete-excluded \
"$source" "$destination_dir"
if [ "$?" -ne 0 ]; then
echo "rsync reported an error. The backup may be broken (see rsync errors)"
exit 1
fi
echo "$(date) Successfully finished backup"
#
# post_exec
#
if [ -x "$c_post_exec" ]; then
echo "$(date) Executing $c_post_exec ..."
"$c_post_exec"
echo "$(date) Finished ${c_post_exec}."
fi
end_s=$(date +%s)
full_seconds=$(echo "$end_s - $begin_s" | bc -l)
hours=$(echo $full_seconds / 3600 | bc)
seconds=$(echo "$full_seconds - ($hours * 3600)" | bc)
minutes=$(echo $seconds / 60 | bc)
seconds=$(echo "$seconds - ($minutes * 60)" | bc)
echo "Backup lasted: ${hours}:$minutes:$seconds (h:m:s)"
) | add_name
done
#
# Be a good parent and wait for our children, if they are running wild parallel
#
if [ "$PARALLEL" ]; then
echo "Waiting for child jobs to complete..."
wait
fi
#
# Look for post-exec command (general)
#
if [ -x "$CPOSTEXEC" ]; then
echo "Executing $CPOSTEXEC ..."
"$CPOSTEXEC"
echo "Finished ${CPOSTEXEC}."
fi
rm -f "$TMP"
echo "==> Finished $WE <=="

1
software/ccollect/ccollect-0.3.2/conf/defaults/intervalls/daily Normal file
View file

@ -0,0 +1 @@
28

Some files were not shown because too many files have changed in this diff Show more