Commit graph

768 commits

Author SHA1 Message Date
Nico Schottelius
de6a7893fc [DOC] Move German documentation to old/ (unmaintained)
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-01 10:03:00 +02:00
Nico Schottelius
194148b5b3 update url of homepage
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-01 10:00:18 +02:00
jll2
6fd22b6416 Move "interval definition section".
The interval definition section was down to just before the
maximum backup check.  This makes the code more friendly to
automatic interval selection.  Auto interval selection needs to
have ddir defined first and it is best if it is done after
delete_incomplete.  This change accomplishes that while still
placing it before the maximum backup check which needs to know
the interval.
2009-06-26 15:22:09 -07:00
jll2
72830a4647 If the user specifies "delete_incomplete", this patch makes it
so all incomplete backups are deleted, not just the ones with
the particular interval that the user specified.

The advantage of this is that those to-be-deleted incomplete
backups will not interfere with calculations required for
automatic interval selection.
2009-06-26 14:54:28 -07:00
jll2
76e6094247 Simplify interval code.
Eight lines and two variables are removed which makes the code,
I think, easier to read.

The main motivation for this change, however, is that it makes
ccollect.sh more friendly to (future) auto interval selection.
The removed lines and variables assumed that the interval was
known prior to the start of the source loop.  With auto interval
selection, the selected interval can be different for each
source.
2009-06-26 14:42:45 -07:00
jll2
0b064e0565 Beautify: remove trailing white space.
If your editor does not highlight trailing white space,
you won't see a difference.
2009-06-26 14:09:21 -07:00
jll2
dd7a047408 Add option quiet_if_down.
If a source is not connectable, ccollect.sh issues a series of error
messages such as:

$ ccollect.sh  "int 1" dummy
2009-06-25-21:04:14: ccollect 0.7.1: Beginning backup using interval int 1
[dummy] 2009-06-25-21:04:14: Beginning to backup
[dummy] ssh: connect to host Ha port 20: No route to host
[dummy] rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
[dummy] rsync error: unexplained error (code 255) at io.c(600) [receiver=3.0.5]
[dummy] 2009-06-25-21:04:17: Error: source Ha:/tmp is not readable. Skipping.
2009-06-25-21:04:17: Finished

If you expect the source to be up, you want to see these messages.
However, for a notebook computer or other portable machine, it may be
normal for it to be disconnected.  If quiet_if_down is specified for
that source, then the ssh and rsync errors are suppressed and the
"Error:" prefix is removed from the "skipping" message:

$ ccollect.sh  "int 1" dummy
2009-06-25-21:03:33: ccollect 0.7.1: Beginning backup using interval int 1
[dummy] 2009-06-25-21:03:34: Beginning to backup
[dummy] 2009-06-25-21:03:37: Source Ha:/tmp is not readable. Skipping.
2009-06-25-21:03:37: Finished

I considered the alternative implementation of adding the logic to
ccollect_analyse_logs.sh to enable it to separate rsync messages
generated the initial connection test from messages generated by
rsync used for an actual backup data transfer.  Adding this approach
to ccollect.sh appeared much simpler.
2009-06-25 21:34:42 -07:00
jll2
010449bafa Add option to sort backup directories based on modification time.
By default, ccollect.sh sorts backup directories based on last change
time (ctime).  This adds the option to sort based on modification
time (mtime).

I have updated doc/ccollect.text but it needs some work to simplify
and explain the issue.
2009-06-25 20:35:13 -07:00
jll2
97df2c14de Update the destination directory's mtime with each backup.
After rsync, the destination directory's mtime reflects the
modification time of its immediate contents.  This patch overrides
that and sets the mtime to the time that the backup finished.

With this patch, the age of a backup can be assessed by looking at
its mtime.  The advantages of this are (1) that mtime can be
preserved, via cp -a or rsync -a, when copying a backup repository
to a new hard disk or a new machine and (2) that incorrect mtimes,
such as might happen after a user meddles with his backup
repository, can be, via touch, corrected.  The disadvantage is that
mtime for the immediate contents of the destination directory is
lost.
2009-06-25 16:20:00 -07:00
jll2
923350907d Merge branch 'master' of git@github.com:jll2/ccollect into updates 2009-06-24 19:58:20 -07:00
jll2
544a7d269e Create rsync_failure_codes option.
User may optionally create a file rsync_failure_codes with a
newline-separated list of rsync return codes that are to be
regarded as complete failure.

If rsync exits with such a code, then the backup will be marked
for deletion during the next ccollect run (if delete_incomplete).

I added documentation for this feature in doc/ccollect.text

In my experience (yours may differ), two rsync exit codes that
belong in this file are 12 and 255.  I have seen 12 result from
ssh errors and 255 result from a kernel module conflict.  In both
cases, the resulting backups were empty.  Without the
rsync_failure_codes option, such errors cause good backups to be
deleted, as per c_interval, leaving behind the new empty backups.

In the long run, we may want a different and more comprehensive
method for analyzing rsync errors.  In the short run, I find this
option necessary.
2009-06-24 17:01:14 -07:00
jll2
cd643f1c0b Merge branch 'master' of git://github.com/telmich/ccollect 2009-06-24 11:11:39 -07:00
jll2
64b5ae8b03 Defensive programming patch:
"set -u" was added to make sure that there will never be misspelled or
uninitialized variables in ccollect.sh.
2009-06-24 18:42:39 +02:00
jll2
142fd24fc8 Add ARM to GNU/Linux architecture list
I run ccollect nightly on a Thecus N2100 running Debian-ARM.

The Linksys NSLU2 also runs an ARM processor and I have lightly tested
ccollect on that under SlugOS.
2009-06-22 12:54:06 -07:00
jll2
5477b39a25 Merge branch 'master' of git://github.com/telmich/ccollect 2009-06-21 14:06:18 -07:00
Nico Schottelius
c9439be432 add automatic backup manager CREDITS
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-06-21 00:50:04 +02:00
Nico Schottelius
2b28567588 Add README for contrib
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-06-21 00:15:21 +02:00
Nico Schottelius
cbff479c65 Add John to the CREDITS list
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-06-21 00:15:04 +02:00
jll2
d6ea94c6dc Fix delete_incomplete bugs:
1).  On systems I tried, delete_incomplete failed because the line:

< pcmd rm $VVERBOSE -rf "${ddir}/${realincomplete}" || \

should read:

> pcmd rm $VVERBOSE -rf "${realincomplete}" || \

(Is this true of all systems?)

2).  The marker file was not deleted.  Code was added to delete it.

3).  The delete_incomplete code was simplified.
2009-06-20 21:06:10 +02:00
Nico Schottelius
4db6b78a13 Correctly sort CREDITS by alphabet
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-06-20 21:05:49 +02:00
Nico Schottelius
10d420614c [DOC] give some hints on how to hack ccollect
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-06-20 21:00:21 +02:00
jll2
ea16af51b2 Simplify handling of command-line verbose option:
Previously, there was a $VERBOSE script variable that was set according to
the "-v" command line option and then reset to null within each source
sub-shell.  With no loss of functionality, this patch removes all references
to that variable.

This makes the script 13 lines shorter.
2009-06-20 11:42:49 +02:00
jll2
a4c61e7b68 Bug fix for very_verbose:
According to the documentation, "if [the very_verbose] file exists in the
source specification -v will be passed to rsync, rm and mkdir."  Previously,
the -v option was passed only to rsync.  This patch passes it to rm and mkdir
as well.

Actually, as per the behavior of the previous version, it is verbose that
sends the -v option to rsync while very_verbose sends it the -vv option.  I
left it this way because this behavior seems reasonable.  Maybe the
documentation should be corrected on this point.
2009-06-20 11:40:01 +02:00
jll2
192b55b98d Initialize four variables to prevent unwanted interaction with user's environment. 2009-06-20 11:37:14 +02:00
jll2
122982b0b9 Modifies ccollect.sh's interpretation of options:
1).  If an option doesn't exist in a source directory, check the defaults directory.
2).  For every option, create a corresponding "no_" option so that a source directory
     can override an option set in defaults.
(i.patch)
2009-06-20 11:36:37 +02:00
jll2
f2aef9d4dd Merge branch 'master' of git://github.com/telmich/ccollect 2009-06-18 10:24:28 -07:00
Nico Schottelius
b121e545f7 WARNING: THIS TREE WAS REWRITTEN
I added some patches from John with an E-Mail address he
does not to be public on the internet.

I did not ask him before whether this is fine, so I screwed
up (similar to the description in git-tag(1)).

Thus I replaced his e-mail and would like you to accept
this forced push and remove old trees on the net.

Thanks,

   Nico

Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-06-18 10:02:39 +02:00
Nico Schottelius
f4f9564bde add backup manager from Jens-Christoph Brendel
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-06-16 10:34:49 +02:00
Nico Schottelius
6595fe7b97 add updated patches from john
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-06-16 10:32:05 +02:00
Nico Schottelius
2b31f8f229 add changes for the next release
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-06-10 09:56:13 +02:00
Nico Schottelius
02264020f5 add changes for the next release
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-06-10 09:56:13 +02:00
Nico Schottelius
382c159b41 Beautify a comment
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-06-10 09:55:38 +02:00
Nico Schottelius
ba538ea623 Beautify a comment
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-06-10 09:55:38 +02:00
Nico Schottelius
62e8190a94 Introduce consistent time sorting
Based on patches by John Lawless <jlawless@redwoodscientific.com>.
Skipped the sort changing part (from -tc to -t)

c.patch:

--- ccollect-0.7.1-b.sh	2009-05-24 21:32:00.000000000 -0700
+++ ccollect-0.7.1-c.sh	2009-05-24 21:39:43.000000000 -0700
@@ -40,10 +40,13 @@
 VERSION=0.7.1
 RELEASE="2009-02-02"
 HALF_VERSION="ccollect ${VERSION}"
 FULL_VERSION="ccollect ${VERSION} (${RELEASE})"

+#TSORT="tc" ; NEWER="cnewer"
+TSORT="t" ; NEWER="newer"
+
 #
 # CDATE: how we use it for naming of the archives
 # DDATE: how the user should see it in our output (DISPLAY)
 #
 CDATE="date +%Y%m%d-%H%M"
@@ -513,14 +516,14 @@

    #
    # Check for backup directory to clone from: Always clone from the latest one!
    #
-   # Use ls -1c instead of -1t, because last modification maybe the same on all
-   # and metadate update (-c) is updated by rsync locally.
-   #
-   last_dir="$(pcmd ls -tcp1 "${ddir}" | grep '/$' | head -n 1)" || \
+   # Depending on your file system, you may want to sort on:
+   #   1. mtime (modification time) with TSORT=t, or
+   #   2. ctime (last change time, usually) with TSORT=tc
+   last_dir="$(pcmd ls -${TSORT}p1 "${ddir}" | grep '/$' | head -n 1)" || \
       _exit_err "Failed to list contents of ${ddir}."

    #
    # clone from old backup, if existing
    #

d.patch:

--- ccollect-0.7.1-c.sh	2009-05-24 21:39:43.000000000 -0700
+++ ccollect-0.7.1-d.sh	2009-05-24 21:47:09.000000000 -0700
@@ -492,12 +492,12 @@
    if [ "${count}" -ge "${c_interval}" ]; then
       substract=$((${c_interval} - 1))
       remove=$((${count} - ${substract}))
       _techo "Removing ${remove} backup(s)..."

-      pcmd ls -p1 "$ddir" | grep "^${INTERVAL}\..*/\$" | \
-        sort -n | head -n "${remove}" > "${TMP}"      || \
+      pcmd ls -${TSORT}p1r "$ddir" | grep "^${INTERVAL}\..*/\$" | \
+        head -n "${remove}" > "${TMP}"      || \
         _exit_err "Listing old backups failed"

       i=0
       while read to_remove; do
          eval remove_$i=\"${to_remove}\"

Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-06-10 09:50:05 +02:00
Nico Schottelius
bce57a1ac1 Introduce consistent time sorting
Based on patches by John Lawless <jll2_8854b@redwoodscientific.com>.
Skipped the sort changing part (from -tc to -t)

c.patch:

--- ccollect-0.7.1-b.sh	2009-05-24 21:32:00.000000000 -0700
+++ ccollect-0.7.1-c.sh	2009-05-24 21:39:43.000000000 -0700
@@ -40,10 +40,13 @@
 VERSION=0.7.1
 RELEASE="2009-02-02"
 HALF_VERSION="ccollect ${VERSION}"
 FULL_VERSION="ccollect ${VERSION} (${RELEASE})"

+#TSORT="tc" ; NEWER="cnewer"
+TSORT="t" ; NEWER="newer"
+
 #
 # CDATE: how we use it for naming of the archives
 # DDATE: how the user should see it in our output (DISPLAY)
 #
 CDATE="date +%Y%m%d-%H%M"
@@ -513,14 +516,14 @@

    #
    # Check for backup directory to clone from: Always clone from the latest one!
    #
-   # Use ls -1c instead of -1t, because last modification maybe the same on all
-   # and metadate update (-c) is updated by rsync locally.
-   #
-   last_dir="$(pcmd ls -tcp1 "${ddir}" | grep '/$' | head -n 1)" || \
+   # Depending on your file system, you may want to sort on:
+   #   1. mtime (modification time) with TSORT=t, or
+   #   2. ctime (last change time, usually) with TSORT=tc
+   last_dir="$(pcmd ls -${TSORT}p1 "${ddir}" | grep '/$' | head -n 1)" || \
       _exit_err "Failed to list contents of ${ddir}."

    #
    # clone from old backup, if existing
    #

d.patch:

--- ccollect-0.7.1-c.sh	2009-05-24 21:39:43.000000000 -0700
+++ ccollect-0.7.1-d.sh	2009-05-24 21:47:09.000000000 -0700
@@ -492,12 +492,12 @@
    if [ "${count}" -ge "${c_interval}" ]; then
       substract=$((${c_interval} - 1))
       remove=$((${count} - ${substract}))
       _techo "Removing ${remove} backup(s)..."

-      pcmd ls -p1 "$ddir" | grep "^${INTERVAL}\..*/\$" | \
-        sort -n | head -n "${remove}" > "${TMP}"      || \
+      pcmd ls -${TSORT}p1r "$ddir" | grep "^${INTERVAL}\..*/\$" | \
+        head -n "${remove}" > "${TMP}"      || \
         _exit_err "Listing old backups failed"

       i=0
       while read to_remove; do
          eval remove_$i=\"${to_remove}\"

Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-06-10 09:50:05 +02:00
John Lawless
a030a98982 First, I added the following before any old backup gets deleted:
>    # Verify source is up and accepting connections before deleting any old backups
>    rsync "$source" >/dev/null || _exit_err "Source ${source} is not readable. Skipping."

I think that this quick test is a much better than, say, pinging
the source in a pre-exec script: this tests not only that the
source is up and connected to the net, it also verifies (1) that
ssh is up and accepting our key (if we are using ssh), and (2) that
the source directory is mounted (if it needs to be mounted) and
readable.
2009-05-25 08:39:28 +02:00
John Lawless
ae23a04925 First, I added the following before any old backup gets deleted:
>    # Verify source is up and accepting connections before deleting any old backups
>    rsync "$source" >/dev/null || _exit_err "Source ${source} is not readable. Skipping."

I think that this quick test is a much better than, say, pinging
the source in a pre-exec script: this tests not only that the
source is up and connected to the net, it also verifies (1) that
ssh is up and accepting our key (if we are using ssh), and (2) that
the source directory is mounted (if it needs to be mounted) and
readable.
2009-05-25 08:39:28 +02:00
Nico Schottelius
8cc0f04874 add patched version of ccollect-0.7.1 by John Lawless
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-05-23 14:34:09 +02:00
Nico Schottelius
27c838163a add macosx as tested
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-02-04 16:28:57 +01:00
Nico Schottelius
38ca0a1546 Revert "Keep a symlink to the current backup around. This allows us to"
This patch is broken for non local destinations.

This reverts commit 6de3c9877c.
2009-02-04 15:12:27 +01:00
Tonnerre Lombard
6de3c9877c Keep a symlink to the current backup around. This allows us to
establish automatic restore features (e.g. through Puppet file
facts).
2009-02-04 08:25:35 +01:00
Nico Schottelius
1943bfd244 update release script
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-02-02 12:39:42 +01:00
Nico Schottelius
bf22075407 update todo and changes
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-02-02 12:16:04 +01:00
Nico Schottelius
00c1303fb2 add ccollect_list_intervals.sh to the tools
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-02-02 12:14:34 +01:00
Nico Schottelius
0516749a0c update changes
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-02-02 12:10:56 +01:00
Nico Schottelius
c133ba5df9 add release date
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-02-02 12:10:48 +01:00
Nico Schottelius
af242905af add testsource for &-test
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-02-02 11:49:12 +01:00
Nico Schottelius
b3ad86f270 [BUGFIX] allow "&" in sourcepath
Bug description:

From: Tiziano Müller <dev-zero@g.o>
Subject: Notiz: Probleme mit Ampersand in source
Date: Tue, 07 Oct 2008 16:11:10 +0000

Hoi Nico

Kleine Notiz wie im IRC besprochen:

- Hat es in /etc/ccollect/sources/$BACKUPNAME/source ein Ampersand muss
dieses Escaped werden, da sonst folgender Fehler auftritt:
*snip*
[volumes_ForschungEntwicklung] 2008-10-05-03:10:03: Beginning to backup,
this may take some time...
[volumes_ForschungEntwicklung] 2008-10-05-03:10:03:
Creating /home/backup/volumes/Forschung&Entwicklung/daily.20081005-0310.15173 ...
[volumes_ForschungEntwicklung] 2008-10-05-03:10:03: Transferring
files...
[volumes_ForschungEntwicklung] bash: Entwicklung/*: No such file or
directory
[volumes_ForschungEntwicklung] rsync: connection unexpectedly closed (4
bytes received so far) [receiver]
[volumes_ForschungEntwicklung] rsync error: error in rsync protocol data
stream (code 12) at io.c(635) [receiver=3.0.2]
[volumes_ForschungEntwicklung] 2008-10-05-03:10:03: Finished backup
(rsync return code: 12).
[volumes_ForschungEntwicklung] 2008-10-05-03:10:03: Warning: rsync
exited non-zero, the backup may be broken (see rsync errors).
*snip*

- ein "&" im Namen fürs Backup,
also: /etc/ccollect/sources/volumes_Forschung&Entwicklung/ wird beim
Anzeigen (wie oben zu sehen) verschluckt.

Cheers,
Tiziano

Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-02-02 11:47:45 +01:00
Nico Schottelius
337fec115b +tab
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-02-02 11:46:48 +01:00
Nico Schottelius
e5e1cc865a more testing in makefile
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-02-02 11:45:59 +01:00