Commit graph

255 commits

Author SHA1 Message Date
Nico Schottelius
b014c00d24 Remove double code imported with merge
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-27 09:16:35 +02:00
Nico Schottelius
d61c9625f4 Only check configuration options, if the base is exitsing
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-27 09:06:01 +02:00
Nico Schottelius
65c34deb43 remove unecessary whitespace in line 204 :-)
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-27 09:01:13 +02:00
Nico Schottelius
2b890b0316 Merge commit 'john/updates'
Fixed alot of conflicts due to parallel work, but it looks very good now!

Conflicts:
	ccollect.sh
2009-07-24 09:09:16 +02:00
Nico Schottelius
b44fdb6107 some cleanups
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-24 08:58:48 +02:00
Nico Schottelius
ed30a4d25b Remove check for directories with -a
Type of source is checked later anyway and using
ls -p1 results in "dir/", which we should cleanup
with "sed 's;/$;;' again, which is ugly.

Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-23 17:56:41 +02:00
Nico Schottelius
8a87e7effa Move option check below the check for existence of source
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-23 17:53:35 +02:00
Nico Schottelius
f5e1920a15 join source handling for --all and normal
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-23 17:47:54 +02:00
Nico Schottelius
8491a54b0d extract the interval handling from parsing
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-23 17:46:04 +02:00
Nico Schottelius
debdd9d004 BUGFIX: Fix --all handling
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-23 17:39:42 +02:00
Nico Schottelius
37dcda8e3b BUGFIX: ignore -- as parameter
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-23 17:37:55 +02:00
Nico Schottelius
5da5506c65 add quotes to every possible problematic variable, add braces
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-01 22:39:42 +02:00
Nico Schottelius
09ed55a17e only consider directories as sources
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-01 17:09:30 +02:00
Nico Schottelius
a9aad1ed8f correct indent
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-01 17:08:59 +02:00
Nico Schottelius
bd1e365ca0 replace multiple echos with one cat
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-01 12:42:32 +02:00
Nico Schottelius
ca1231a576 begin to implement new style option parsing
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-01 12:03:40 +02:00
Nico Schottelius
194148b5b3 update url of homepage
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-07-01 10:00:18 +02:00
jll2
6fd22b6416 Move "interval definition section".
The interval definition section was down to just before the
maximum backup check.  This makes the code more friendly to
automatic interval selection.  Auto interval selection needs to
have ddir defined first and it is best if it is done after
delete_incomplete.  This change accomplishes that while still
placing it before the maximum backup check which needs to know
the interval.
2009-06-26 15:22:09 -07:00
jll2
72830a4647 If the user specifies "delete_incomplete", this patch makes it
so all incomplete backups are deleted, not just the ones with
the particular interval that the user specified.

The advantage of this is that those to-be-deleted incomplete
backups will not interfere with calculations required for
automatic interval selection.
2009-06-26 14:54:28 -07:00
jll2
76e6094247 Simplify interval code.
Eight lines and two variables are removed which makes the code,
I think, easier to read.

The main motivation for this change, however, is that it makes
ccollect.sh more friendly to (future) auto interval selection.
The removed lines and variables assumed that the interval was
known prior to the start of the source loop.  With auto interval
selection, the selected interval can be different for each
source.
2009-06-26 14:42:45 -07:00
jll2
0b064e0565 Beautify: remove trailing white space.
If your editor does not highlight trailing white space,
you won't see a difference.
2009-06-26 14:09:21 -07:00
jll2
dd7a047408 Add option quiet_if_down.
If a source is not connectable, ccollect.sh issues a series of error
messages such as:

$ ccollect.sh  "int 1" dummy
2009-06-25-21:04:14: ccollect 0.7.1: Beginning backup using interval int 1
[dummy] 2009-06-25-21:04:14: Beginning to backup
[dummy] ssh: connect to host Ha port 20: No route to host
[dummy] rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
[dummy] rsync error: unexplained error (code 255) at io.c(600) [receiver=3.0.5]
[dummy] 2009-06-25-21:04:17: Error: source Ha:/tmp is not readable. Skipping.
2009-06-25-21:04:17: Finished

If you expect the source to be up, you want to see these messages.
However, for a notebook computer or other portable machine, it may be
normal for it to be disconnected.  If quiet_if_down is specified for
that source, then the ssh and rsync errors are suppressed and the
"Error:" prefix is removed from the "skipping" message:

$ ccollect.sh  "int 1" dummy
2009-06-25-21:03:33: ccollect 0.7.1: Beginning backup using interval int 1
[dummy] 2009-06-25-21:03:34: Beginning to backup
[dummy] 2009-06-25-21:03:37: Source Ha:/tmp is not readable. Skipping.
2009-06-25-21:03:37: Finished

I considered the alternative implementation of adding the logic to
ccollect_analyse_logs.sh to enable it to separate rsync messages
generated the initial connection test from messages generated by
rsync used for an actual backup data transfer.  Adding this approach
to ccollect.sh appeared much simpler.
2009-06-25 21:34:42 -07:00
jll2
010449bafa Add option to sort backup directories based on modification time.
By default, ccollect.sh sorts backup directories based on last change
time (ctime).  This adds the option to sort based on modification
time (mtime).

I have updated doc/ccollect.text but it needs some work to simplify
and explain the issue.
2009-06-25 20:35:13 -07:00
jll2
97df2c14de Update the destination directory's mtime with each backup.
After rsync, the destination directory's mtime reflects the
modification time of its immediate contents.  This patch overrides
that and sets the mtime to the time that the backup finished.

With this patch, the age of a backup can be assessed by looking at
its mtime.  The advantages of this are (1) that mtime can be
preserved, via cp -a or rsync -a, when copying a backup repository
to a new hard disk or a new machine and (2) that incorrect mtimes,
such as might happen after a user meddles with his backup
repository, can be, via touch, corrected.  The disadvantage is that
mtime for the immediate contents of the destination directory is
lost.
2009-06-25 16:20:00 -07:00
jll2
923350907d Merge branch 'master' of git@github.com:jll2/ccollect into updates 2009-06-24 19:58:20 -07:00
jll2
544a7d269e Create rsync_failure_codes option.
User may optionally create a file rsync_failure_codes with a
newline-separated list of rsync return codes that are to be
regarded as complete failure.

If rsync exits with such a code, then the backup will be marked
for deletion during the next ccollect run (if delete_incomplete).

I added documentation for this feature in doc/ccollect.text

In my experience (yours may differ), two rsync exit codes that
belong in this file are 12 and 255.  I have seen 12 result from
ssh errors and 255 result from a kernel module conflict.  In both
cases, the resulting backups were empty.  Without the
rsync_failure_codes option, such errors cause good backups to be
deleted, as per c_interval, leaving behind the new empty backups.

In the long run, we may want a different and more comprehensive
method for analyzing rsync errors.  In the short run, I find this
option necessary.
2009-06-24 17:01:14 -07:00
jll2
64b5ae8b03 Defensive programming patch:
"set -u" was added to make sure that there will never be misspelled or
uninitialized variables in ccollect.sh.
2009-06-24 18:42:39 +02:00
jll2
d6ea94c6dc Fix delete_incomplete bugs:
1).  On systems I tried, delete_incomplete failed because the line:

< pcmd rm $VVERBOSE -rf "${ddir}/${realincomplete}" || \

should read:

> pcmd rm $VVERBOSE -rf "${realincomplete}" || \

(Is this true of all systems?)

2).  The marker file was not deleted.  Code was added to delete it.

3).  The delete_incomplete code was simplified.
2009-06-20 21:06:10 +02:00
jll2
ea16af51b2 Simplify handling of command-line verbose option:
Previously, there was a $VERBOSE script variable that was set according to
the "-v" command line option and then reset to null within each source
sub-shell.  With no loss of functionality, this patch removes all references
to that variable.

This makes the script 13 lines shorter.
2009-06-20 11:42:49 +02:00
jll2
a4c61e7b68 Bug fix for very_verbose:
According to the documentation, "if [the very_verbose] file exists in the
source specification -v will be passed to rsync, rm and mkdir."  Previously,
the -v option was passed only to rsync.  This patch passes it to rm and mkdir
as well.

Actually, as per the behavior of the previous version, it is verbose that
sends the -v option to rsync while very_verbose sends it the -vv option.  I
left it this way because this behavior seems reasonable.  Maybe the
documentation should be corrected on this point.
2009-06-20 11:40:01 +02:00
jll2
192b55b98d Initialize four variables to prevent unwanted interaction with user's environment. 2009-06-20 11:37:14 +02:00
jll2
122982b0b9 Modifies ccollect.sh's interpretation of options:
1).  If an option doesn't exist in a source directory, check the defaults directory.
2).  For every option, create a corresponding "no_" option so that a source directory
     can override an option set in defaults.
(i.patch)
2009-06-20 11:36:37 +02:00
Nico Schottelius
ba538ea623 Beautify a comment
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-06-10 09:55:38 +02:00
Nico Schottelius
bce57a1ac1 Introduce consistent time sorting
Based on patches by John Lawless <jll2_8854b@redwoodscientific.com>.
Skipped the sort changing part (from -tc to -t)

c.patch:

--- ccollect-0.7.1-b.sh	2009-05-24 21:32:00.000000000 -0700
+++ ccollect-0.7.1-c.sh	2009-05-24 21:39:43.000000000 -0700
@@ -40,10 +40,13 @@
 VERSION=0.7.1
 RELEASE="2009-02-02"
 HALF_VERSION="ccollect ${VERSION}"
 FULL_VERSION="ccollect ${VERSION} (${RELEASE})"

+#TSORT="tc" ; NEWER="cnewer"
+TSORT="t" ; NEWER="newer"
+
 #
 # CDATE: how we use it for naming of the archives
 # DDATE: how the user should see it in our output (DISPLAY)
 #
 CDATE="date +%Y%m%d-%H%M"
@@ -513,14 +516,14 @@

    #
    # Check for backup directory to clone from: Always clone from the latest one!
    #
-   # Use ls -1c instead of -1t, because last modification maybe the same on all
-   # and metadate update (-c) is updated by rsync locally.
-   #
-   last_dir="$(pcmd ls -tcp1 "${ddir}" | grep '/$' | head -n 1)" || \
+   # Depending on your file system, you may want to sort on:
+   #   1. mtime (modification time) with TSORT=t, or
+   #   2. ctime (last change time, usually) with TSORT=tc
+   last_dir="$(pcmd ls -${TSORT}p1 "${ddir}" | grep '/$' | head -n 1)" || \
       _exit_err "Failed to list contents of ${ddir}."

    #
    # clone from old backup, if existing
    #

d.patch:

--- ccollect-0.7.1-c.sh	2009-05-24 21:39:43.000000000 -0700
+++ ccollect-0.7.1-d.sh	2009-05-24 21:47:09.000000000 -0700
@@ -492,12 +492,12 @@
    if [ "${count}" -ge "${c_interval}" ]; then
       substract=$((${c_interval} - 1))
       remove=$((${count} - ${substract}))
       _techo "Removing ${remove} backup(s)..."

-      pcmd ls -p1 "$ddir" | grep "^${INTERVAL}\..*/\$" | \
-        sort -n | head -n "${remove}" > "${TMP}"      || \
+      pcmd ls -${TSORT}p1r "$ddir" | grep "^${INTERVAL}\..*/\$" | \
+        head -n "${remove}" > "${TMP}"      || \
         _exit_err "Listing old backups failed"

       i=0
       while read to_remove; do
          eval remove_$i=\"${to_remove}\"

Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-06-10 09:50:05 +02:00
John Lawless
ae23a04925 First, I added the following before any old backup gets deleted:
>    # Verify source is up and accepting connections before deleting any old backups
>    rsync "$source" >/dev/null || _exit_err "Source ${source} is not readable. Skipping."

I think that this quick test is a much better than, say, pinging
the source in a pre-exec script: this tests not only that the
source is up and connected to the net, it also verifies (1) that
ssh is up and accepting our key (if we are using ssh), and (2) that
the source directory is mounted (if it needs to be mounted) and
readable.
2009-05-25 08:39:28 +02:00
Nico Schottelius
38ca0a1546 Revert "Keep a symlink to the current backup around. This allows us to"
This patch is broken for non local destinations.

This reverts commit 6de3c9877c.
2009-02-04 15:12:27 +01:00
Tonnerre Lombard
6de3c9877c Keep a symlink to the current backup around. This allows us to
establish automatic restore features (e.g. through Puppet file
facts).
2009-02-04 08:25:35 +01:00
Nico Schottelius
c133ba5df9 add release date
Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-02-02 12:10:48 +01:00
Nico Schottelius
b3ad86f270 [BUGFIX] allow "&" in sourcepath
Bug description:

From: Tiziano Müller <dev-zero@g.o>
Subject: Notiz: Probleme mit Ampersand in source
Date: Tue, 07 Oct 2008 16:11:10 +0000

Hoi Nico

Kleine Notiz wie im IRC besprochen:

- Hat es in /etc/ccollect/sources/$BACKUPNAME/source ein Ampersand muss
dieses Escaped werden, da sonst folgender Fehler auftritt:
*snip*
[volumes_ForschungEntwicklung] 2008-10-05-03:10:03: Beginning to backup,
this may take some time...
[volumes_ForschungEntwicklung] 2008-10-05-03:10:03:
Creating /home/backup/volumes/Forschung&Entwicklung/daily.20081005-0310.15173 ...
[volumes_ForschungEntwicklung] 2008-10-05-03:10:03: Transferring
files...
[volumes_ForschungEntwicklung] bash: Entwicklung/*: No such file or
directory
[volumes_ForschungEntwicklung] rsync: connection unexpectedly closed (4
bytes received so far) [receiver]
[volumes_ForschungEntwicklung] rsync error: error in rsync protocol data
stream (code 12) at io.c(635) [receiver=3.0.2]
[volumes_ForschungEntwicklung] 2008-10-05-03:10:03: Finished backup
(rsync return code: 12).
[volumes_ForschungEntwicklung] 2008-10-05-03:10:03: Warning: rsync
exited non-zero, the backup may be broken (see rsync errors).
*snip*

- ein "&" im Namen fürs Backup,
also: /etc/ccollect/sources/volumes_Forschung&Entwicklung/ wird beim
Anzeigen (wie oben zu sehen) verschluckt.

Cheers,
Tiziano

Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-02-02 11:47:45 +01:00
Nico Schottelius
b8b0ca107a [BUGFIX] cd must be in subshell, otherwise breaks later
Due to wrong WD

Signed-off-by: Nico Schottelius <nico@ikn.schottelius.org>
2009-01-16 14:44:59 +01:00
Jeroen Bruijning
5809571ca0 Fix bug in shell arithmetic
1. I got a syntax error in ccollect.sh once in a while, i think at
rollover. I changed the line

   j=$((j+1))

to

   j=$(($j+1))

(twice) and the error went away.
2008-06-13 11:34:43 +02:00
Nico Schottelius
fef686b449 remove basename - use standard vars defined in cconf
Signed-off-by: Nico Schottelius <nico@denkbrett.schottelius.org>
2008-06-13 11:28:43 +02:00
Nico Schottelius
d5c7b57b09 makefile / hinweise für tools/
Signed-off-by: Nico Schottelius <nico@denkbrett.schottelius.org>
2008-06-03 14:16:11 +02:00
Nico Schottelius
f096e412ea Add support for global delete_incomplete to ccollect
Signed-off-by: Nico Schottelius <nico@denkbrett.schottelius.org>
2008-04-26 12:59:24 +02:00
Nico Schottelius
0c2f5df283 update release date 2008-03-17 18:23:44 +01:00
Nico Schottelius
7b76e1ee65 bugfix: removing old backups 2008-03-17 12:21:35 +01:00
Nico Schottelius
842e668c4c fix another endless loop 2008-03-17 12:11:09 +01:00
Nico Schottelius
96171314d7 add error detection to backup listing 2008-03-17 12:09:30 +01:00
Nico Schottelius
4e47c881fb add more quotes 2008-03-17 12:08:40 +01:00
Nico Schottelius
8ace2a6520 fix endless loop 2008-03-17 12:06:17 +01:00
Nico Schottelius
0d5cc4ddbc add ddir to --link-dest 2008-03-17 11:44:20 +01:00
Nico Schottelius
49ee2a24a3 remote support seems to work, even tested it ;-) 2008-03-14 07:53:11 +01:00
Nico Schottelius
89b4c993a5 import new pcmd 2008-03-14 06:50:24 +01:00
Nico Schottelius
e87f4e9a92 add gpl3 header to ccollect 2008-03-07 23:03:55 +01:00
Nico Schottelius
eef2d55c61 begin to implement backup to a host 2008-03-05 00:39:17 +01:00
Nico Schottelius
42e2c7e95d Update version string 2007-10-02 13:28:14 +02:00
Nico Schottelius
be207d7520 update date and version 2007-08-27 19:15:35 +02:00
Nico Schottelius
f68a68a1e2 Finish problem with the marker 2007-08-27 19:10:02 +02:00
Nico Schottelius
c516175669 delete real path, print relative path 2007-08-27 19:00:43 +02:00
Nico Schottelius
5acca63d53 [BUGFIX]: Removing of backups was broken (forgot to prepend the path :/) 2007-08-27 18:57:14 +02:00
Nico Schottelius
feea0661a7 use . as a seperator for marking, not directly in the dir, which will be removed anyway. 2007-08-27 18:56:03 +02:00
Nico Schottelius
dcf92a1788 Update release date 2007-08-20 18:21:54 +02:00
Nico Schottelius
c0c19d2598 Minor cleanup 2007-08-20 17:33:09 +02:00
Nico Schottelius
2a45844e1c Also warn if there's a mostly impossible error 2007-08-20 17:29:24 +02:00
Nico Schottelius
69ae9076e3 Add comment about --link-dest 2007-08-20 12:45:23 +02:00
Nico Schottelius
95b4b66a99 Add version information 2007-08-18 00:14:10 +02:00
Nico Schottelius
fd6a50a36b with -tcp1 need to head -n1, not tail -n1 anymore 2007-08-17 23:50:40 +02:00
Nico Schottelius
2c80eab8e1 Include intelligent cloning 2007-08-17 23:31:45 +02:00
Nico Schottelius
d442277d68 Prepare release date 2007-08-17 00:19:28 +02:00
Nico Schottelius
88c295a5b7 typo 2007-08-16 21:46:02 +02:00
Nico Schottelius
aaf29c92b5 another case for _techo 2007-08-16 21:45:02 +02:00
Nico Schottelius
8f65880db5 Cleanups, cleanups, cleanups, ... 2007-08-16 21:37:39 +02:00
Nico Schottelius
dc28b25e3c use _techo, not echo 2007-08-16 21:26:56 +02:00
Nico Schottelius
e2a143e0b6 Some cleanups 2007-08-16 21:25:46 +02:00
Nico Schottelius
a35e31e86c In theory ccollect-0.6 is ready, now let's begin testing 2007-08-16 21:17:25 +02:00
Nico Schottelius
86960388df Added initial support for cloning from different intervals 2007-08-16 21:11:13 +02:00
Nico Schottelius
45c5c29f77 Cleaned up old remove code 2007-08-16 20:41:03 +02:00
Nico Schottelius
8f6a942ae9 Finish incomplete deletion in general 2007-08-16 20:31:30 +02:00
Nico Schottelius
799c24faf3 Many formal cleanups 2007-08-16 20:17:54 +02:00
Nico Schottelius
624e5b419b Many preparations for 0.6 2007-08-16 20:08:23 +02:00
Nico Schottelius
e92bd762d4 Begin reordering options 2007-08-14 15:40:23 +02:00
Nico Schottelius
ed91b488da Change rsync_options to one argument per line 2007-08-14 15:31:49 +02:00
Nico Schottelius
63a683941f Pass exclude in a safe way to rsync 2007-08-14 15:30:00 +02:00
Nico Schottelius
c042479097 Add much more timing information
And empty $@ and $#, which we'll reuse later.
2007-08-14 15:26:20 +02:00
Nico Schottelius
dc7d5d614b Reorder: first check if we should take all sources, then execute pre-exec 2007-08-14 14:41:36 +02:00
Nico Schottelius
30352b83a8 Always print return code 2007-08-14 14:36:33 +02:00
Nico Schottelius
d08ac69af6 Better parsable date output 2007-08-14 14:33:22 +02:00
Nico Schottelius
1d68408541 export number of sources 2007-05-31 11:55:37 +02:00
Nico Schottelius
13b47585a8 export interval 2007-05-31 11:52:33 +02:00
Nico Schottelius
c3312ea54d Cosmetic changes (III) 2007-03-13 08:47:04 +01:00
Nico Schottelius
c291d4a8f5 Cosmetic changes (II) plus integration of _exit_err 2007-03-13 08:31:50 +01:00
Nico Schottelius
17fde8943d Release today 2007-01-27 18:56:29 +01:00
Nico Schottelius
afbab56496 introduce _exit_err 2007-01-27 18:27:27 +01:00
Nico Schottelius
19d84f65da Add _exit_err 2007-01-20 16:41:38 +01:00
Nico Schottelius
775a7e89a9 Remove testing 2007-01-20 16:22:36 +01:00
Nico Schottelius
cd21d9c072 Merge todos
Cleanup 0.5.2 for release
2007-01-19 20:07:34 +01:00
Nico Schottelius
d95861c81c Add new todos 2007-01-19 20:01:53 +01:00
Nico Schottelius
7c53c28c5d Remove experiemental 2007-01-17 19:32:21 +01:00
Nico Schottelius
07cb3473db More comments 2007-01-17 19:18:42 +01:00
Nico Schottelius
44c80899fe Cosmetic changase 2007-01-17 18:45:13 +01:00