diff --git a/about.mdwn b/about.mdwn index a5e8761c..70e8eea5 100644 --- a/about.mdwn +++ b/about.mdwn @@ -11,8 +11,10 @@ I personally do not think it's wise to publish detailled personal information in the internet, because they are personal (versus public). To get an impression of what I do and who I am, you can have a look at -[[some press articles|press]], my -[[project list|projects]] or some of my [[websites|websites]]. +[[some press articles|press]], +my [[project list|projects]], +the [commit list](http://l.schottelius.org/pipermail/commits/) +or some of my other [[websites|websites]]. If you want to know more about me, there are [many](http://www.google.com/search?q=%22nico+schottelius%22) diff --git a/software/ccollect/ccollect-0.8.tar.bz2 b/software/ccollect/ccollect-0.8.tar.bz2 new file mode 100644 index 00000000..8dad0ba9 Binary files /dev/null and b/software/ccollect/ccollect-0.8.tar.bz2 differ diff --git a/software/ccollect/ccollect-0.8/.gitignore b/software/ccollect/ccollect-0.8/.gitignore new file mode 100644 index 00000000..4e9c74d5 --- /dev/null +++ b/software/ccollect/ccollect-0.8/.gitignore @@ -0,0 +1,17 @@ +conf/sources/*/destination/* +doc/old +doc/*.html +doc/*.htm +doc/*.docbook +doc/*.texi +doc/man/*.html +doc/man/*.htm +doc/man/*.texi +doc/man/*.man +test/* +.*.swp +doc/man/*.[0-9] +doc/*.xml +doc/*/*.xml +*.texi +*.fo diff --git a/software/ccollect/ccollect-0.8/COPYING b/software/ccollect/ccollect-0.8/COPYING new file mode 100644 index 00000000..94a9ed02 --- /dev/null +++ b/software/ccollect/ccollect-0.8/COPYING @@ -0,0 +1,674 @@ + GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + Copyright (C) + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +. diff --git a/software/ccollect/ccollect-0.8/CREDITS b/software/ccollect/ccollect-0.8/CREDITS new file mode 100644 index 00000000..aac73e27 --- /dev/null +++ b/software/ccollect/ccollect-0.8/CREDITS @@ -0,0 +1,16 @@ +Thanks go to the following people (sorted by alphabet): + +* Alexey Maximov + - for finding return-value and shell limitation bugs +* #cLinux IRC channel on irc.freenode.org + - for testing and debugging (those I mean should know ;-) +* Daniel Aubry + - for reporting many hints +* Jens-Christoph Brendel + - Added automatic backup manager (contrib/jbrendel-autobackup) +* John Lawless + - A lot of patches and some very interesting discussions. +* Markus Meier + - for finding a really simple solution for choosing the right backup to + clone from: Make it independent of the interval, simply choose the last + one created. diff --git a/software/ccollect/ccollect-0.8/Makefile b/software/ccollect/ccollect-0.8/Makefile new file mode 100644 index 00000000..82bab0e6 --- /dev/null +++ b/software/ccollect/ccollect-0.8/Makefile @@ -0,0 +1,207 @@ +# +# 2006-2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Initially written on Fri Jan 13 12:13:08 CET 2006 +# +# FIXME: add prefix-support? +# + +INSTALL=install +CCOLLECT_SOURCE=ccollect.sh +CCOLLECT_DEST=ccollect.sh +LN=ln -sf +ASCIIDOC=asciidoc +DOCBOOKTOTEXI=docbook2x-texi +DOCBOOKTOMAN=docbook2x-man +XSLTPROC=xsltproc +XSL=/usr/share/xml/docbook/stylesheet/nwalsh/html/docbook.xsl +A2X=a2x + +prefix=/usr/packages/ccollect-git +bindir=${prefix}/bin +destination=${bindir}/${CCOLLECT_DEST} + +mandest=${prefix}/man/man1 +manlink=/usr/local/man/man1 + +path_dir=/usr/local/bin +path_destination=${path_dir}/${CCOLLECT_DEST} + +# where to publish +host=localhost +dir=/home/users/nico/privat/computer/net/netzseiten/www.nico.schottelius.org/src/software/ccollect +docdir=${dir}/doc + +# +# Asciidoc will be used to generate other formats later +# +MANDOCS = doc/man/ccollect.text \ + doc/man/ccollect_add_source.text \ + doc/man/ccollect_analyse_logs.text \ + doc/man/ccollect_delete_source.text \ + doc/man/ccollect_logwrapper.text \ + doc/man/ccollect_list_intervals.text + +DOCS = ${MANDOCS} doc/ccollect.text + +# +# Doku +# +HTMLDOCS = ${DOCS:.text=.html} +DBHTMLDOCS = ${DOCS:.text=.htm} + +# texi is broken currently, don't know why xslt things complain yet +TEXIDOCS = ${DOCS:.text=.texi} +TEXIDOCS = + +# fop fails here, so disable it for now +PDFDOCS = ${DOCS:.text=.pdf} +PDFDOCS = + +MANPDOCS = ${MANDOCS:.text=.1} + +DOCBDOCS = ${DOCS:.text=.docbook} + +DOC_ALL = ${HTMLDOCS} ${DBHTMLDOCS} ${TEXIDOCS} ${MANPDOCS} ${PDFDOCS} + +html: ${HTMLDOCS} +htm: ${DBHTMLDOCS} +info: ${TEXIDOCS} +man: ${MANPDOCS} +pdf: ${PDFDOCS} +documentation: ${DOC_ALL} + +# +# End user targets +# +all: + @echo "----------- ccollect make targets --------------" + @echo "documentation: generate HTMl, Texinfo and manpage" + @echo "html: only generate HTML" + @echo "info: only generate Texinfo" + @echo "man: only generate manpage{s}" + @echo "install: install ccollect to ${prefix}" + +install: install-link install-manlink + +install-link: install-script + ${LN} ${destination} ${path_destination} + +install-script: + ${INSTALL} -D -m 0755 ${CCOLLECT_SOURCE} ${destination} + +install-man: man + ${INSTALL} -d -m 0755 ${mandest} + ${INSTALL} -D -m 0644 doc/man/*.1 ${mandest} + +install-manlink: install-man + ${INSTALL} -d -m 0755 ${manlink} + for man in ${mandest}/*; do ${LN} $$man ${manlink}; done + +# +# Tools +# +TOOLS=ccollect_add_source.sh \ + ccollect_analyse_logs.sh \ + ccollect_delete_source.sh \ + ccollect_list_intervals.sh \ + ccollect_logwrapper.sh \ + ccollect_list_intervals.sh + +TOOLSMAN1 = $(subst ccollect,doc/man/ccollect,$(TOOLS)) +TOOLSMAN = $(subst .sh,.text,$(TOOLSMAN1)) + +TOOLSFP = $(subst ccollect,tools/ccollect,$(TOOLS)) + +#t2: $(TOOLSMAN) +t2: + echo $(TOOLS) - $(TOOLSMAN) - $(TOOLSFP) + + +# docbook gets .htm, asciidoc directly .html +%.htm: %.docbook + ${XSLTPROC} -o $@ ${XSL} $< + +%.html: %.text %.docbook + ${ASCIIDOC} -n -o $@ $< + +%.html: %.text + ${ASCIIDOC} -n -o $@ $< + +%.docbook: %.text + ${ASCIIDOC} -n -b docbook -o $@ $< + +%.texi: %.docbook + ${DOCBOOKTOTEXI} --to-stdout $< > $@ + +#%.mandocbook: %.text +# ${ASCIIDOC} -b docbook -d manpage -o $@ $< + +#%.man: %.mandocbook +# ${DOCBOOKTOMAN} --to-stdout $< > $@ + +#%.man: %.text +%.1: %.text + ${A2X} -f manpage $< + +%.pdf: %.text + ${A2X} -f pdf $< + + +# +# Developer targets +# +update: + @git push + +publish-doc: documentation + @echo "Transferring files to ${host}" + @chmod a+r ${DOCS} ${DOC_ALL} + @tar c ${DOCS} ${DOC_ALL} | ssh ${host} "cd ${dir}; tar xv" + +# +# Distribution +# +clean: + rm -f ${DOC_ALL} + rm -f doc/man/*.[0-9] doc/man/*.xml doc/*.fo doc/man/*.fo + +distclean: clean + rm -f ${DOCBDOCS} + +# +# Be nice with the users and generate documentation for them +# +dist: distclean documentation + +#test: ccollect.sh documentation +test: ccollect.sh + mkdir -p /tmp/ccollect + CCOLLECT_CONF=./conf ./ccollect.sh daily from-remote + CCOLLECT_CONF=./conf ./ccollect.sh daily local + CCOLLECT_CONF=./conf ./ccollect.sh daily "local-with&ersand" + CCOLLECT_CONF=./conf ./ccollect.sh daily source-without-destination + CCOLLECT_CONF=./conf ./ccollect.sh daily "source with spaces and interval" + CCOLLECT_CONF=./conf ./ccollect.sh daily to-remote + CCOLLECT_CONF=./conf ./ccollect.sh daily with_exec + CCOLLECT_CONF=./conf ./ccollect.sh daily very_verbose + touch /tmp/ccollect/$$(ls /tmp/ccollect | head -n1).ccollect-marker + CCOLLECT_CONF=./conf ./ccollect.sh daily delete_incomplete + CCOLLECT_CONF=./conf ./ccollect.sh daily no-source-must-fail +# for s in $$(ls ./conf/sources); do CCOLLECT_CONF=./conf echo ./ccollect.sh daily $$s; done +# CCOLLECT_CONF=./conf ./ccollect.sh -a daily diff --git a/software/ccollect/ccollect-0.8/README b/software/ccollect/ccollect-0.8/README new file mode 100644 index 00000000..558594d8 --- /dev/null +++ b/software/ccollect/ccollect-0.8/README @@ -0,0 +1,65 @@ +-------------------------------------------------------------------------------- +ccollect.sh, Nico Schottelius, 2005-12-06 +-------------------------------------------------------------------------------- + +ccollect backups (local or remote) data to local or remote destinations. + +You can retrieve the latest version of ccollect at [0]. + +ccollect was inspired by rsnapshot [1], which has some problems: + - configuration parameters has to be TAB seperated + - you can not specify per source exclude lists + - no per source pre/post execution support + - no parallel execution + - does unecessary moving of backup directories + - I didn't like the configuration at all, so I used the cconfig style [2]. + +Please use tools/report_success.sh to report success, if you are successfully +using ccollect. + +Have a look at doc/HACKING, if you plan to change ccollect. + +A small try to visualize the differences in a table: + ++---------------+-------------------------------------------------------------+ +| What? | rsnapshot | ccollect | ++---------------+-------------------------------------------------------------+ +| Configuration | tab separated, needs | plain cconfig-style | +| | parsing | | ++---------------+-------------------------------------------------------------+ +| Per source | | | +| post-/pre- | no | yes | +| execution | | | ++---------------+-------------------------------------------------------------+ +| Per source | | | +| exclude lists | no | yes | ++---------------+-------------------------------------------------------------+ +| Parallel | | | +| execution | | | +| of multiple | no | yes | +| backups | | | ++---------------+-------------------------------------------------------------+ +| Programming | perl | sh | +| language | | (posix compatible) | ++---------------+-------------------------------------------------------------+ +| Lines of code | 6772 (5353 w/o comments, | 546 (375 w/o comments, | +| (2006-10-25) | 4794 w/o empty lines) | 288 w/o empty lines) | ++---------------+-------------------------------------------------------------+ +| Lines of code | 7269 (6778 w/o comments, | 587 (397 w/o comments, | +| (2009-07-23) | 6139 w/o empty lines) | 315 w/o empty lines) | ++---------------+-------------------------------------------------------------+ +| Age | Available since 2002/2003 | Written at 2005-11-14 | ++---------------+-------------------------------------------------------------+ + +Included documentation: + +doc/ccollect.text Manual in text format +doc/ccollect.html Manual in xhtml (generated) + +doc/man/ccollect.text Manpage in text format +doc/man/ccollect.man Manpage in manpage format (generated) + +-------------------------------------------------------------------------------- +[0]: ccollect: http://www.nico.schottelius.org/software/ccollect/ +[1]: rsnapshot: http://www.rsnapshot.org/ +[2]: cconfig: http://nico.schotteli.us/papers/linux/cconfig/ diff --git a/software/ccollect/ccollect-0.8/ccollect.sh b/software/ccollect/ccollect-0.8/ccollect.sh new file mode 100644 index 00000000..13ceb339 --- /dev/null +++ b/software/ccollect/ccollect-0.8/ccollect.sh @@ -0,0 +1,596 @@ +#!/bin/sh +# +# 2005-2009 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Initially written for SyGroup (www.sygroup.ch) +# Date: Mon Nov 14 11:45:11 CET 2005 + +# Error upon expanding unset variables: +set -u + +# +# Standard variables (stolen from cconf) +# +__pwd="$(pwd -P)" +__mydir="${0%/*}"; __abs_mydir="$(cd "$__mydir" && pwd -P)" +__myname=${0##*/}; __abs_myname="$__abs_mydir/$__myname" + +# +# where to find our configuration and temporary file +# +CCOLLECT_CONF="${CCOLLECT_CONF:-/etc/ccollect}" +CSOURCES="${CCOLLECT_CONF}/sources" +CDEFAULTS="${CCOLLECT_CONF}/defaults" +CPREEXEC="${CDEFAULTS}/pre_exec" +CPOSTEXEC="${CDEFAULTS}/post_exec" + +export TMP=$(mktemp "/tmp/${__myname}.XXXXXX") +VERSION="0.8" +RELEASE="2009-08-20" +HALF_VERSION="ccollect ${VERSION}" +FULL_VERSION="ccollect ${VERSION} (${RELEASE})" + +# +# CDATE: how we use it for naming of the archives +# DDATE: how the user should see it in our output (DISPLAY) +# +CDATE="date +%Y%m%d-%H%M" +DDATE="date +%Y-%m-%d-%H:%M:%S" +SDATE="date +%s" + +# +# unset values +# +PARALLEL="" +USE_ALL="" + +# +# catch signals +# +trap "rm -f \"${TMP}\"" 1 2 15 + +# +# Functions +# + +# time displaying echo +_techo() +{ + echo "$(${DDATE}): $@" +} + +# exit on error +_exit_err() +{ + _techo "$@" + rm -f "${TMP}" + exit 1 +} + +add_name() +{ + awk "{ print \"[${name}] \" \$0 }" +} + +# +# Execute on remote host, if backing up to a remote host +# +pcmd() +{ + if [ "${remote_host}" ]; then + ssh "${remote_host}" "$@" + else + "$@" + fi +} + +delete_from_file() +{ + # + # ssh-"feature": we cannot do '... read ...; ssh ...; < file', + # because ssh reads stdin! -n does not work -> does not ask for password + # + file="$1"; shift + while read to_remove; do set -- "$@" "${ddir}/${to_remove}"; done < "${file}" + _techo "Removing $@ ..." + pcmd rm ${VVERBOSE} -rf "$@" || _exit_err "Removing $@ failed." +} + +display_version() +{ + echo "${FULL_VERSION}" + exit 0 +} + +usage() +{ + cat << eof +${__myname}: [args] + + ccollect creates (pseudo) incremental backups + + -h, --help: Show this help screen + -p, --parallel: Parallelise backup processes + -a, --all: Backup all sources specified in ${CSOURCES} + -v, --verbose: Be very verbose (uses set -x) + -V, --version: Print version information + + This is version ${VERSION}, released on ${RELEASE} + (the first version was written on 2005-12-05 by Nico Schottelius). + + Retrieve latest ccollect at http://www.nico.schottelius.org/software/ccollect/ +eof + exit 0 +} + +# +# Parse options +# +while [ "$#" -ge 1 ]; do + case "$1" in + -a|--all) + USE_ALL=1 + ;; + -v|--verbose) + set -x + ;; + -p|--parallel) + PARALLEL=1 + ;; + -h|--help) + usage + ;; + -V|--version) + display_version + ;; + -h|--help|-*) + usage + ;; + --) + # ignore the -- itself + shift + break + ;; + *) + break + ;; + esac + shift +done + +# +# Setup interval +# +if [ $# -ge 1 ]; then + export INTERVAL="$1" + shift +else + usage +fi + +# +# Check for configuraton directory +# +[ -d "${CCOLLECT_CONF}" ] || _exit_err "No configuration found in " \ + "\"${CCOLLECT_CONF}\" (is \$CCOLLECT_CONF properly set?)" + +# +# Create (portable!) source "array" +# +export no_sources=0 + +if [ "${USE_ALL}" = 1 ]; then + # + # Get sources from source configuration + # + ( cd "${CSOURCES}" && ls -1 > "${TMP}" ); ret=$? + + [ "${ret}" -eq 0 ] || _exit_err "Listing of sources failed. Aborting." + + while read tmp; do + eval export source_${no_sources}=\"${tmp}\" + no_sources=$((${no_sources}+1)) + done < "${TMP}" +else + # + # Get sources from command line + # + while [ "$#" -ge 1 ]; do + eval arg=\"\$1\"; shift + + eval export source_${no_sources}=\"${arg}\" + no_sources="$((${no_sources}+1))" + done +fi + +# +# Need at least ONE source to backup +# +if [ "${no_sources}" -lt 1 ]; then + usage +else + _techo "${HALF_VERSION}: Beginning backup using interval ${INTERVAL}" +fi + +# +# Look for pre-exec command (general) +# +if [ -x "${CPREEXEC}" ]; then + _techo "Executing ${CPREEXEC} ..." + "${CPREEXEC}"; ret=$? + _techo "Finished ${CPREEXEC} (return code: ${ret})." + + [ "${ret}" -eq 0 ] || _exit_err "${CPREEXEC} failed. Aborting" +fi + +# +# Let's do the backup +# +i=0 +while [ "${i}" -lt "${no_sources}" ]; do + # + # Get current source + # + eval name=\"\$source_${i}\" + i=$((${i}+1)) + + export name + + # + # start ourself, if we want parallel execution + # + if [ "${PARALLEL}" ]; then + "$0" "${INTERVAL}" "${name}" & + continue + fi + +# +# Start subshell for easy log editing +# +( + backup="${CSOURCES}/${name}" + # + # Stderr to stdout, so we can produce nice logs + # + exec 2>&1 + + # + # Record start of backup: internal and for the user + # + begin_s="$(${SDATE})" + _techo "Beginning to backup" + + # + # Standard configuration checks + # + if [ ! -e "${backup}" ]; then + _exit_err "Source does not exist." + fi + + # + # Configuration _must_ be a directory (cconfig style) + # + if [ ! -d "${backup}" ]; then + _exit_err "\"${name}\" is not a cconfig-directory. Skipping." + fi + + # + # Read / create configuration + # + c_source="${backup}/source" + c_dest="${backup}/destination" + c_pre_exec="${backup}/pre_exec" + c_post_exec="${backup}/post_exec" + c_marker="ccollect-marker" + for opt in verbose very_verbose summary exclude rsync_options \ + delete_incomplete remote_host rsync_failure_codes \ + mtime quiet_if_down ; do + if [ -f "${backup}/${opt}" -o -f "${backup}/no_${opt}" ]; then + eval c_$opt=\"${backup}/$opt\" + else + eval c_$opt=\"${CDEFAULTS}/$opt\" + fi + done + + # + # Sort by ctime (default) or mtime (configuration option) + # + if [ -f "$c_mtime" ] ; then + TSORT="t" + else + TSORT="tc" + fi + + # + # First execute pre_exec, which may generate destination or other parameters + # + if [ -x "${c_pre_exec}" ]; then + _techo "Executing ${c_pre_exec} ..." + "${c_pre_exec}"; ret="$?" + _techo "Finished ${c_pre_exec} (return code ${ret})." + + if [ "${ret}" -ne 0 ]; then + _exit_err "${c_pre_exec} failed. Skipping." + fi + fi + + # + # Source configuration checks + # + if [ ! -f "${c_source}" ]; then + _exit_err "Source description \"${c_source}\" is not a file. Skipping." + else + source=$(cat "${c_source}"); ret="$?" + if [ "${ret}" -ne 0 ]; then + _exit_err "Source ${c_source} is not readable. Skipping." + fi + fi + + # + # Destination is a path + # + if [ ! -f "${c_dest}" ]; then + _exit_err "Destination ${c_dest} is not a file. Skipping." + else + ddir="$(cat "${c_dest}")"; ret="$?" + if [ "${ret}" -ne 0 ]; then + _exit_err "Destination ${c_dest} is not readable. Skipping." + fi + fi + + # + # Set pre-cmd, if we backup to a remote host. + # + if [ -f "${c_remote_host}" ]; then + remote_host="$(cat "${c_remote_host}")"; ret="$?" + if [ "${ret}" -ne 0 ]; then + _exit_err "Remote host file ${c_remote_host} exists, but is not readable. Skipping." + fi + destination="${remote_host}:${ddir}" + else + remote_host="" + destination="${ddir}" + fi + export remote_host + + # + # Parameters: ccollect defaults, configuration options, user options + # + + # + # Rsync standard options + # + set -- "$@" "--archive" "--delete" "--numeric-ids" "--relative" \ + "--delete-excluded" "--sparse" + + # + # Exclude list + # + if [ -f "${c_exclude}" ]; then + set -- "$@" "--exclude-from=${c_exclude}" + fi + + # + # Output a summary + # + if [ -f "${c_summary}" ]; then + set -- "$@" "--stats" + fi + + # + # Verbosity for rsync, rm, and mkdir + # + VVERBOSE="" + if [ -f "${c_very_verbose}" ]; then + set -- "$@" "-vv" + VVERBOSE="-v" + elif [ -f "${c_verbose}" ]; then + set -- "$@" "-v" + fi + + # + # Extra options for rsync provided by the user + # + if [ -f "${c_rsync_options}" ]; then + while read line; do + set -- "$@" "$line" + done < "${c_rsync_options}" + fi + + # + # Check: source is up and accepting connections (before deleting old backups!) + # + if ! rsync "${source}" >/dev/null 2>"${TMP}" ; then + if [ ! -f "${c_quiet_if_down}" ]; then + cat "${TMP}" + fi + _exit_err "Source ${source} is not readable. Skipping." + fi + + # + # Check: destination exists? + # + ( pcmd cd "${ddir}" ) || _exit_err "Cannot change to ${ddir}. Skipping." + + # + # Check: incomplete backups? (needs echo to remove newlines) + # + incomplete="$(echo \ + $(pcmd ls -1 "${ddir}/" | \ + awk "/\.${c_marker}\$/ { print \$0; gsub(\"\.${c_marker}\$\",\"\",\$0); print \$0 }" | \ + tee "${TMP}"))" + + if [ "${incomplete}" ]; then + _techo "Incomplete backups: ${incomplete}" + if [ -f "${c_delete_incomplete}" ]; then + delete_from_file "${TMP}" + fi + fi + + # + # Interval definition: First try source specific, fallback to default + # + c_interval="$(cat "${backup}/intervals/${INTERVAL}" 2>/dev/null)" + + if [ -z "${c_interval}" ]; then + c_interval="$(cat "${CDEFAULTS}/intervals/${INTERVAL}" 2>/dev/null)" + + if [ -z "${c_interval}" ]; then + _exit_err "No definition for interval \"${INTERVAL}\" found. Skipping." + fi + fi + + # + # Check: maximum number of backups is reached? + # If so remove. Use grep and ls -p so we only look at directories + # + count="$(pcmd ls -p1 "${ddir}" | grep "^${INTERVAL}\..*/\$" | wc -l \ + | sed 's/^ *//g')" || _exit_err "Counting backups failed" + + _techo "Existing backups: ${count} Total keeping backups: ${c_interval}" + + if [ "${count}" -ge "${c_interval}" ]; then + substract="$((${c_interval} - 1))" + remove="$((${count} - ${substract}))" + _techo "Removing ${remove} backup(s)..." + + pcmd ls -${TSORT}p1r "${ddir}" | grep "^${INTERVAL}\..*/\$" | \ + head -n "${remove}" > "${TMP}" || \ + _exit_err "Listing old backups failed" + + delete_from_file "${TMP}" + fi + + # + # Check for backup directory to clone from: Always clone from the latest one! + # + last_dir="$(pcmd ls -${TSORT}p1 "${ddir}" | grep '/$' | head -n 1)" || \ + _exit_err "Failed to list contents of ${ddir}." + + # + # clone from old backup, if existing + # + if [ "${last_dir}" ]; then + set -- "$@" "--link-dest=${ddir}/${last_dir}" + _techo "Hard linking from ${last_dir}" + fi + + # set time when we really begin to backup, not when we began to remove above + destination_date="$(${CDATE})" + destination_dir="${ddir}/${INTERVAL}.${destination_date}.$$" + destination_full="${destination}/${INTERVAL}.${destination_date}.$$" + + # give some info + _techo "Beginning to backup, this may take some time..." + + _techo "Creating ${destination_dir} ..." + pcmd mkdir ${VVERBOSE} "${destination_dir}" || \ + _exit_err "Creating ${destination_dir} failed. Skipping." + + # + # added marking in 0.6 (and remove it, if successful later) + # + pcmd touch "${destination_dir}.${c_marker}" + + # + # the rsync part + # + _techo "Transferring files..." + rsync "$@" "${source}" "${destination_full}"; ret=$? + _techo "Finished backup (rsync return code: $ret)." + + # + # Set modification time (mtime) to current time, if sorting by mtime is enabled + # + [ -f "$c_mtime" ] && pcmd touch "${destination_dir}" + + # + # Check if rsync exit code indicates failure. + # + fail="" + if [ -f "$c_rsync_failure_codes" ]; then + while read code ; do + if [ "$ret" = "$code" ]; then + fail=1 + fi + done <"$c_rsync_failure_codes" + fi + + # + # Remove marking here unless rsync failed. + # + if [ -z "$fail" ]; then + pcmd rm "${destination_dir}.${c_marker}" || \ + _exit_err "Removing ${destination_dir}.${c_marker} failed." + if [ "${ret}" -ne 0 ]; then + _techo "Warning: rsync exited non-zero, the backup may be broken (see rsync errors)." + fi + else + _techo "Warning: rsync failed with return code $ret." + fi + + # + # post_exec + # + if [ -x "${c_post_exec}" ]; then + _techo "Executing ${c_post_exec} ..." + "${c_post_exec}"; ret=$? + _techo "Finished ${c_post_exec}." + + if [ "${ret}" -ne 0 ]; then + _exit_err "${c_post_exec} failed." + fi + fi + + # Calculation + end_s="$(${SDATE})" + + full_seconds="$((${end_s} - ${begin_s}))" + hours="$((${full_seconds} / 3600))" + seconds="$((${full_seconds} - (${hours} * 3600)))" + minutes="$((${seconds} / 60))" + seconds="$((${seconds} - (${minutes} * 60)))" + + _techo "Backup lasted: ${hours}:${minutes}:${seconds} (h:m:s)" + +) | add_name +done + +# +# Be a good parent and wait for our children, if they are running wild parallel +# +if [ "${PARALLEL}" ]; then + _techo "Waiting for children to complete..." + wait +fi + +# +# Look for post-exec command (general) +# +if [ -x "${CPOSTEXEC}" ]; then + _techo "Executing ${CPOSTEXEC} ..." + "${CPOSTEXEC}"; ret=$? + _techo "Finished ${CPOSTEXEC} (return code: ${ret})." + + if [ "${ret}" -ne 0 ]; then + _techo "${CPOSTEXEC} failed." + fi +fi + +rm -f "${TMP}" +_techo "Finished" diff --git a/software/ccollect/ccollect-0.8/conf/README b/software/ccollect/ccollect-0.8/conf/README new file mode 100644 index 00000000..8402c76f --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/README @@ -0,0 +1,3 @@ +This is my personal test configuration. +It can perhaps be some example for you, although it may be pretty +unsorted and highly chaotic. diff --git a/software/ccollect/ccollect-0.8/conf/defaults/intervals/daily b/software/ccollect/ccollect-0.8/conf/defaults/intervals/daily new file mode 100644 index 00000000..9902f178 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/defaults/intervals/daily @@ -0,0 +1 @@ +28 diff --git a/software/ccollect/ccollect-0.8/conf/defaults/intervals/monthly b/software/ccollect/ccollect-0.8/conf/defaults/intervals/monthly new file mode 100644 index 00000000..48082f72 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/defaults/intervals/monthly @@ -0,0 +1 @@ +12 diff --git a/software/ccollect/ccollect-0.8/conf/defaults/intervals/normal b/software/ccollect/ccollect-0.8/conf/defaults/intervals/normal new file mode 100644 index 00000000..7273c0fa --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/defaults/intervals/normal @@ -0,0 +1 @@ +25 diff --git a/software/ccollect/ccollect-0.8/conf/defaults/intervals/weekly b/software/ccollect/ccollect-0.8/conf/defaults/intervals/weekly new file mode 100644 index 00000000..b8626c4c --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/defaults/intervals/weekly @@ -0,0 +1 @@ +4 diff --git a/software/ccollect/ccollect-0.8/conf/defaults/post_exec b/software/ccollect/ccollect-0.8/conf/defaults/post_exec new file mode 100644 index 00000000..29655746 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/defaults/post_exec @@ -0,0 +1,5 @@ +#!/bin/cat + +###################################################################### +If you see this content, post_exec was executed. (general post_exec) +###################################################################### diff --git a/software/ccollect/ccollect-0.8/conf/defaults/pre_exec b/software/ccollect/ccollect-0.8/conf/defaults/pre_exec new file mode 100644 index 00000000..09084a62 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/defaults/pre_exec @@ -0,0 +1,4 @@ +#!/bin/cat + +If you see this content, pre_exec was executed. +(general pre_exec, not source dependent) diff --git a/software/ccollect/ccollect-0.8/conf/defaults/sources/exclude b/software/ccollect/ccollect-0.8/conf/defaults/sources/exclude new file mode 100644 index 00000000..ee4c9268 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/defaults/sources/exclude @@ -0,0 +1 @@ +/test diff --git a/software/ccollect/ccollect-0.8/conf/defaults/sources/rsync_options b/software/ccollect/ccollect-0.8/conf/defaults/sources/rsync_options new file mode 100644 index 00000000..e69de29b diff --git a/software/ccollect/ccollect-0.8/conf/defaults/sources/source_postfix b/software/ccollect/ccollect-0.8/conf/defaults/sources/source_postfix new file mode 100644 index 00000000..0b5cde79 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/defaults/sources/source_postfix @@ -0,0 +1 @@ +:/ diff --git a/software/ccollect/ccollect-0.8/conf/defaults/sources/source_prefix b/software/ccollect/ccollect-0.8/conf/defaults/sources/source_prefix new file mode 100644 index 00000000..fc37f9d6 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/defaults/sources/source_prefix @@ -0,0 +1 @@ +root@ diff --git a/software/ccollect/ccollect-0.8/conf/defaults/sources/verbose b/software/ccollect/ccollect-0.8/conf/defaults/sources/verbose new file mode 100644 index 00000000..e69de29b diff --git a/software/ccollect/ccollect-0.8/conf/defaults/verbose b/software/ccollect/ccollect-0.8/conf/defaults/verbose new file mode 100644 index 00000000..e69de29b diff --git a/software/ccollect/ccollect-0.8/conf/sources/delete_incomplete/delete_incomplete b/software/ccollect/ccollect-0.8/conf/sources/delete_incomplete/delete_incomplete new file mode 100644 index 00000000..e69de29b diff --git a/software/ccollect/ccollect-0.8/conf/sources/delete_incomplete/destination b/software/ccollect/ccollect-0.8/conf/sources/delete_incomplete/destination new file mode 100644 index 00000000..8cac69d3 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/delete_incomplete/destination @@ -0,0 +1 @@ +/tmp/ccollect diff --git a/software/ccollect/ccollect-0.8/conf/sources/delete_incomplete/exclude b/software/ccollect/ccollect-0.8/conf/sources/delete_incomplete/exclude new file mode 100644 index 00000000..6b8710a7 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/delete_incomplete/exclude @@ -0,0 +1 @@ +.git diff --git a/software/ccollect/ccollect-0.8/conf/sources/delete_incomplete/source b/software/ccollect/ccollect-0.8/conf/sources/delete_incomplete/source new file mode 100644 index 00000000..e64611b7 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/delete_incomplete/source @@ -0,0 +1 @@ +/home/users/nico/bin diff --git a/software/ccollect/ccollect-0.8/conf/sources/from-remote/README b/software/ccollect/ccollect-0.8/conf/sources/from-remote/README new file mode 100644 index 00000000..c778cfe8 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/from-remote/README @@ -0,0 +1 @@ +This is based on a production example I use for my notebook. diff --git a/software/ccollect/ccollect-0.8/conf/sources/from-remote/destination b/software/ccollect/ccollect-0.8/conf/sources/from-remote/destination new file mode 100644 index 00000000..8cac69d3 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/from-remote/destination @@ -0,0 +1 @@ +/tmp/ccollect diff --git a/software/ccollect/ccollect-0.8/conf/sources/from-remote/exclude b/software/ccollect/ccollect-0.8/conf/sources/from-remote/exclude new file mode 100644 index 00000000..f5b5b7c8 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/from-remote/exclude @@ -0,0 +1 @@ +/home/server/raid diff --git a/software/ccollect/ccollect-0.8/conf/sources/from-remote/source b/software/ccollect/ccollect-0.8/conf/sources/from-remote/source new file mode 100644 index 00000000..540dd91c --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/from-remote/source @@ -0,0 +1 @@ +localhost:/home/users/nico/bin diff --git a/software/ccollect/ccollect-0.8/conf/sources/from-remote/summary b/software/ccollect/ccollect-0.8/conf/sources/from-remote/summary new file mode 100644 index 00000000..e69de29b diff --git a/software/ccollect/ccollect-0.8/conf/sources/from-remote/verbose b/software/ccollect/ccollect-0.8/conf/sources/from-remote/verbose new file mode 100644 index 00000000..e69de29b diff --git a/software/ccollect/ccollect-0.8/conf/sources/local-with&ersand/destination b/software/ccollect/ccollect-0.8/conf/sources/local-with&ersand/destination new file mode 100644 index 00000000..8cac69d3 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/local-with&ersand/destination @@ -0,0 +1 @@ +/tmp/ccollect diff --git a/software/ccollect/ccollect-0.8/conf/sources/local-with&ersand/exclude b/software/ccollect/ccollect-0.8/conf/sources/local-with&ersand/exclude new file mode 100644 index 00000000..6b8710a7 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/local-with&ersand/exclude @@ -0,0 +1 @@ +.git diff --git a/software/ccollect/ccollect-0.8/conf/sources/local-with&ersand/source b/software/ccollect/ccollect-0.8/conf/sources/local-with&ersand/source new file mode 100644 index 00000000..e64611b7 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/local-with&ersand/source @@ -0,0 +1 @@ +/home/users/nico/bin diff --git a/software/ccollect/ccollect-0.8/conf/sources/local/destination b/software/ccollect/ccollect-0.8/conf/sources/local/destination new file mode 100644 index 00000000..8cac69d3 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/local/destination @@ -0,0 +1 @@ +/tmp/ccollect diff --git a/software/ccollect/ccollect-0.8/conf/sources/local/exclude b/software/ccollect/ccollect-0.8/conf/sources/local/exclude new file mode 100644 index 00000000..6b8710a7 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/local/exclude @@ -0,0 +1 @@ +.git diff --git a/software/ccollect/ccollect-0.8/conf/sources/local/no_verbose b/software/ccollect/ccollect-0.8/conf/sources/local/no_verbose new file mode 100644 index 00000000..e69de29b diff --git a/software/ccollect/ccollect-0.8/conf/sources/local/source b/software/ccollect/ccollect-0.8/conf/sources/local/source new file mode 100644 index 00000000..e64611b7 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/local/source @@ -0,0 +1 @@ +/home/users/nico/bin diff --git a/software/ccollect/ccollect-0.8/conf/sources/source with spaces and interval/delete_incomplete b/software/ccollect/ccollect-0.8/conf/sources/source with spaces and interval/delete_incomplete new file mode 100644 index 00000000..e69de29b diff --git a/software/ccollect/ccollect-0.8/conf/sources/source with spaces and interval/destination b/software/ccollect/ccollect-0.8/conf/sources/source with spaces and interval/destination new file mode 100644 index 00000000..8cac69d3 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/source with spaces and interval/destination @@ -0,0 +1 @@ +/tmp/ccollect diff --git a/software/ccollect/ccollect-0.8/conf/sources/source with spaces and interval/exclude b/software/ccollect/ccollect-0.8/conf/sources/source with spaces and interval/exclude new file mode 100644 index 00000000..bbdbdf1a --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/source with spaces and interval/exclude @@ -0,0 +1,3 @@ +openvpn-2.0.1.tar.gz +nicht_reinnehmen +etwas mit leerzeichenli diff --git a/software/ccollect/ccollect-0.8/conf/sources/source with spaces and interval/intervals/daily b/software/ccollect/ccollect-0.8/conf/sources/source with spaces and interval/intervals/daily new file mode 100644 index 00000000..64bb6b74 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/source with spaces and interval/intervals/daily @@ -0,0 +1 @@ +30 diff --git a/software/ccollect/ccollect-0.8/conf/sources/source with spaces and interval/source b/software/ccollect/ccollect-0.8/conf/sources/source with spaces and interval/source new file mode 100644 index 00000000..e64611b7 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/source with spaces and interval/source @@ -0,0 +1 @@ +/home/users/nico/bin diff --git a/software/ccollect/ccollect-0.8/conf/sources/source with spaces and interval/verbose b/software/ccollect/ccollect-0.8/conf/sources/source with spaces and interval/verbose new file mode 100644 index 00000000..e69de29b diff --git a/software/ccollect/ccollect-0.8/conf/sources/source-without-destination/exclude b/software/ccollect/ccollect-0.8/conf/sources/source-without-destination/exclude new file mode 100644 index 00000000..6b8710a7 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/source-without-destination/exclude @@ -0,0 +1 @@ +.git diff --git a/software/ccollect/ccollect-0.8/conf/sources/source-without-destination/source b/software/ccollect/ccollect-0.8/conf/sources/source-without-destination/source new file mode 100644 index 00000000..e64611b7 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/source-without-destination/source @@ -0,0 +1 @@ +/home/users/nico/bin diff --git a/software/ccollect/ccollect-0.8/conf/sources/this_is_not_a_source b/software/ccollect/ccollect-0.8/conf/sources/this_is_not_a_source new file mode 100644 index 00000000..e69de29b diff --git a/software/ccollect/ccollect-0.8/conf/sources/to-remote/destination b/software/ccollect/ccollect-0.8/conf/sources/to-remote/destination new file mode 100644 index 00000000..8cac69d3 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/to-remote/destination @@ -0,0 +1 @@ +/tmp/ccollect diff --git a/software/ccollect/ccollect-0.8/conf/sources/to-remote/exclude b/software/ccollect/ccollect-0.8/conf/sources/to-remote/exclude new file mode 100644 index 00000000..6b8710a7 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/to-remote/exclude @@ -0,0 +1 @@ +.git diff --git a/software/ccollect/ccollect-0.8/conf/sources/to-remote/remote_host b/software/ccollect/ccollect-0.8/conf/sources/to-remote/remote_host new file mode 100644 index 00000000..2fbb50c4 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/to-remote/remote_host @@ -0,0 +1 @@ +localhost diff --git a/software/ccollect/ccollect-0.8/conf/sources/to-remote/source b/software/ccollect/ccollect-0.8/conf/sources/to-remote/source new file mode 100644 index 00000000..e64611b7 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/to-remote/source @@ -0,0 +1 @@ +/home/users/nico/bin diff --git a/software/ccollect/ccollect-0.8/conf/sources/very_verbose/README b/software/ccollect/ccollect-0.8/conf/sources/very_verbose/README new file mode 100644 index 00000000..c778cfe8 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/very_verbose/README @@ -0,0 +1 @@ +This is based on a production example I use for my notebook. diff --git a/software/ccollect/ccollect-0.8/conf/sources/very_verbose/destination b/software/ccollect/ccollect-0.8/conf/sources/very_verbose/destination new file mode 100644 index 00000000..8cac69d3 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/very_verbose/destination @@ -0,0 +1 @@ +/tmp/ccollect diff --git a/software/ccollect/ccollect-0.8/conf/sources/very_verbose/exclude b/software/ccollect/ccollect-0.8/conf/sources/very_verbose/exclude new file mode 100644 index 00000000..f5b5b7c8 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/very_verbose/exclude @@ -0,0 +1 @@ +/home/server/raid diff --git a/software/ccollect/ccollect-0.8/conf/sources/very_verbose/source b/software/ccollect/ccollect-0.8/conf/sources/very_verbose/source new file mode 100644 index 00000000..e64611b7 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/very_verbose/source @@ -0,0 +1 @@ +/home/users/nico/bin diff --git a/software/ccollect/ccollect-0.8/conf/sources/very_verbose/summary b/software/ccollect/ccollect-0.8/conf/sources/very_verbose/summary new file mode 100644 index 00000000..e69de29b diff --git a/software/ccollect/ccollect-0.8/conf/sources/very_verbose/verbose b/software/ccollect/ccollect-0.8/conf/sources/very_verbose/verbose new file mode 100644 index 00000000..e69de29b diff --git a/software/ccollect/ccollect-0.8/conf/sources/very_verbose/very_verbose b/software/ccollect/ccollect-0.8/conf/sources/very_verbose/very_verbose new file mode 100644 index 00000000..e69de29b diff --git a/software/ccollect/ccollect-0.8/conf/sources/with_exec/destination b/software/ccollect/ccollect-0.8/conf/sources/with_exec/destination new file mode 100644 index 00000000..8cac69d3 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/with_exec/destination @@ -0,0 +1 @@ +/tmp/ccollect diff --git a/software/ccollect/ccollect-0.8/conf/sources/with_exec/post_exec b/software/ccollect/ccollect-0.8/conf/sources/with_exec/post_exec new file mode 100644 index 00000000..fb8e8e05 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/with_exec/post_exec @@ -0,0 +1,5 @@ +#!/bin/sh + +# Show whats free after + +df -h diff --git a/software/ccollect/ccollect-0.8/conf/sources/with_exec/pre_exec b/software/ccollect/ccollect-0.8/conf/sources/with_exec/pre_exec new file mode 100644 index 00000000..869e6d6a --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/with_exec/pre_exec @@ -0,0 +1,5 @@ +#!/bin/sh + +# Show whats free before + +df -h diff --git a/software/ccollect/ccollect-0.8/conf/sources/with_exec/source b/software/ccollect/ccollect-0.8/conf/sources/with_exec/source new file mode 100644 index 00000000..e64611b7 --- /dev/null +++ b/software/ccollect/ccollect-0.8/conf/sources/with_exec/source @@ -0,0 +1 @@ +/home/users/nico/bin diff --git a/software/ccollect/ccollect-0.8/contrib/README b/software/ccollect/ccollect-0.8/contrib/README new file mode 100644 index 00000000..ab30b7a7 --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/README @@ -0,0 +1,3 @@ +This directory contains patches or programs contributed by others +which are either not yet integrated into ccollect or may be kept +seperated generally. diff --git a/software/ccollect/ccollect-0.8/contrib/jbrendel-autobackup/backup.sh b/software/ccollect/ccollect-0.8/contrib/jbrendel-autobackup/backup.sh new file mode 100644 index 00000000..ea21635c --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jbrendel-autobackup/backup.sh @@ -0,0 +1,22 @@ +#!/bin/bash + +function mkbackup { + find /etc/ccollect/logwrapper/destination -type f -atime +2 -exec sudo rm {} \; + /home/jcb/bm.pl & +} + +mkdir -p /media/backupdisk +grep backupdisk /etc/mtab &> /dev/null + +if [ $? == 0 ] +then + mkbackup +else + mount /media/backupdisk + if [ $? == 0 ] + then + mkbackup + else + echo "Error mounting backup disk" + fi +fi diff --git a/software/ccollect/ccollect-0.8/contrib/jbrendel-autobackup/bm.pl b/software/ccollect/ccollect-0.8/contrib/jbrendel-autobackup/bm.pl new file mode 100644 index 00000000..3a3da84e --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jbrendel-autobackup/bm.pl @@ -0,0 +1,242 @@ +#!/usr/bin/perl + +############################### +# +# Jens-Christoph Brendel, 2009 +# licensed under GPL3 NO WARRANTY +# +############################### + +use Date::Calc qw(:all); +use strict; +use warnings; + +# +#!!!!!!!!!!!!!!!!! you need to customize these settings !!!!!!!!!!!!!!!!!!!! +# +my $backupdir = "/media/backupdisk"; +my $logwrapper = "/home/jcb/ccollect/tools/ccollect_logwrapper.sh"; + +#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +# +------------------------------------------------------------------------+ +# | | +# | V A R I A B L E S | +# | | +# +------------------------------------------------------------------------+ +# + +# get the current date +# +my ($sek, $min, $hour, $day, $month, $year) = localtime(); + +my $curr_year = $year + 1900; +my $curr_month = $month +1; +my ($curr_week,$cur_year) = Week_of_Year($curr_year,$curr_month,$day); + +# initialize some variables +# +my %most_recent_daily = ( + 'age' => 9999, + 'file' => '' +); + +my %most_recent_weekly = ( + 'age' => 9999, + 'file' => '' +); + +my %most_recent_monthly = ( + 'age' => 9999, + 'file' => '' +); + +# prepare the output formatting +# +#--------------------------------------------------------------------------- +my ($msg1, $msg2, $msg3, $msg4); + +format = + @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< + $msg1 + @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< @<<<<<<<<<<<<<<<<< + $msg2, $msg3 + + @|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| + $msg4 +. + +my @months = (' ','January', 'February', 'March', 'April', + 'May', 'June', 'July', 'August', + 'September', 'October', 'November', + 'December'); + +# +------------------------------------------------------------------------+ +# | | +# | P r o c e d u r e s | +# | | +# +------------------------------------------------------------------------+ +# + +# PURPOSE: extract the date from the file name +# PARAMETER VALUE: file name +# RETURN VALUE: pointer of a hash containing year, month, day +# +sub decodeDate { + my $file = shift; + $file =~ /^(daily|weekly|monthly)\.(\d+)-.*/; + my %date = ( + 'y' => substr($2,0,4), + 'm' => substr($2,4,2), + 'd' => substr($2,6,2) + ); + return \%date; +} + +# PURPOSE: calculate the file age in days +# PARAMETER VALUE: name of a ccollect backup file +# RETURN VALUE: age in days +# +sub AgeInDays { + my $file = shift; + my $date=decodeDate($file); + my $ageindays = Delta_Days($$date{'y'}, $$date{'m'}, $$date{'d'}, $curr_year, $curr_month, $day); + return $ageindays; +} + +# PURPOSE: calculate the file age in number of weeks +# PARAMETER VALUE: name of a ccollect backup file +# RETURN VALUE: age in weeks +# +sub AgeInWeeks { + my($y,$m,$d); + + my $file = shift; + my $date = decodeDate($file); + my ($weeknr,$yr) = Week_of_Year($$date{'y'}, $$date{'m'}, $$date{'d'}); + my $ageinweeks = $curr_week - $weeknr; + return $ageinweeks; +} + +# PURPOSE: calculate the file age in number of months +# PARAMETER VALUE: name of a ccollect backup file +# RETURN VALUE: age in months +# +sub AgeInMonths { + my $ageinmonths; + my $ageinmonths; + my $file = shift; + my $date = decodeDate($file); + if ($curr_year == $$date{'y'}) { + $ageinmonths = $curr_month - $$date{'m'}; + } else { + $ageinmonths = $curr_month + (12-$$date{'m'}) + ($curr_year-$$date{'y'}-1)*12; + } + return $ageinmonths; +} + +# +------------------------------------------------------------------------+ +# | | +# | M A I N | +# | | +# +------------------------------------------------------------------------+ +# + +# +# find the most recent daily, weekly and monthly backup file +# + +opendir(DIRH, $backupdir) or die "Can't open $backupdir \n"; + +my @files = readdir(DIRH); + +die "Zielverzeichnis leer \n" if ( $#files <= 1 ); + +foreach my $file (@files) { + + next if $file eq "." or $file eq ".."; + + SWITCH: { + if ($file =~ /^daily/) { + my $curr_age=AgeInDays($file); + if ($curr_age<$most_recent_daily{'age'}) { + $most_recent_daily{'age'} =$curr_age; + $most_recent_daily{'file'}= $file; + } + last SWITCH; + } + + if ($file =~ /^weekly/) { + my $curr_week_age = AgeInWeeks($file); + if ($curr_week_age<$most_recent_weekly{'age'}) { + $most_recent_weekly{'age'} =$curr_week_age; + $most_recent_weekly{'file'}=$file; + } + last SWITCH; + } + + if ($file =~ /^monthly/) { + my $curr_month_age=AgeInMonths($file); + if ($curr_month_age < $most_recent_monthly{'age'}) { + $most_recent_monthly{'age'} =$curr_month_age; + $most_recent_monthly{'file'}=$file; + } + last SWITCH; + } + print "\n\n unknown file $file \n\n"; + } +} + +printf("\nBackup Manager started: %02u.%02u. %u, week %02u\n\n", $day, $curr_month, $curr_year, $curr_week); + +# +# compare the most recent daily, weekly and monthly backup file +# and decide if it's necessary to start a new backup process in +# each category +# + +if ($most_recent_monthly{'age'} == 0) { + $msg1="The most recent monthly backup"; + $msg2="$most_recent_monthly{'file'} from $months[$curr_month - $most_recent_monthly{'age'}]"; + $msg3="is still valid."; + $msg4=""; + write; +} else { + $msg1="The most recent monthly backup"; + $msg2="$most_recent_monthly{'file'} from $months[$curr_month - $most_recent_monthly{'age'}]"; + $msg3="is out-dated."; + $msg4="Starting new monthly backup."; + write; + exec "sudo $logwrapper monthly FULL"; + exit; +} + +if ($most_recent_weekly{'age'} == 0) { + $msg1="The most recent weekly backup"; + $msg2="$most_recent_weekly{'file'} from week nr: $curr_week-$most_recent_weekly{'age'}"; + $msg3="is still valid."; + $msg4=""; + write; +} else { + $msg1="The most recent weekly backup"; + $msg2="$most_recent_weekly{'file'} from week nr: $curr_week-$most_recent_weekly{'age'}"; + $msg3="is out-dated."; + $msg4="Starting new weekly backup."; + write; + exec "sudo $logwrapper weekly FULL"; + exit; +} + +if ($most_recent_daily{'age'} == 0 ) { + $msg1=" The most recent daily backup"; + $msg2="$most_recent_daily{'file'}"; + $msg3="is still valid."; + $msg4=""; + write; +} else { + $msg1="The most recent daily backup"; + $msg2="$most_recent_daily{'file'}"; + $msg3="is out-dated."; + $msg4="Starting new daily backup."; + write; + exec "sudo $logwrapper daily FULL"; diff --git a/software/ccollect/ccollect-0.8/contrib/jbrendel-autobackup/correction_1 b/software/ccollect/ccollect-0.8/contrib/jbrendel-autobackup/correction_1 new file mode 100644 index 00000000..4fec4408 --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jbrendel-autobackup/correction_1 @@ -0,0 +1,3 @@ +- Zeile 126/127 (my $ageinmonths;) ist doppelt, einmal streichen. +- in die allerletzte Zeile gehört eine schließende geschweifte Klammer +"}", die irgendwo verlorengegangen ist. diff --git a/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/README_g-i.txt b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/README_g-i.txt new file mode 100644 index 00000000..b782d63d --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/README_g-i.txt @@ -0,0 +1,15 @@ +Hello Nico, + + I have attached three more patches for ccollect. Each patch +has comments explaining its motivation. + + All of these patches work-for-me (but I continue to test +them). I would be interested in your opinion on, for example, the +general approach used in i.patch which changes the way options are +handled. I think it is a big improvement. If, however, you wanted +the code to go in a different direction, let me know before we +diverge too far. + +Regards, + +John diff --git a/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/ccollect-f.sh b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/ccollect-f.sh new file mode 100644 index 00000000..5c8952e8 --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/ccollect-f.sh @@ -0,0 +1,683 @@ +#!/bin/sh +# +# 2005-2009 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Initially written for SyGroup (www.sygroup.ch) +# Date: Mon Nov 14 11:45:11 CET 2005 + +# +# Standard variables (stolen from cconf) +# +__pwd="$(pwd -P)" +__mydir="${0%/*}"; __abs_mydir="$(cd "$__mydir" && pwd -P)" +__myname=${0##*/}; __abs_myname="$__abs_mydir/$__myname" + +# +# where to find our configuration and temporary file +# +CCOLLECT_CONF=${CCOLLECT_CONF:-/etc/ccollect} +CSOURCES=${CCOLLECT_CONF}/sources +CDEFAULTS=${CCOLLECT_CONF}/defaults +CPREEXEC="${CDEFAULTS}/pre_exec" +CPOSTEXEC="${CDEFAULTS}/post_exec" + +TMP=$(mktemp "/tmp/${__myname}.XXXXXX") +VERSION=0.7.1 +RELEASE="2009-02-02" +HALF_VERSION="ccollect ${VERSION}" +FULL_VERSION="ccollect ${VERSION} (${RELEASE})" + +#TSORT="tc" ; NEWER="cnewer" +TSORT="t" ; NEWER="newer" + +# +# CDATE: how we use it for naming of the archives +# DDATE: how the user should see it in our output (DISPLAY) +# +CDATE="date +%Y%m%d-%H%M" +DDATE="date +%Y-%m-%d-%H:%M:%S" + +# +# unset parallel execution +# +PARALLEL="" + +# +# catch signals +# +trap "rm -f \"${TMP}\"" 1 2 15 + +# +# Functions +# + +# time displaying echo +_techo() +{ + echo "$(${DDATE}): $@" +} + +# exit on error +_exit_err() +{ + _techo "$@" + rm -f "${TMP}" + exit 1 +} + +add_name() +{ + awk "{ print \"[${name}] \" \$0 }" +} + +pcmd() +{ + if [ "$remote_host" ]; then + ssh "$remote_host" "$@" + else + "$@" + fi +} + +# +# Version +# +display_version() +{ + echo "${FULL_VERSION}" + exit 0 +} + +# +# Tell how to use us +# +usage() +{ + echo "${__myname}: [args] " + echo "" + echo " ccollect creates (pseudo) incremental backups" + echo "" + echo " -h, --help: Show this help screen" + echo " -p, --parallel: Parallelise backup processes" + echo " -a, --all: Backup all sources specified in ${CSOURCES}" + echo " -v, --verbose: Be very verbose (uses set -x)" + echo " -V, --version: Print version information" + echo "" + echo " This is version ${VERSION}, released on ${RELEASE}" + echo " (the first version was written on 2005-12-05 by Nico Schottelius)." + echo "" + echo " Retrieve latest ccollect at http://unix.schottelius.org/ccollect/" + exit 0 +} + +# +# Select interval if AUTO +# +# For this to work nicely, you have to choose interval names that sort nicely +# such as int1, int2, int3 or a_daily, b_weekly, c_monthly, etc. +# +auto_interval() +{ + if [ -d "${backup}/intervals" -a -n "$(ls "${backup}/intervals" 2>/dev/null)" ] ; then + intervals_dir="${backup}/intervals" + elif [ -d "${CDEFAULTS}/intervals" -a -n "$(ls "${CDEFAULTS}/intervals" 2>/dev/null)" ] ; then + intervals_dir="${CDEFAULTS}/intervals" + else + _exit_err "No intervals are defined. Skipping." + fi + echo intervals_dir=${intervals_dir} + + trial_interval="$(ls -1r "${intervals_dir}/" | head -n 1)" || \ + _exit_err "Failed to list contents of ${intervals_dir}/." + _techo "Considering interval ${trial_interval}" + most_recent="$(pcmd ls -${TSORT}p1 "${ddir}" | grep "^${trial_interval}.*/$" | head -n 1)" || \ + _exit_err "Failed to list contents of ${ddir}/." + _techo " Most recent ${trial_interval}: '${most_recent}'" + if [ -n "${most_recent}" ]; then + no_intervals="$(ls -1 "${intervals_dir}/" | wc -l)" + n=1 + while [ "${n}" -le "${no_intervals}" ]; do + trial_interval="$(ls -p1 "${intervals_dir}/" | tail -n+${n} | head -n 1)" + _techo "Considering interval '${trial_interval}'" + c_interval="$(cat "${intervals_dir}/${trial_interval}" 2>/dev/null)" + m=$((${n}+1)) + set -- "${ddir}" -maxdepth 1 + while [ "${m}" -le "${no_intervals}" ]; do + interval_m="$(ls -1 "${intervals_dir}/" | tail -n+${m} | head -n 1)" + most_recent="$(pcmd ls -${TSORT}p1 "${ddir}" | grep "^${interval_m}\..*/$" | head -n 1)" + _techo " Most recent ${interval_m}: '${most_recent}'" + if [ -n "${most_recent}" ] ; then + set -- "$@" -$NEWER "${ddir}/${most_recent}" + fi + m=$((${m}+1)) + done + count=$(pcmd find "$@" -iname "${trial_interval}*" | wc -l) + _techo " Found $count more recent backups of ${trial_interval} (limit: ${c_interval})" + if [ "$count" -lt "${c_interval}" ] ; then + break + fi + n=$((${n}+1)) + done + fi + export INTERVAL="${trial_interval}" + D_FILE_INTERVAL="${intervals_dir}/${INTERVAL}" + D_INTERVAL=$(cat "${D_FILE_INTERVAL}" 2>/dev/null) +} + +# +# need at least interval and one source or --all +# +if [ $# -lt 2 ]; then + if [ "$1" = "-V" -o "$1" = "--version" ]; then + display_version + else + usage + fi +fi + +# +# check for configuraton directory +# +[ -d "${CCOLLECT_CONF}" ] || _exit_err "No configuration found in " \ + "\"${CCOLLECT_CONF}\" (is \$CCOLLECT_CONF properly set?)" + +# +# Filter arguments +# +export INTERVAL="$1"; shift +i=1 +no_sources=0 + +# +# Create source "array" +# +while [ "$#" -ge 1 ]; do + eval arg=\"\$1\"; shift + + if [ "${NO_MORE_ARGS}" = 1 ]; then + eval source_${no_sources}=\"${arg}\" + no_sources=$((${no_sources}+1)) + + # make variable available for subscripts + eval export source_${no_sources} + else + case "${arg}" in + -a|--all) + ALL=1 + ;; + -v|--verbose) + VERBOSE=1 + ;; + -p|--parallel) + PARALLEL=1 + ;; + -h|--help) + usage + ;; + --) + NO_MORE_ARGS=1 + ;; + *) + eval source_${no_sources}=\"$arg\" + no_sources=$(($no_sources+1)) + ;; + esac + fi + + i=$(($i+1)) +done + +# also export number of sources +export no_sources + +# +# be really, really, really verbose +# +if [ "${VERBOSE}" = 1 ]; then + set -x +fi + +# +# Look, if we should take ALL sources +# +if [ "${ALL}" = 1 ]; then + # reset everything specified before + no_sources=0 + + # + # get entries from sources + # + cwd=$(pwd -P) + ( cd "${CSOURCES}" && ls > "${TMP}" ); ret=$? + + [ "${ret}" -eq 0 ] || _exit_err "Listing of sources failed. Aborting." + + while read tmp; do + eval source_${no_sources}=\"${tmp}\" + no_sources=$((${no_sources}+1)) + done < "${TMP}" +fi + +# +# Need at least ONE source to backup +# +if [ "${no_sources}" -lt 1 ]; then + usage +else + _techo "${HALF_VERSION}: Beginning backup using interval ${INTERVAL}" +fi + +# +# Look for pre-exec command (general) +# +if [ -x "${CPREEXEC}" ]; then + _techo "Executing ${CPREEXEC} ..." + "${CPREEXEC}"; ret=$? + _techo "Finished ${CPREEXEC} (return code: ${ret})." + + [ "${ret}" -eq 0 ] || _exit_err "${CPREEXEC} failed. Aborting" +fi + +# +# check default configuration +# + +D_FILE_INTERVAL="${CDEFAULTS}/intervals/${INTERVAL}" +D_INTERVAL=$(cat "${D_FILE_INTERVAL}" 2>/dev/null) + + +# +# Let's do the backup +# +i=0 +while [ "${i}" -lt "${no_sources}" ]; do + + # + # Get current source + # + eval name=\"\$source_${i}\" + i=$((${i}+1)) + + export name + + # + # start ourself, if we want parallel execution + # + if [ "${PARALLEL}" ]; then + "$0" "${INTERVAL}" "${name}" & + continue + fi + +# +# Start subshell for easy log editing +# +( + # + # Stderr to stdout, so we can produce nice logs + # + exec 2>&1 + + # + # Configuration + # + backup="${CSOURCES}/${name}" + c_source="${backup}/source" + c_dest="${backup}/destination" + c_exclude="${backup}/exclude" + c_verbose="${backup}/verbose" + c_vverbose="${backup}/very_verbose" + c_rsync_extra="${backup}/rsync_options" + c_summary="${backup}/summary" + c_pre_exec="${backup}/pre_exec" + c_post_exec="${backup}/post_exec" + f_incomplete="delete_incomplete" + c_incomplete="${backup}/${f_incomplete}" + c_remote_host="${backup}/remote_host" + + # + # Marking backups: If we abort it's not removed => Backup is broken + # + c_marker=".ccollect-marker" + + # + # Times + # + begin_s=$(date +%s) + + # + # unset possible options + # + EXCLUDE="" + RSYNC_EXTRA="" + SUMMARY="" + VERBOSE="" + VVERBOSE="" + DELETE_INCOMPLETE="" + + _techo "Beginning to backup" + + # + # Standard configuration checks + # + if [ ! -e "${backup}" ]; then + _exit_err "Source does not exist." + fi + + # + # configuration _must_ be a directory + # + if [ ! -d "${backup}" ]; then + _exit_err "\"${name}\" is not a cconfig-directory. Skipping." + fi + + # + # first execute pre_exec, which may generate destination or other + # parameters + # + if [ -x "${c_pre_exec}" ]; then + _techo "Executing ${c_pre_exec} ..." + "${c_pre_exec}"; ret="$?" + _techo "Finished ${c_pre_exec} (return code ${ret})." + + if [ "${ret}" -ne 0 ]; then + _exit_err "${c_pre_exec} failed. Skipping." + fi + fi + + # + # Destination is a path + # + if [ ! -f "${c_dest}" ]; then + _exit_err "Destination ${c_dest} is not a file. Skipping." + else + ddir=$(cat "${c_dest}"); ret="$?" + if [ "${ret}" -ne 0 ]; then + _exit_err "Destination ${c_dest} is not readable. Skipping." + fi + fi + + # + # interval definition: First try source specific, fallback to default + # + if [ ${INTERVAL} = "AUTO" ] ; then + auto_interval + _techo "Selected interval: '$INTERVAL'" + fi + c_interval="$(cat "${backup}/intervals/${INTERVAL}" 2>/dev/null)" + + if [ -z "${c_interval}" ]; then + c_interval="${D_INTERVAL}" + + if [ -z "${c_interval}" ]; then + _exit_err "No definition for interval \"${INTERVAL}\" found. Skipping." + fi + fi + + # + # Source checks + # + if [ ! -f "${c_source}" ]; then + _exit_err "Source description \"${c_source}\" is not a file. Skipping." + else + source=$(cat "${c_source}"); ret="$?" + if [ "${ret}" -ne 0 ]; then + _exit_err "Source ${c_source} is not readable. Skipping." + fi + fi + # Verify source is up and accepting connections before deleting any old backups + rsync "$source" >/dev/null || _exit_err "Source ${source} is not readable. Skipping." + + # + # do we backup to a remote host? then set pre-cmd + # + if [ -f "${c_remote_host}" ]; then + # adjust ls and co + remote_host=$(cat "${c_remote_host}"); ret="$?" + if [ "${ret}" -ne 0 ]; then + _exit_err "Remote host file ${c_remote_host} exists, but is not readable. Skipping." + fi + destination="${remote_host}:${ddir}" + else + remote_host="" + destination="${ddir}" + fi + export remote_host + + # + # check for existence / use real name + # + ( pcmd cd "$ddir" ) || _exit_err "Cannot change to ${ddir}. Skipping." + + + # + # Check whether to delete incomplete backups + # + if [ -f "${c_incomplete}" -o -f "${CDEFAULTS}/${f_incomplete}" ]; then + DELETE_INCOMPLETE="yes" + fi + + # NEW method as of 0.6: + # - insert ccollect default parameters + # - insert options + # - insert user options + + # + # rsync standard options + # + + set -- "$@" "--archive" "--delete" "--numeric-ids" "--relative" \ + "--delete-excluded" "--sparse" + + # + # exclude list + # + if [ -f "${c_exclude}" ]; then + set -- "$@" "--exclude-from=${c_exclude}" + fi + + # + # Output a summary + # + if [ -f "${c_summary}" ]; then + set -- "$@" "--stats" + fi + + # + # Verbosity for rsync + # + if [ -f "${c_vverbose}" ]; then + set -- "$@" "-vv" + elif [ -f "${c_verbose}" ]; then + set -- "$@" "-v" + fi + + # + # extra options for rsync provided by the user + # + if [ -f "${c_rsync_extra}" ]; then + while read line; do + set -- "$@" "$line" + done < "${c_rsync_extra}" + fi + + # + # Check for incomplete backups + # + pcmd ls -1 "$ddir/${INTERVAL}"*".${c_marker}" > "${TMP}" 2>/dev/null + + i=0 + while read incomplete; do + eval incomplete_$i=\"$(echo ${incomplete} | sed "s/\\.${c_marker}\$//")\" + i=$(($i+1)) + done < "${TMP}" + + j=0 + while [ "$j" -lt "$i" ]; do + eval realincomplete=\"\$incomplete_$j\" + _techo "Incomplete backup: ${realincomplete}" + if [ "${DELETE_INCOMPLETE}" = "yes" ]; then + _techo "Deleting ${realincomplete} ..." + pcmd rm $VVERBOSE -rf "${ddir}/${realincomplete}" || \ + _exit_err "Removing ${realincomplete} failed." + fi + j=$(($j+1)) + done + + # + # check if maximum number of backups is reached, if so remove + # use grep and ls -p so we only look at directories + # + count="$(pcmd ls -p1 "${ddir}" | grep "^${INTERVAL}\..*/\$" | wc -l \ + | sed 's/^ *//g')" || _exit_err "Counting backups failed" + + _techo "Existing backups: ${count} Total keeping backups: ${c_interval}" + + if [ "${count}" -ge "${c_interval}" ]; then + substract=$((${c_interval} - 1)) + remove=$((${count} - ${substract})) + _techo "Removing ${remove} backup(s)..." + + pcmd ls -${TSORT}p1r "$ddir" | grep "^${INTERVAL}\..*/\$" | \ + head -n "${remove}" > "${TMP}" || \ + _exit_err "Listing old backups failed" + + i=0 + while read to_remove; do + eval remove_$i=\"${to_remove}\" + i=$(($i+1)) + done < "${TMP}" + + j=0 + while [ "$j" -lt "$i" ]; do + eval to_remove=\"\$remove_$j\" + _techo "Removing ${to_remove} ..." + pcmd rm ${VVERBOSE} -rf "${ddir}/${to_remove}" || \ + _exit_err "Removing ${to_remove} failed." + j=$(($j+1)) + done + fi + + + # + # Check for backup directory to clone from: Always clone from the latest one! + # + # Depending on your file system, you may want to sort on: + # 1. mtime (modification time) with TSORT=t, or + # 2. ctime (last change time, usually) with TSORT=tc + last_dir="$(pcmd ls -${TSORT}p1 "${ddir}" | grep '/$' | head -n 1)" || \ + _exit_err "Failed to list contents of ${ddir}." + + # + # clone from old backup, if existing + # + if [ "${last_dir}" ]; then + set -- "$@" "--link-dest=${ddir}/${last_dir}" + _techo "Hard linking from ${last_dir}" + fi + + + # set time when we really begin to backup, not when we began to remove above + destination_date=$(${CDATE}) + destination_dir="${ddir}/${INTERVAL}.${destination_date}.$$" + destination_full="${destination}/${INTERVAL}.${destination_date}.$$" + + # give some info + _techo "Beginning to backup, this may take some time..." + + _techo "Creating ${destination_dir} ..." + pcmd mkdir ${VVERBOSE} "${destination_dir}" || \ + _exit_err "Creating ${destination_dir} failed. Skipping." + + # + # added marking in 0.6 (and remove it, if successful later) + # + pcmd touch "${destination_dir}.${c_marker}" + + # + # the rsync part + # + _techo "Transferring files..." + rsync "$@" "${source}" "${destination_full}"; ret=$? + # Correct the modification time: + pcmd touch "${destination_dir}" + + # + # remove marking here + # + if [ "$ret" -ne 12 ] ; then + pcmd rm "${destination_dir}.${c_marker}" || \ + _exit_err "Removing ${destination_dir}/${c_marker} failed." + fi + + _techo "Finished backup (rsync return code: $ret)." + if [ "${ret}" -ne 0 ]; then + _techo "Warning: rsync exited non-zero, the backup may be broken (see rsync errors)." + fi + + # + # post_exec + # + if [ -x "${c_post_exec}" ]; then + _techo "Executing ${c_post_exec} ..." + "${c_post_exec}"; ret=$? + _techo "Finished ${c_post_exec}." + + if [ ${ret} -ne 0 ]; then + _exit_err "${c_post_exec} failed." + fi + fi + + # Calculation + end_s=$(date +%s) + + full_seconds=$((${end_s} - ${begin_s})) + hours=$((${full_seconds} / 3600)) + seconds=$((${full_seconds} - (${hours} * 3600))) + minutes=$((${seconds} / 60)) + seconds=$((${seconds} - (${minutes} * 60))) + + _techo "Backup lasted: ${hours}:${minutes}:${seconds} (h:m:s)" + +) | add_name +done + +# +# Be a good parent and wait for our children, if they are running wild parallel +# +if [ "${PARALLEL}" ]; then + _techo "Waiting for children to complete..." + wait +fi + +# +# Look for post-exec command (general) +# +if [ -x "${CPOSTEXEC}" ]; then + _techo "Executing ${CPOSTEXEC} ..." + "${CPOSTEXEC}"; ret=$? + _techo "Finished ${CPOSTEXEC} (return code: ${ret})." + + if [ ${ret} -ne 0 ]; then + _techo "${CPOSTEXEC} failed." + fi +fi + +rm -f "${TMP}" +_techo "Finished ${WE}" + +# vim: set shiftwidth=3 tabstop=3 expandtab : diff --git a/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/ccollect-i.sh b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/ccollect-i.sh new file mode 100644 index 00000000..58fab09d --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/ccollect-i.sh @@ -0,0 +1,663 @@ +#!/bin/sh +# +# 2005-2009 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Initially written for SyGroup (www.sygroup.ch) +# Date: Mon Nov 14 11:45:11 CET 2005 + +# +# Standard variables (stolen from cconf) +# +__pwd="$(pwd -P)" +__mydir="${0%/*}"; __abs_mydir="$(cd "$__mydir" && pwd -P)" +__myname=${0##*/}; __abs_myname="$__abs_mydir/$__myname" + +# +# where to find our configuration and temporary file +# +CCOLLECT_CONF=${CCOLLECT_CONF:-/etc/ccollect} +CSOURCES=${CCOLLECT_CONF}/sources +CDEFAULTS=${CCOLLECT_CONF}/defaults +CPREEXEC="${CDEFAULTS}/pre_exec" +CPOSTEXEC="${CDEFAULTS}/post_exec" + +TMP=$(mktemp "/tmp/${__myname}.XXXXXX") +VERSION=0.7.1 +RELEASE="2009-02-02" +HALF_VERSION="ccollect ${VERSION}" +FULL_VERSION="ccollect ${VERSION} (${RELEASE})" + +#TSORT="tc" ; NEWER="cnewer" +TSORT="t" ; NEWER="newer" + +# +# CDATE: how we use it for naming of the archives +# DDATE: how the user should see it in our output (DISPLAY) +# +CDATE="date +%Y%m%d-%H%M" +DDATE="date +%Y-%m-%d-%H:%M:%S" + +# +# unset parallel execution +# +PARALLEL="" + +# +# catch signals +# +trap "rm -f \"${TMP}\"" 1 2 15 + +# +# Functions +# + +# time displaying echo +_techo() +{ + echo "$(${DDATE}): $@" +} + +# exit on error +_exit_err() +{ + _techo "$@" + rm -f "${TMP}" + exit 1 +} + +add_name() +{ + awk "{ print \"[${name}] \" \$0 }" +} + +pcmd() +{ + if [ "$remote_host" ]; then + ssh "$remote_host" "$@" + else + "$@" + fi +} + +# +# Version +# +display_version() +{ + echo "${FULL_VERSION}" + exit 0 +} + +# +# Tell how to use us +# +usage() +{ + echo "${__myname}: [args] " + echo "" + echo " ccollect creates (pseudo) incremental backups" + echo "" + echo " -h, --help: Show this help screen" + echo " -p, --parallel: Parallelise backup processes" + echo " -a, --all: Backup all sources specified in ${CSOURCES}" + echo " -v, --verbose: Be very verbose (uses set -x)" + echo " -V, --version: Print version information" + echo "" + echo " This is version ${VERSION}, released on ${RELEASE}" + echo " (the first version was written on 2005-12-05 by Nico Schottelius)." + echo "" + echo " Retrieve latest ccollect at http://unix.schottelius.org/ccollect/" + exit 0 +} + +# +# Select interval if AUTO +# +# For this to work nicely, you have to choose interval names that sort nicely +# such as int1, int2, int3 or a_daily, b_weekly, c_monthly, etc. +# +auto_interval() +{ + if [ -d "${backup}/intervals" -a -n "$(ls "${backup}/intervals" 2>/dev/null)" ] ; then + intervals_dir="${backup}/intervals" + elif [ -d "${CDEFAULTS}/intervals" -a -n "$(ls "${CDEFAULTS}/intervals" 2>/dev/null)" ] ; then + intervals_dir="${CDEFAULTS}/intervals" + else + _exit_err "No intervals are defined. Skipping." + fi + echo intervals_dir=${intervals_dir} + + trial_interval="$(ls -1r "${intervals_dir}/" | head -n 1)" || \ + _exit_err "Failed to list contents of ${intervals_dir}/." + _techo "Considering interval ${trial_interval}" + most_recent="$(pcmd ls -${TSORT}p1 "${ddir}" | grep "^${trial_interval}.*/$" | head -n 1)" || \ + _exit_err "Failed to list contents of ${ddir}/." + _techo " Most recent ${trial_interval}: '${most_recent}'" + if [ -n "${most_recent}" ]; then + no_intervals="$(ls -1 "${intervals_dir}/" | wc -l)" + n=1 + while [ "${n}" -le "${no_intervals}" ]; do + trial_interval="$(ls -p1 "${intervals_dir}/" | tail -n+${n} | head -n 1)" + _techo "Considering interval '${trial_interval}'" + c_interval="$(cat "${intervals_dir}/${trial_interval}" 2>/dev/null)" + m=$((${n}+1)) + set -- "${ddir}" -maxdepth 1 + while [ "${m}" -le "${no_intervals}" ]; do + interval_m="$(ls -1 "${intervals_dir}/" | tail -n+${m} | head -n 1)" + most_recent="$(pcmd ls -${TSORT}p1 "${ddir}" | grep "^${interval_m}\..*/$" | head -n 1)" + _techo " Most recent ${interval_m}: '${most_recent}'" + if [ -n "${most_recent}" ] ; then + set -- "$@" -$NEWER "${ddir}/${most_recent}" + fi + m=$((${m}+1)) + done + count=$(pcmd find "$@" -iname "${trial_interval}*" | wc -l) + _techo " Found $count more recent backups of ${trial_interval} (limit: ${c_interval})" + if [ "$count" -lt "${c_interval}" ] ; then + break + fi + n=$((${n}+1)) + done + fi + export INTERVAL="${trial_interval}" + D_FILE_INTERVAL="${intervals_dir}/${INTERVAL}" + D_INTERVAL=$(cat "${D_FILE_INTERVAL}" 2>/dev/null) +} + +# +# need at least interval and one source or --all +# +if [ $# -lt 2 ]; then + if [ "$1" = "-V" -o "$1" = "--version" ]; then + display_version + else + usage + fi +fi + +# +# check for configuraton directory +# +[ -d "${CCOLLECT_CONF}" ] || _exit_err "No configuration found in " \ + "\"${CCOLLECT_CONF}\" (is \$CCOLLECT_CONF properly set?)" + +# +# Filter arguments +# +export INTERVAL="$1"; shift +i=1 +no_sources=0 + +# +# Create source "array" +# +while [ "$#" -ge 1 ]; do + eval arg=\"\$1\"; shift + + if [ "${NO_MORE_ARGS}" = 1 ]; then + eval source_${no_sources}=\"${arg}\" + no_sources=$((${no_sources}+1)) + + # make variable available for subscripts + eval export source_${no_sources} + else + case "${arg}" in + -a|--all) + ALL=1 + ;; + -v|--verbose) + VERBOSE=1 + ;; + -p|--parallel) + PARALLEL=1 + ;; + -h|--help) + usage + ;; + --) + NO_MORE_ARGS=1 + ;; + *) + eval source_${no_sources}=\"$arg\" + no_sources=$(($no_sources+1)) + ;; + esac + fi + + i=$(($i+1)) +done + +# also export number of sources +export no_sources + +# +# be really, really, really verbose +# +if [ "${VERBOSE}" = 1 ]; then + set -x +fi + +# +# Look, if we should take ALL sources +# +if [ "${ALL}" = 1 ]; then + # reset everything specified before + no_sources=0 + + # + # get entries from sources + # + cwd=$(pwd -P) + ( cd "${CSOURCES}" && ls > "${TMP}" ); ret=$? + + [ "${ret}" -eq 0 ] || _exit_err "Listing of sources failed. Aborting." + + while read tmp; do + eval source_${no_sources}=\"${tmp}\" + no_sources=$((${no_sources}+1)) + done < "${TMP}" +fi + +# +# Need at least ONE source to backup +# +if [ "${no_sources}" -lt 1 ]; then + usage +else + _techo "${HALF_VERSION}: Beginning backup using interval ${INTERVAL}" +fi + +# +# Look for pre-exec command (general) +# +if [ -x "${CPREEXEC}" ]; then + _techo "Executing ${CPREEXEC} ..." + "${CPREEXEC}"; ret=$? + _techo "Finished ${CPREEXEC} (return code: ${ret})." + + [ "${ret}" -eq 0 ] || _exit_err "${CPREEXEC} failed. Aborting" +fi + +# +# check default configuration +# + +D_FILE_INTERVAL="${CDEFAULTS}/intervals/${INTERVAL}" +D_INTERVAL=$(cat "${D_FILE_INTERVAL}" 2>/dev/null) + + +# +# Let's do the backup +# +i=0 +while [ "${i}" -lt "${no_sources}" ]; do + + # + # Get current source + # + eval name=\"\$source_${i}\" + i=$((${i}+1)) + + export name + + # + # start ourself, if we want parallel execution + # + if [ "${PARALLEL}" ]; then + "$0" "${INTERVAL}" "${name}" & + continue + fi + +# +# Start subshell for easy log editing +# +( + # + # Stderr to stdout, so we can produce nice logs + # + exec 2>&1 + + # + # Configuration + # + backup="${CSOURCES}/${name}" + c_source="${backup}/source" + c_dest="${backup}/destination" + c_pre_exec="${backup}/pre_exec" + c_post_exec="${backup}/post_exec" + for opt in exclude verbose very_verbose rsync_options summary delete_incomplete remote_host ; do + if [ -f "${backup}/$opt" -o -f "${backup}/no_$opt" ]; then + eval c_$opt=\"${backup}/$opt\" + else + eval c_$opt=\"${CDEFAULTS}/$opt\" + fi + done + + # + # Marking backups: If we abort it's not removed => Backup is broken + # + c_marker=".ccollect-marker" + + # + # Times + # + begin_s=$(date +%s) + + # + # unset possible options + # + VERBOSE="" + VVERBOSE="" + + _techo "Beginning to backup" + + # + # Standard configuration checks + # + if [ ! -e "${backup}" ]; then + _exit_err "Source does not exist." + fi + + # + # configuration _must_ be a directory + # + if [ ! -d "${backup}" ]; then + _exit_err "\"${name}\" is not a cconfig-directory. Skipping." + fi + + # + # first execute pre_exec, which may generate destination or other + # parameters + # + if [ -x "${c_pre_exec}" ]; then + _techo "Executing ${c_pre_exec} ..." + "${c_pre_exec}"; ret="$?" + _techo "Finished ${c_pre_exec} (return code ${ret})." + + if [ "${ret}" -ne 0 ]; then + _exit_err "${c_pre_exec} failed. Skipping." + fi + fi + + # + # Destination is a path + # + if [ ! -f "${c_dest}" ]; then + _exit_err "Destination ${c_dest} is not a file. Skipping." + else + ddir=$(cat "${c_dest}"); ret="$?" + if [ "${ret}" -ne 0 ]; then + _exit_err "Destination ${c_dest} is not readable. Skipping." + fi + fi + + # + # interval definition: First try source specific, fallback to default + # + if [ "${INTERVAL}" = "AUTO" ] ; then + auto_interval + _techo "Selected interval: '$INTERVAL'" + fi + c_interval="$(cat "${backup}/intervals/${INTERVAL}" 2>/dev/null)" + + if [ -z "${c_interval}" ]; then + c_interval="${D_INTERVAL}" + + if [ -z "${c_interval}" ]; then + _exit_err "No definition for interval \"${INTERVAL}\" found. Skipping." + fi + fi + + # + # Source checks + # + if [ ! -f "${c_source}" ]; then + _exit_err "Source description \"${c_source}\" is not a file. Skipping." + else + source=$(cat "${c_source}"); ret="$?" + if [ "${ret}" -ne 0 ]; then + _exit_err "Source ${c_source} is not readable. Skipping." + fi + fi + # Verify source is up and accepting connections before deleting any old backups + rsync "$source" >/dev/null || _exit_err "Source ${source} is not readable. Skipping." + + # + # do we backup to a remote host? then set pre-cmd + # + if [ -f "${c_remote_host}" ]; then + # adjust ls and co + remote_host=$(cat "${c_remote_host}"); ret="$?" + if [ "${ret}" -ne 0 ]; then + _exit_err "Remote host file ${c_remote_host} exists, but is not readable. Skipping." + fi + destination="${remote_host}:${ddir}" + else + remote_host="" + destination="${ddir}" + fi + export remote_host + + # + # check for existence / use real name + # + ( pcmd cd "$ddir" ) || _exit_err "Cannot change to ${ddir}. Skipping." + + + # NEW method as of 0.6: + # - insert ccollect default parameters + # - insert options + # - insert user options + + # + # rsync standard options + # + + set -- "$@" "--archive" "--delete" "--numeric-ids" "--relative" \ + "--delete-excluded" "--sparse" + + # + # exclude list + # + if [ -f "${c_exclude}" ]; then + set -- "$@" "--exclude-from=${c_exclude}" + fi + + # + # Output a summary + # + if [ -f "${c_summary}" ]; then + set -- "$@" "--stats" + fi + + # + # Verbosity for rsync + # + if [ -f "${c_very_verbose}" ]; then + set -- "$@" "-vv" + elif [ -f "${c_verbose}" ]; then + set -- "$@" "-v" + fi + + # + # extra options for rsync provided by the user + # + if [ -f "${c_rsync_options}" ]; then + while read line; do + set -- "$@" "$line" + done < "${c_rsync_options}" + fi + + # + # Check for incomplete backups + # + pcmd ls -1 "$ddir/${INTERVAL}"*".${c_marker}" 2>/dev/null | while read marker; do + incomplete="$(echo ${marker} | sed "s/\\.${c_marker}\$//")" + _techo "Incomplete backup: ${incomplete}" + if [ -f "${c_delete_incomplete}" ]; then + _techo "Deleting ${incomplete} ..." + pcmd rm $VVERBOSE -rf "${incomplete}" || \ + _exit_err "Removing ${incomplete} failed." + pcmd rm $VVERBOSE -f "${marker}" || \ + _exit_err "Removing ${marker} failed." + fi + done + + # + # check if maximum number of backups is reached, if so remove + # use grep and ls -p so we only look at directories + # + count="$(pcmd ls -p1 "${ddir}" | grep "^${INTERVAL}\..*/\$" | wc -l \ + | sed 's/^ *//g')" || _exit_err "Counting backups failed" + + _techo "Existing backups: ${count} Total keeping backups: ${c_interval}" + + if [ "${count}" -ge "${c_interval}" ]; then + substract=$((${c_interval} - 1)) + remove=$((${count} - ${substract})) + _techo "Removing ${remove} backup(s)..." + + pcmd ls -${TSORT}p1r "$ddir" | grep "^${INTERVAL}\..*/\$" | \ + head -n "${remove}" > "${TMP}" || \ + _exit_err "Listing old backups failed" + + i=0 + while read to_remove; do + eval remove_$i=\"${to_remove}\" + i=$(($i+1)) + done < "${TMP}" + + j=0 + while [ "$j" -lt "$i" ]; do + eval to_remove=\"\$remove_$j\" + _techo "Removing ${to_remove} ..." + pcmd rm ${VVERBOSE} -rf "${ddir}/${to_remove}" || \ + _exit_err "Removing ${to_remove} failed." + j=$(($j+1)) + done + fi + + + # + # Check for backup directory to clone from: Always clone from the latest one! + # + # Depending on your file system, you may want to sort on: + # 1. mtime (modification time) with TSORT=t, or + # 2. ctime (last change time, usually) with TSORT=tc + last_dir="$(pcmd ls -${TSORT}p1 "${ddir}" | grep '/$' | head -n 1)" || \ + _exit_err "Failed to list contents of ${ddir}." + + # + # clone from old backup, if existing + # + if [ "${last_dir}" ]; then + set -- "$@" "--link-dest=${ddir}/${last_dir}" + _techo "Hard linking from ${last_dir}" + fi + + + # set time when we really begin to backup, not when we began to remove above + destination_date=$(${CDATE}) + destination_dir="${ddir}/${INTERVAL}.${destination_date}.$$" + destination_full="${destination}/${INTERVAL}.${destination_date}.$$" + + # give some info + _techo "Beginning to backup, this may take some time..." + + _techo "Creating ${destination_dir} ..." + pcmd mkdir ${VVERBOSE} "${destination_dir}" || \ + _exit_err "Creating ${destination_dir} failed. Skipping." + + # + # added marking in 0.6 (and remove it, if successful later) + # + pcmd touch "${destination_dir}.${c_marker}" + + # + # the rsync part + # + _techo "Transferring files..." + rsync "$@" "${source}" "${destination_full}"; ret=$? + # Correct the modification time: + pcmd touch "${destination_dir}" + + # + # remove marking here + # + if [ "$ret" -ne 12 ] ; then + pcmd rm "${destination_dir}.${c_marker}" || \ + _exit_err "Removing ${destination_dir}/${c_marker} failed." + fi + + _techo "Finished backup (rsync return code: $ret)." + if [ "${ret}" -ne 0 ]; then + _techo "Warning: rsync exited non-zero, the backup may be broken (see rsync errors)." + fi + + # + # post_exec + # + if [ -x "${c_post_exec}" ]; then + _techo "Executing ${c_post_exec} ..." + "${c_post_exec}"; ret=$? + _techo "Finished ${c_post_exec}." + + if [ ${ret} -ne 0 ]; then + _exit_err "${c_post_exec} failed." + fi + fi + + # Calculation + end_s=$(date +%s) + + full_seconds=$((${end_s} - ${begin_s})) + hours=$((${full_seconds} / 3600)) + seconds=$((${full_seconds} - (${hours} * 3600))) + minutes=$((${seconds} / 60)) + seconds=$((${seconds} - (${minutes} * 60))) + + _techo "Backup lasted: ${hours}:${minutes}:${seconds} (h:m:s)" + +) | add_name +done + +# +# Be a good parent and wait for our children, if they are running wild parallel +# +if [ "${PARALLEL}" ]; then + _techo "Waiting for children to complete..." + wait +fi + +# +# Look for post-exec command (general) +# +if [ -x "${CPOSTEXEC}" ]; then + _techo "Executing ${CPOSTEXEC} ..." + "${CPOSTEXEC}"; ret=$? + _techo "Finished ${CPOSTEXEC} (return code: ${ret})." + + if [ ${ret} -ne 0 ]; then + _techo "${CPOSTEXEC} failed." + fi +fi + +rm -f "${TMP}" +_techo "Finished ${WE}" + +# vim: set shiftwidth=3 tabstop=3 expandtab : diff --git a/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/g.patch b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/g.patch new file mode 100644 index 00000000..0c9a73eb --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/g.patch @@ -0,0 +1,74 @@ +# I found that ccollect was not deleting incomplete backups despite the +# delete_incomplete option being specified. I traced the problem to: +# +# < pcmd rm $VVERBOSE -rf "${ddir}/${realincomplete}" || \ +# +# which, at least on all the systems I tested, should read: +# +# > pcmd rm $VVERBOSE -rf "${realincomplete}" || \ +# +# Also, the marker file is not deleted. I didn't see any reason to keep +# those files around (what do you think?), so I deleted them also: +# +# > pcmd rm $VVERBOSE -rf "${ddir}/${realincomplete}" || \ +# > _exit_err "Removing ${realincomplete} failed." +# +# As long as I was messing with the delete incomplete code and therefore need +# to test it, I took the liberty of simplifying it. The v0.7.1 code uses +# multiple loops with multiple loop counters and creates many variables. I +# simplified that to a single loop: +# +# > pcmd ls -1 "$ddir/${INTERVAL}"*".${c_marker}" 2>/dev/null | while read marker; do +# > incomplete="$(echo ${marker} | sed "s/\\.${c_marker}\$//")" +# > _techo "Incomplete backup: ${incomplete}" +# > if [ "${DELETE_INCOMPLETE}" = "yes" ]; then +# > _techo "Deleting ${incomplete} ..." +# > pcmd rm $VVERBOSE -rf "${incomplete}" || \ +# > _exit_err "Removing ${incomplete} failed." +# > pcmd rm $VVERBOSE -f "${marker}" || \ +# > _exit_err "Removing ${marker} failed." +# > fi +# > done +# +# The final code (a) fixes the delete bug, (b) also deletes the marker, and +# (c) is eight lines shorter than the original. +# +--- ccollect-f.sh 2009-05-12 12:49:28.000000000 -0700 ++++ ccollect-g.sh 2009-06-03 14:32:03.000000000 -0700 +@@ -516,28 +516,20 @@ + fi + + # + # Check for incomplete backups + # +- pcmd ls -1 "$ddir/${INTERVAL}"*".${c_marker}" > "${TMP}" 2>/dev/null +- +- i=0 +- while read incomplete; do +- eval incomplete_$i=\"$(echo ${incomplete} | sed "s/\\.${c_marker}\$//")\" +- i=$(($i+1)) +- done < "${TMP}" +- +- j=0 +- while [ "$j" -lt "$i" ]; do +- eval realincomplete=\"\$incomplete_$j\" +- _techo "Incomplete backup: ${realincomplete}" ++ pcmd ls -1 "$ddir/${INTERVAL}"*".${c_marker}" 2>/dev/null | while read marker; do ++ incomplete="$(echo ${marker} | sed "s/\\.${c_marker}\$//")" ++ _techo "Incomplete backup: ${incomplete}" + if [ "${DELETE_INCOMPLETE}" = "yes" ]; then +- _techo "Deleting ${realincomplete} ..." +- pcmd rm $VVERBOSE -rf "${ddir}/${realincomplete}" || \ +- _exit_err "Removing ${realincomplete} failed." ++ _techo "Deleting ${incomplete} ..." ++ pcmd rm $VVERBOSE -rf "${incomplete}" || \ ++ _exit_err "Removing ${incomplete} failed." ++ pcmd rm $VVERBOSE -f "${marker}" || \ ++ _exit_err "Removing ${marker} failed." + fi +- j=$(($j+1)) + done + + # + # check if maximum number of backups is reached, if so remove + # use grep and ls -p so we only look at directories diff --git a/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/h.patch b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/h.patch new file mode 100644 index 00000000..b850b734 --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/h.patch @@ -0,0 +1,18 @@ +# A line in my f.patch was missing needed quotation marks. +# This fixes that. +# +--- ccollect-g.sh 2009-06-03 14:32:03.000000000 -0700 ++++ ccollect-h.sh 2009-06-03 14:32:19.000000000 -0700 +@@ -412,11 +412,11 @@ + fi + + # + # interval definition: First try source specific, fallback to default + # +- if [ ${INTERVAL} = "AUTO" ] ; then ++ if [ "${INTERVAL}" = "AUTO" ] ; then + auto_interval + _techo "Selected interval: '$INTERVAL'" + fi + c_interval="$(cat "${backup}/intervals/${INTERVAL}" 2>/dev/null)" + diff --git a/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/i.patch b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/i.patch new file mode 100644 index 00000000..e8edbafb --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/i.patch @@ -0,0 +1,134 @@ +# I have many sources that use the same options so I put those +# options in the defaults directory. I found that ccollect was +# ignoring most of them. I thought that this was a bug so I wrote +# some code to correct this: +# +# > for opt in exclude verbose very_verbose rsync_options summary delete_incomplete remote_host ; do +# > if [ -f "${backup}/$opt" -o -f "${backup}/no_$opt" ]; then +# > eval c_$opt=\"${backup}/$opt\" +# > else +# > eval c_$opt=\"${CDEFAULTS}/$opt\" +# > fi +# > done +# +# This also adds a new feature: if some option, say verbose, is +# specified in the defaults directory, it can be turned off for +# particular sources by specifying no_verbose as a source option. +# +# A side effect of this approach is that it forces script variable +# names to be consistent with option file names. Thus, there are +# several changes such as: +# +# < if [ -f "${c_rsync_extra}" ]; then +# > if [ -f "${c_rsync_options}" ]; then +# +# and +# +# < if [ -f "${c_vverbose}" ]; then +# > if [ -f "${c_very_verbose}" ]; then +# +# After correcting the bug and adding the "no_" feature, the code is +# 12 lines shorter. +# +--- ccollect-h.sh 2009-06-01 15:59:11.000000000 -0700 ++++ ccollect-i.sh 2009-06-03 14:27:58.000000000 -0700 +@@ -336,20 +336,19 @@ + # Configuration + # + backup="${CSOURCES}/${name}" + c_source="${backup}/source" + c_dest="${backup}/destination" +- c_exclude="${backup}/exclude" +- c_verbose="${backup}/verbose" +- c_vverbose="${backup}/very_verbose" +- c_rsync_extra="${backup}/rsync_options" +- c_summary="${backup}/summary" + c_pre_exec="${backup}/pre_exec" + c_post_exec="${backup}/post_exec" +- f_incomplete="delete_incomplete" +- c_incomplete="${backup}/${f_incomplete}" +- c_remote_host="${backup}/remote_host" ++ for opt in exclude verbose very_verbose rsync_options summary delete_incomplete remote_host ; do ++ if [ -f "${backup}/$opt" -o -f "${backup}/no_$opt" ]; then ++ eval c_$opt=\"${backup}/$opt\" ++ else ++ eval c_$opt=\"${CDEFAULTS}/$opt\" ++ fi ++ done + + # + # Marking backups: If we abort it's not removed => Backup is broken + # + c_marker=".ccollect-marker" +@@ -360,16 +359,12 @@ + begin_s=$(date +%s) + + # + # unset possible options + # +- EXCLUDE="" +- RSYNC_EXTRA="" +- SUMMARY="" + VERBOSE="" + VVERBOSE="" +- DELETE_INCOMPLETE="" + + _techo "Beginning to backup" + + # + # Standard configuration checks +@@ -462,17 +457,10 @@ + # check for existence / use real name + # + ( pcmd cd "$ddir" ) || _exit_err "Cannot change to ${ddir}. Skipping." + + +- # +- # Check whether to delete incomplete backups +- # +- if [ -f "${c_incomplete}" -o -f "${CDEFAULTS}/${f_incomplete}" ]; then +- DELETE_INCOMPLETE="yes" +- fi +- + # NEW method as of 0.6: + # - insert ccollect default parameters + # - insert options + # - insert user options + +@@ -498,32 +486,32 @@ + fi + + # + # Verbosity for rsync + # +- if [ -f "${c_vverbose}" ]; then ++ if [ -f "${c_very_verbose}" ]; then + set -- "$@" "-vv" + elif [ -f "${c_verbose}" ]; then + set -- "$@" "-v" + fi + + # + # extra options for rsync provided by the user + # +- if [ -f "${c_rsync_extra}" ]; then ++ if [ -f "${c_rsync_options}" ]; then + while read line; do + set -- "$@" "$line" +- done < "${c_rsync_extra}" ++ done < "${c_rsync_options}" + fi + + # + # Check for incomplete backups + # + pcmd ls -1 "$ddir/${INTERVAL}"*".${c_marker}" 2>/dev/null | while read marker; do + incomplete="$(echo ${marker} | sed "s/\\.${c_marker}\$//")" + _techo "Incomplete backup: ${incomplete}" +- if [ "${DELETE_INCOMPLETE}" = "yes" ]; then ++ if [ -f "${c_delete_incomplete}" ]; then + _techo "Deleting ${incomplete} ..." + pcmd rm $VVERBOSE -rf "${incomplete}" || \ + _exit_err "Removing ${incomplete} failed." + pcmd rm $VVERBOSE -f "${marker}" || \ + _exit_err "Removing ${marker} failed." diff --git a/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/README_a-f.txt b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/README_a-f.txt new file mode 100644 index 00000000..e3bfe575 --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/README_a-f.txt @@ -0,0 +1,296 @@ +Dear Nico Schottelius, + + I have started using ccollect and I very much like its design: +it is elegant and effective. + + In the process of getting ccollect setup and running, I made +five changes, including one major new feature, that I hope you will +find useful. + + First, I added the following before any old backup gets deleted: + +> # Verify source is up and accepting connections before deleting any old backups +> rsync "$source" >/dev/null || _exit_err "Source ${source} is not readable. Skipping." + +I think that this quick test is a much better than, say, pinging +the source in a pre-exec script: this tests not only that the +source is up and connected to the net, it also verifies (1) that +ssh is up and accepting our key (if we are using ssh), and (2) that +the source directory is mounted (if it needs to be mounted) and +readable. + + Second, I found ccollect's use of ctime problematic. After +copying an old backup over to my ccollect destination, I adjusted +mtime and atime where needed using touch, e.g.: + +touch -d"28 Apr 2009 3:00" destination/daily.01 + +However, as far as I know, there is no way to correct a bad ctime. +I ran into this issue repeatedly while adjusting my backup +configuration. (For example, "cp -a" preserves mtime but not +ctime. Even worse, "cp -al old new" also changes ctime on old.) + + Another potential problem with ctime is that it is file-system +dependent: I have read that Windows sets ctime to create-time not +last change-time. + + However, It is simple to give a new backup the correct mtime. +After the rsync step, I added the command: + +553a616,617 +> # Correct the modification time: +> pcmd touch "${destination_dir}" + +Even if ccollect continues to use ctime for sorting, I see no +reason not to have the backup directory have the correct mtime. + + To allow the rest of the code to use either ctime or mtime, I +added definitions: + +44a45,47 +> #TSORT="tc" ; NEWER="cnewer" +> TSORT="t" ; NEWER="newer" + +(It would be better if this choice was user-configurable because +those with existing backup directories should continue to use ctime +until the mtimes of their directories are correct. The correction +would happen passively over time as new backups created using the +above touch command and the old ones are deleted.) + +With these definitions, the proper link-dest directory can then be +found using this minor change (and comment update): + +516,519c579,582 +< # Use ls -1c instead of -1t, because last modification maybe the same on all +< # and metadate update (-c) is updated by rsync locally. +< # +< last_dir="$(pcmd ls -tcp1 "${ddir}" | grep '/$' | head -n 1)" || \ +--- +> # Depending on your file system, you may want to sort on: +> # 1. mtime (modification time) with TSORT=t, or +> # 2. ctime (last change time, usually) with TSORT=tc +> last_dir="$(pcmd ls -${TSORT}p1 "${ddir}" | grep '/$' | head -n 1)" || \ + + Thirdly, after I copied my old backups over to my ccollect +destination directory, I found that ccollect would delete a +recent backup not an old backup! My problem was that, unknown to +me, the algorithm to find the oldest backup (for deletion) was +inconsistent with that used to find the newest (for link-dest). I +suggest that these two should be consistent. Because time-sorting +seemed more consistent with the ccollect documentation, I suggest: + +492,493c555,556 +< pcmd ls -p1 "$ddir" | grep "^${INTERVAL}\..*/\$" | \ +< sort -n | head -n "${remove}" > "${TMP}" || \ +--- +> pcmd ls -${TSORT}p1r "$ddir" | grep "^${INTERVAL}\..*/\$" | \ +> head -n "${remove}" > "${TMP}" || \ + + Fourthly, in my experience, rsync error code 12 means complete +failure, usually because the source refuses the ssh connection. +So, I left the marker in that case: + +558,559c622,625 +< pcmd rm "${destination_dir}.${c_marker}" || \ +< _exit_err "Removing ${destination_dir}/${c_marker} failed." +--- +> if [ "$ret" -ne 12 ] ; then +> pcmd rm "${destination_dir}.${c_marker}" || \ +> _exit_err "Removing ${destination_dir}/${c_marker} failed." +> fi + +(A better solution might allow a user-configurable list of error +codes that are treated the same as a fail.) + + Fifth, because I was frustrated by the problems of having a +cron-job decide which interval to backup, I added a major new +feature: the modified ccollect can now automatically select an +interval to use for backup. + + Cron-job controlled backup works well if all machines are up and +running all the time and nothing ever goes wrong. I have, however, +some machines that are occasionally turned off, or that are mobile +and only sometimes connected to local net. For these machines, the +use of cron-jobs to select intervals can be a disaster. + + There are several ways one could automatically choose an +appropriate interval. The method I show below has the advantage +that it works with existing ccollect configuration files. The only +requirement is that interval names be chosen to sort nicely (under +ls). For example, I currently use: + +$ ls -1 intervals +a_daily +b_weekly +c_monthly +d_quarterly +e_yearly +$ cat intervals/* +6 +3 +2 +3 +30 + +A simpler example would be: + +$ ls -1 intervals +int1 +int2 +int3 +$ cat intervals/* +2 +3 +4 + +The algorithm works as follows: + + If no backup exists for the least frequent interval (int3 in the + simpler example), then use that interval. Otherwise, use the + most frequent interval (int1) unless there are "$(cat + intervals/int1)" int1 backups more recent than any int2 or int3 + backup, in which case select int2 unless there are "$(cat + intervals/int2)" int2 backups more recent than any int3 backups + in which case choose int3. + +This algorithm works well cycling through all the backups for my +always connected machines as well as for my usually connected +machines, and rarely connected machines. (For a rarely connected +machine, interval names like "b_weekly" lose their English meaning +but it still does a reasonable job of rotating through the +intervals.) + + In addition to being more robust, the automatic interval +selection means that crontab is greatly simplified: only one line +is needed. I use: + +30 3 * * * ccollect.sh AUTO host1 host2 host3 | tee -a /var/log/ccollect-full.log | ccollect_analyse_logs.sh iwe + + Some users might prefer a calendar-driven algorithm such as: do +a yearly backup the first time a machine is connected during a new +year; do a monthly backup the first that a machine is connected +during a month; etc. This, however, would require a change to the +ccollect configuration files. So, I didn't pursue the idea any +further. + + The code checks to see if the user specified the interval as +AUTO. If so, the auto_interval function is called to select the +interval: + +347a417,420 +> if [ ${INTERVAL} = "AUTO" ] ; then +> auto_interval +> _techo "Selected interval: '$INTERVAL'" +> fi + +The code for auto_interval is as follows (note that it allows 'more +recent' to be defined by either ctime or mtime as per the TSORT +variable): + +125a129,182 +> # Select interval if AUTO +> # +> # For this to work nicely, you have to choose interval names that sort nicely +> # such as int1, int2, int3 or a_daily, b_weekly, c_monthly, etc. +> # +> auto_interval() +> { +> if [ -d "${backup}/intervals" -a -n "$(ls "${backup}/intervals" 2>/dev/null)" ] ; then +> intervals_dir="${backup}/intervals" +> elif [ -d "${CDEFAULTS}/intervals" -a -n "$(ls "${CDEFAULTS}/intervals" 2>/dev/null)" ] ; then +> intervals_dir="${CDEFAULTS}/intervals" +> else +> _exit_err "No intervals are defined. Skipping." +> fi +> echo intervals_dir=${intervals_dir} +> +> trial_interval="$(ls -1r "${intervals_dir}/" | head -n 1)" || \ +> _exit_err "Failed to list contents of ${intervals_dir}/." +> _techo "Considering interval ${trial_interval}" +> most_recent="$(pcmd ls -${TSORT}p1 "${ddir}" | grep "^${trial_interval}.*/$" | head -n 1)" || \ +> _exit_err "Failed to list contents of ${ddir}/." +> _techo " Most recent ${trial_interval}: '${most_recent}'" +> if [ -n "${most_recent}" ]; then +> no_intervals="$(ls -1 "${intervals_dir}/" | wc -l)" +> n=1 +> while [ "${n}" -le "${no_intervals}" ]; do +> trial_interval="$(ls -p1 "${intervals_dir}/" | tail -n+${n} | head -n 1)" +> _techo "Considering interval '${trial_interval}'" +> c_interval="$(cat "${intervals_dir}/${trial_interval}" 2>/dev/null)" +> m=$((${n}+1)) +> set -- "${ddir}" -maxdepth 1 +> while [ "${m}" -le "${no_intervals}" ]; do +> interval_m="$(ls -1 "${intervals_dir}/" | tail -n+${m} | head -n 1)" +> most_recent="$(pcmd ls -${TSORT}p1 "${ddir}" | grep "^${interval_m}\..*/$" | head -n 1)" +> _techo " Most recent ${interval_m}: '${most_recent}'" +> if [ -n "${most_recent}" ] ; then +> set -- "$@" -$NEWER "${ddir}/${most_recent}" +> fi +> m=$((${m}+1)) +> done +> count=$(pcmd find "$@" -iname "${trial_interval}*" | wc -l) +> _techo " Found $count more recent backups of ${trial_interval} (limit: ${c_interval})" +> if [ "$count" -lt "${c_interval}" ] ; then +> break +> fi +> n=$((${n}+1)) +> done +> fi +> export INTERVAL="${trial_interval}" +> D_FILE_INTERVAL="${intervals_dir}/${INTERVAL}" +> D_INTERVAL=$(cat "${D_FILE_INTERVAL}" 2>/dev/null) +> } +> +> # + +While I consider the auto_interval code to be developmental, I have +been using it for my nightly backups and it works for me. + + One last change: For auto_interval to work, it needs "ddir" to +be defined first. Consequently, I had to move the following code +so it gets run before auto_interval is called: + +369,380c442,443 +< +< # +< # Destination is a path +< # +< if [ ! -f "${c_dest}" ]; then +< _exit_err "Destination ${c_dest} is not a file. Skipping." +< else +< ddir=$(cat "${c_dest}"); ret="$?" +< if [ "${ret}" -ne 0 ]; then +< _exit_err "Destination ${c_dest} is not readable. Skipping." +< fi +< fi +345a403,414 +> # Destination is a path +> # +> if [ ! -f "${c_dest}" ]; then +> _exit_err "Destination ${c_dest} is not a file. Skipping." +> else +> ddir=$(cat "${c_dest}"); ret="$?" +> if [ "${ret}" -ne 0 ]; then +> _exit_err "Destination ${c_dest} is not readable. Skipping." +> fi +> fi +> +> # + + I have some other ideas but this is all I have implemented at +the moment. Files are attached. + + Thanks again for developing ccollect and let me know what you +think. + +Regards, + +John + +-- + John L. Lawless, Ph.D. + Redwood Scientific, Inc. + 1005 Terra Nova Blvd + Pacifica, CA 94044-4300 + 1-650-738-8083 + diff --git a/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/a.patch b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/a.patch new file mode 100644 index 00000000..bf4b6625 --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/a.patch @@ -0,0 +1,15 @@ +--- ccollect-0.7.1.sh 2009-02-02 03:39:42.000000000 -0800 ++++ ccollect-0.7.1-a.sh 2009-05-24 21:30:38.000000000 -0700 +@@ -364,10 +364,12 @@ + source=$(cat "${c_source}"); ret="$?" + if [ "${ret}" -ne 0 ]; then + _exit_err "Source ${c_source} is not readable. Skipping." + fi + fi ++ # Verify source is up and accepting connections before deleting any old backups ++ rsync "$source" >/dev/null || _exit_err "Source ${source} is not readable. Skipping." + + # + # Destination is a path + # + if [ ! -f "${c_dest}" ]; then diff --git a/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/b.patch b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/b.patch new file mode 100644 index 00000000..c0266d2d --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/b.patch @@ -0,0 +1,15 @@ +--- ccollect-0.7.1-a.sh 2009-05-24 21:30:38.000000000 -0700 ++++ ccollect-0.7.1-b.sh 2009-05-24 21:32:00.000000000 -0700 +@@ -551,10 +551,12 @@ + # the rsync part + # + + _techo "Transferring files..." + rsync "$@" "${source}" "${destination_full}"; ret=$? ++ # Correct the modification time: ++ pcmd touch "${destination_dir}" + + # + # remove marking here + # + pcmd rm "${destination_dir}.${c_marker}" || \ diff --git a/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/c.patch b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/c.patch new file mode 100644 index 00000000..7b5f9a8e --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/c.patch @@ -0,0 +1,35 @@ +--- ccollect-0.7.1-b.sh 2009-05-24 21:32:00.000000000 -0700 ++++ ccollect-0.7.1-c.sh 2009-05-24 21:39:43.000000000 -0700 +@@ -40,10 +40,13 @@ + VERSION=0.7.1 + RELEASE="2009-02-02" + HALF_VERSION="ccollect ${VERSION}" + FULL_VERSION="ccollect ${VERSION} (${RELEASE})" + ++#TSORT="tc" ; NEWER="cnewer" ++TSORT="t" ; NEWER="newer" ++ + # + # CDATE: how we use it for naming of the archives + # DDATE: how the user should see it in our output (DISPLAY) + # + CDATE="date +%Y%m%d-%H%M" +@@ -513,14 +516,14 @@ + + + # + # Check for backup directory to clone from: Always clone from the latest one! + # +- # Use ls -1c instead of -1t, because last modification maybe the same on all +- # and metadate update (-c) is updated by rsync locally. +- # +- last_dir="$(pcmd ls -tcp1 "${ddir}" | grep '/$' | head -n 1)" || \ ++ # Depending on your file system, you may want to sort on: ++ # 1. mtime (modification time) with TSORT=t, or ++ # 2. ctime (last change time, usually) with TSORT=tc ++ last_dir="$(pcmd ls -${TSORT}p1 "${ddir}" | grep '/$' | head -n 1)" || \ + _exit_err "Failed to list contents of ${ddir}." + + # + # clone from old backup, if existing + # diff --git a/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/ccollect-0.7.1.sh b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/ccollect-0.7.1.sh new file mode 100644 index 00000000..e14dcfca --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/ccollect-0.7.1.sh @@ -0,0 +1,615 @@ +#!/bin/sh +# +# 2005-2009 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Initially written for SyGroup (www.sygroup.ch) +# Date: Mon Nov 14 11:45:11 CET 2005 + +# +# Standard variables (stolen from cconf) +# +__pwd="$(pwd -P)" +__mydir="${0%/*}"; __abs_mydir="$(cd "$__mydir" && pwd -P)" +__myname=${0##*/}; __abs_myname="$__abs_mydir/$__myname" + +# +# where to find our configuration and temporary file +# +CCOLLECT_CONF=${CCOLLECT_CONF:-/etc/ccollect} +CSOURCES=${CCOLLECT_CONF}/sources +CDEFAULTS=${CCOLLECT_CONF}/defaults +CPREEXEC="${CDEFAULTS}/pre_exec" +CPOSTEXEC="${CDEFAULTS}/post_exec" + +TMP=$(mktemp "/tmp/${__myname}.XXXXXX") +VERSION=0.7.1 +RELEASE="2009-02-02" +HALF_VERSION="ccollect ${VERSION}" +FULL_VERSION="ccollect ${VERSION} (${RELEASE})" + +# +# CDATE: how we use it for naming of the archives +# DDATE: how the user should see it in our output (DISPLAY) +# +CDATE="date +%Y%m%d-%H%M" +DDATE="date +%Y-%m-%d-%H:%M:%S" + +# +# unset parallel execution +# +PARALLEL="" + +# +# catch signals +# +trap "rm -f \"${TMP}\"" 1 2 15 + +# +# Functions +# + +# time displaying echo +_techo() +{ + echo "$(${DDATE}): $@" +} + +# exit on error +_exit_err() +{ + _techo "$@" + rm -f "${TMP}" + exit 1 +} + +add_name() +{ + awk "{ print \"[${name}] \" \$0 }" +} + +pcmd() +{ + if [ "$remote_host" ]; then + ssh "$remote_host" "$@" + else + "$@" + fi +} + +# +# Version +# +display_version() +{ + echo "${FULL_VERSION}" + exit 0 +} + +# +# Tell how to use us +# +usage() +{ + echo "${__myname}: [args] " + echo "" + echo " ccollect creates (pseudo) incremental backups" + echo "" + echo " -h, --help: Show this help screen" + echo " -p, --parallel: Parallelise backup processes" + echo " -a, --all: Backup all sources specified in ${CSOURCES}" + echo " -v, --verbose: Be very verbose (uses set -x)" + echo " -V, --version: Print version information" + echo "" + echo " This is version ${VERSION}, released on ${RELEASE}" + echo " (the first version was written on 2005-12-05 by Nico Schottelius)." + echo "" + echo " Retrieve latest ccollect at http://unix.schottelius.org/ccollect/" + exit 0 +} + +# +# need at least interval and one source or --all +# +if [ $# -lt 2 ]; then + if [ "$1" = "-V" -o "$1" = "--version" ]; then + display_version + else + usage + fi +fi + +# +# check for configuraton directory +# +[ -d "${CCOLLECT_CONF}" ] || _exit_err "No configuration found in " \ + "\"${CCOLLECT_CONF}\" (is \$CCOLLECT_CONF properly set?)" + +# +# Filter arguments +# +export INTERVAL="$1"; shift +i=1 +no_sources=0 + +# +# Create source "array" +# +while [ "$#" -ge 1 ]; do + eval arg=\"\$1\"; shift + + if [ "${NO_MORE_ARGS}" = 1 ]; then + eval source_${no_sources}=\"${arg}\" + no_sources=$((${no_sources}+1)) + + # make variable available for subscripts + eval export source_${no_sources} + else + case "${arg}" in + -a|--all) + ALL=1 + ;; + -v|--verbose) + VERBOSE=1 + ;; + -p|--parallel) + PARALLEL=1 + ;; + -h|--help) + usage + ;; + --) + NO_MORE_ARGS=1 + ;; + *) + eval source_${no_sources}=\"$arg\" + no_sources=$(($no_sources+1)) + ;; + esac + fi + + i=$(($i+1)) +done + +# also export number of sources +export no_sources + +# +# be really, really, really verbose +# +if [ "${VERBOSE}" = 1 ]; then + set -x +fi + +# +# Look, if we should take ALL sources +# +if [ "${ALL}" = 1 ]; then + # reset everything specified before + no_sources=0 + + # + # get entries from sources + # + cwd=$(pwd -P) + ( cd "${CSOURCES}" && ls > "${TMP}" ); ret=$? + + [ "${ret}" -eq 0 ] || _exit_err "Listing of sources failed. Aborting." + + while read tmp; do + eval source_${no_sources}=\"${tmp}\" + no_sources=$((${no_sources}+1)) + done < "${TMP}" +fi + +# +# Need at least ONE source to backup +# +if [ "${no_sources}" -lt 1 ]; then + usage +else + _techo "${HALF_VERSION}: Beginning backup using interval ${INTERVAL}" +fi + +# +# Look for pre-exec command (general) +# +if [ -x "${CPREEXEC}" ]; then + _techo "Executing ${CPREEXEC} ..." + "${CPREEXEC}"; ret=$? + _techo "Finished ${CPREEXEC} (return code: ${ret})." + + [ "${ret}" -eq 0 ] || _exit_err "${CPREEXEC} failed. Aborting" +fi + +# +# check default configuration +# + +D_FILE_INTERVAL="${CDEFAULTS}/intervals/${INTERVAL}" +D_INTERVAL=$(cat "${D_FILE_INTERVAL}" 2>/dev/null) + + +# +# Let's do the backup +# +i=0 +while [ "${i}" -lt "${no_sources}" ]; do + + # + # Get current source + # + eval name=\"\$source_${i}\" + i=$((${i}+1)) + + export name + + # + # start ourself, if we want parallel execution + # + if [ "${PARALLEL}" ]; then + "$0" "${INTERVAL}" "${name}" & + continue + fi + +# +# Start subshell for easy log editing +# +( + # + # Stderr to stdout, so we can produce nice logs + # + exec 2>&1 + + # + # Configuration + # + backup="${CSOURCES}/${name}" + c_source="${backup}/source" + c_dest="${backup}/destination" + c_exclude="${backup}/exclude" + c_verbose="${backup}/verbose" + c_vverbose="${backup}/very_verbose" + c_rsync_extra="${backup}/rsync_options" + c_summary="${backup}/summary" + c_pre_exec="${backup}/pre_exec" + c_post_exec="${backup}/post_exec" + f_incomplete="delete_incomplete" + c_incomplete="${backup}/${f_incomplete}" + c_remote_host="${backup}/remote_host" + + # + # Marking backups: If we abort it's not removed => Backup is broken + # + c_marker=".ccollect-marker" + + # + # Times + # + begin_s=$(date +%s) + + # + # unset possible options + # + EXCLUDE="" + RSYNC_EXTRA="" + SUMMARY="" + VERBOSE="" + VVERBOSE="" + DELETE_INCOMPLETE="" + + _techo "Beginning to backup" + + # + # Standard configuration checks + # + if [ ! -e "${backup}" ]; then + _exit_err "Source does not exist." + fi + + # + # configuration _must_ be a directory + # + if [ ! -d "${backup}" ]; then + _exit_err "\"${name}\" is not a cconfig-directory. Skipping." + fi + + # + # first execute pre_exec, which may generate destination or other + # parameters + # + if [ -x "${c_pre_exec}" ]; then + _techo "Executing ${c_pre_exec} ..." + "${c_pre_exec}"; ret="$?" + _techo "Finished ${c_pre_exec} (return code ${ret})." + + if [ "${ret}" -ne 0 ]; then + _exit_err "${c_pre_exec} failed. Skipping." + fi + fi + + # + # interval definition: First try source specific, fallback to default + # + c_interval="$(cat "${backup}/intervals/${INTERVAL}" 2>/dev/null)" + + if [ -z "${c_interval}" ]; then + c_interval="${D_INTERVAL}" + + if [ -z "${c_interval}" ]; then + _exit_err "No definition for interval \"${INTERVAL}\" found. Skipping." + fi + fi + + # + # Source checks + # + if [ ! -f "${c_source}" ]; then + _exit_err "Source description \"${c_source}\" is not a file. Skipping." + else + source=$(cat "${c_source}"); ret="$?" + if [ "${ret}" -ne 0 ]; then + _exit_err "Source ${c_source} is not readable. Skipping." + fi + fi + + # + # Destination is a path + # + if [ ! -f "${c_dest}" ]; then + _exit_err "Destination ${c_dest} is not a file. Skipping." + else + ddir=$(cat "${c_dest}"); ret="$?" + if [ "${ret}" -ne 0 ]; then + _exit_err "Destination ${c_dest} is not readable. Skipping." + fi + fi + + # + # do we backup to a remote host? then set pre-cmd + # + if [ -f "${c_remote_host}" ]; then + # adjust ls and co + remote_host=$(cat "${c_remote_host}"); ret="$?" + if [ "${ret}" -ne 0 ]; then + _exit_err "Remote host file ${c_remote_host} exists, but is not readable. Skipping." + fi + destination="${remote_host}:${ddir}" + else + remote_host="" + destination="${ddir}" + fi + export remote_host + + # + # check for existence / use real name + # + ( pcmd cd "$ddir" ) || _exit_err "Cannot change to ${ddir}. Skipping." + + + # + # Check whether to delete incomplete backups + # + if [ -f "${c_incomplete}" -o -f "${CDEFAULTS}/${f_incomplete}" ]; then + DELETE_INCOMPLETE="yes" + fi + + # NEW method as of 0.6: + # - insert ccollect default parameters + # - insert options + # - insert user options + + # + # rsync standard options + # + + set -- "$@" "--archive" "--delete" "--numeric-ids" "--relative" \ + "--delete-excluded" "--sparse" + + # + # exclude list + # + if [ -f "${c_exclude}" ]; then + set -- "$@" "--exclude-from=${c_exclude}" + fi + + # + # Output a summary + # + if [ -f "${c_summary}" ]; then + set -- "$@" "--stats" + fi + + # + # Verbosity for rsync + # + if [ -f "${c_vverbose}" ]; then + set -- "$@" "-vv" + elif [ -f "${c_verbose}" ]; then + set -- "$@" "-v" + fi + + # + # extra options for rsync provided by the user + # + if [ -f "${c_rsync_extra}" ]; then + while read line; do + set -- "$@" "$line" + done < "${c_rsync_extra}" + fi + + # + # Check for incomplete backups + # + pcmd ls -1 "$ddir/${INTERVAL}"*".${c_marker}" > "${TMP}" 2>/dev/null + + i=0 + while read incomplete; do + eval incomplete_$i=\"$(echo ${incomplete} | sed "s/\\.${c_marker}\$//")\" + i=$(($i+1)) + done < "${TMP}" + + j=0 + while [ "$j" -lt "$i" ]; do + eval realincomplete=\"\$incomplete_$j\" + _techo "Incomplete backup: ${realincomplete}" + if [ "${DELETE_INCOMPLETE}" = "yes" ]; then + _techo "Deleting ${realincomplete} ..." + pcmd rm $VVERBOSE -rf "${ddir}/${realincomplete}" || \ + _exit_err "Removing ${realincomplete} failed." + fi + j=$(($j+1)) + done + + # + # check if maximum number of backups is reached, if so remove + # use grep and ls -p so we only look at directories + # + count="$(pcmd ls -p1 "${ddir}" | grep "^${INTERVAL}\..*/\$" | wc -l \ + | sed 's/^ *//g')" || _exit_err "Counting backups failed" + + _techo "Existing backups: ${count} Total keeping backups: ${c_interval}" + + if [ "${count}" -ge "${c_interval}" ]; then + substract=$((${c_interval} - 1)) + remove=$((${count} - ${substract})) + _techo "Removing ${remove} backup(s)..." + + pcmd ls -p1 "$ddir" | grep "^${INTERVAL}\..*/\$" | \ + sort -n | head -n "${remove}" > "${TMP}" || \ + _exit_err "Listing old backups failed" + + i=0 + while read to_remove; do + eval remove_$i=\"${to_remove}\" + i=$(($i+1)) + done < "${TMP}" + + j=0 + while [ "$j" -lt "$i" ]; do + eval to_remove=\"\$remove_$j\" + _techo "Removing ${to_remove} ..." + pcmd rm ${VVERBOSE} -rf "${ddir}/${to_remove}" || \ + _exit_err "Removing ${to_remove} failed." + j=$(($j+1)) + done + fi + + + # + # Check for backup directory to clone from: Always clone from the latest one! + # + # Use ls -1c instead of -1t, because last modification maybe the same on all + # and metadate update (-c) is updated by rsync locally. + # + last_dir="$(pcmd ls -tcp1 "${ddir}" | grep '/$' | head -n 1)" || \ + _exit_err "Failed to list contents of ${ddir}." + + # + # clone from old backup, if existing + # + if [ "${last_dir}" ]; then + set -- "$@" "--link-dest=${ddir}/${last_dir}" + _techo "Hard linking from ${last_dir}" + fi + + + # set time when we really begin to backup, not when we began to remove above + destination_date=$(${CDATE}) + destination_dir="${ddir}/${INTERVAL}.${destination_date}.$$" + destination_full="${destination}/${INTERVAL}.${destination_date}.$$" + + # give some info + _techo "Beginning to backup, this may take some time..." + + _techo "Creating ${destination_dir} ..." + pcmd mkdir ${VVERBOSE} "${destination_dir}" || \ + _exit_err "Creating ${destination_dir} failed. Skipping." + + # + # added marking in 0.6 (and remove it, if successful later) + # + pcmd touch "${destination_dir}.${c_marker}" + + # + # the rsync part + # + + _techo "Transferring files..." + rsync "$@" "${source}" "${destination_full}"; ret=$? + + # + # remove marking here + # + pcmd rm "${destination_dir}.${c_marker}" || \ + _exit_err "Removing ${destination_dir}/${c_marker} failed." + + _techo "Finished backup (rsync return code: $ret)." + if [ "${ret}" -ne 0 ]; then + _techo "Warning: rsync exited non-zero, the backup may be broken (see rsync errors)." + fi + + # + # post_exec + # + if [ -x "${c_post_exec}" ]; then + _techo "Executing ${c_post_exec} ..." + "${c_post_exec}"; ret=$? + _techo "Finished ${c_post_exec}." + + if [ ${ret} -ne 0 ]; then + _exit_err "${c_post_exec} failed." + fi + fi + + # Calculation + end_s=$(date +%s) + + full_seconds=$((${end_s} - ${begin_s})) + hours=$((${full_seconds} / 3600)) + seconds=$((${full_seconds} - (${hours} * 3600))) + minutes=$((${seconds} / 60)) + seconds=$((${seconds} - (${minutes} * 60))) + + _techo "Backup lasted: ${hours}:${minutes}:${seconds} (h:m:s)" + +) | add_name +done + +# +# Be a good parent and wait for our children, if they are running wild parallel +# +if [ "${PARALLEL}" ]; then + _techo "Waiting for children to complete..." + wait +fi + +# +# Look for post-exec command (general) +# +if [ -x "${CPOSTEXEC}" ]; then + _techo "Executing ${CPOSTEXEC} ..." + "${CPOSTEXEC}"; ret=$? + _techo "Finished ${CPOSTEXEC} (return code: ${ret})." + + if [ ${ret} -ne 0 ]; then + _techo "${CPOSTEXEC} failed." + fi +fi + +rm -f "${TMP}" +_techo "Finished ${WE}" diff --git a/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/ccollect-f.sh b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/ccollect-f.sh new file mode 100644 index 00000000..5c8952e8 --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/ccollect-f.sh @@ -0,0 +1,683 @@ +#!/bin/sh +# +# 2005-2009 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Initially written for SyGroup (www.sygroup.ch) +# Date: Mon Nov 14 11:45:11 CET 2005 + +# +# Standard variables (stolen from cconf) +# +__pwd="$(pwd -P)" +__mydir="${0%/*}"; __abs_mydir="$(cd "$__mydir" && pwd -P)" +__myname=${0##*/}; __abs_myname="$__abs_mydir/$__myname" + +# +# where to find our configuration and temporary file +# +CCOLLECT_CONF=${CCOLLECT_CONF:-/etc/ccollect} +CSOURCES=${CCOLLECT_CONF}/sources +CDEFAULTS=${CCOLLECT_CONF}/defaults +CPREEXEC="${CDEFAULTS}/pre_exec" +CPOSTEXEC="${CDEFAULTS}/post_exec" + +TMP=$(mktemp "/tmp/${__myname}.XXXXXX") +VERSION=0.7.1 +RELEASE="2009-02-02" +HALF_VERSION="ccollect ${VERSION}" +FULL_VERSION="ccollect ${VERSION} (${RELEASE})" + +#TSORT="tc" ; NEWER="cnewer" +TSORT="t" ; NEWER="newer" + +# +# CDATE: how we use it for naming of the archives +# DDATE: how the user should see it in our output (DISPLAY) +# +CDATE="date +%Y%m%d-%H%M" +DDATE="date +%Y-%m-%d-%H:%M:%S" + +# +# unset parallel execution +# +PARALLEL="" + +# +# catch signals +# +trap "rm -f \"${TMP}\"" 1 2 15 + +# +# Functions +# + +# time displaying echo +_techo() +{ + echo "$(${DDATE}): $@" +} + +# exit on error +_exit_err() +{ + _techo "$@" + rm -f "${TMP}" + exit 1 +} + +add_name() +{ + awk "{ print \"[${name}] \" \$0 }" +} + +pcmd() +{ + if [ "$remote_host" ]; then + ssh "$remote_host" "$@" + else + "$@" + fi +} + +# +# Version +# +display_version() +{ + echo "${FULL_VERSION}" + exit 0 +} + +# +# Tell how to use us +# +usage() +{ + echo "${__myname}: [args] " + echo "" + echo " ccollect creates (pseudo) incremental backups" + echo "" + echo " -h, --help: Show this help screen" + echo " -p, --parallel: Parallelise backup processes" + echo " -a, --all: Backup all sources specified in ${CSOURCES}" + echo " -v, --verbose: Be very verbose (uses set -x)" + echo " -V, --version: Print version information" + echo "" + echo " This is version ${VERSION}, released on ${RELEASE}" + echo " (the first version was written on 2005-12-05 by Nico Schottelius)." + echo "" + echo " Retrieve latest ccollect at http://unix.schottelius.org/ccollect/" + exit 0 +} + +# +# Select interval if AUTO +# +# For this to work nicely, you have to choose interval names that sort nicely +# such as int1, int2, int3 or a_daily, b_weekly, c_monthly, etc. +# +auto_interval() +{ + if [ -d "${backup}/intervals" -a -n "$(ls "${backup}/intervals" 2>/dev/null)" ] ; then + intervals_dir="${backup}/intervals" + elif [ -d "${CDEFAULTS}/intervals" -a -n "$(ls "${CDEFAULTS}/intervals" 2>/dev/null)" ] ; then + intervals_dir="${CDEFAULTS}/intervals" + else + _exit_err "No intervals are defined. Skipping." + fi + echo intervals_dir=${intervals_dir} + + trial_interval="$(ls -1r "${intervals_dir}/" | head -n 1)" || \ + _exit_err "Failed to list contents of ${intervals_dir}/." + _techo "Considering interval ${trial_interval}" + most_recent="$(pcmd ls -${TSORT}p1 "${ddir}" | grep "^${trial_interval}.*/$" | head -n 1)" || \ + _exit_err "Failed to list contents of ${ddir}/." + _techo " Most recent ${trial_interval}: '${most_recent}'" + if [ -n "${most_recent}" ]; then + no_intervals="$(ls -1 "${intervals_dir}/" | wc -l)" + n=1 + while [ "${n}" -le "${no_intervals}" ]; do + trial_interval="$(ls -p1 "${intervals_dir}/" | tail -n+${n} | head -n 1)" + _techo "Considering interval '${trial_interval}'" + c_interval="$(cat "${intervals_dir}/${trial_interval}" 2>/dev/null)" + m=$((${n}+1)) + set -- "${ddir}" -maxdepth 1 + while [ "${m}" -le "${no_intervals}" ]; do + interval_m="$(ls -1 "${intervals_dir}/" | tail -n+${m} | head -n 1)" + most_recent="$(pcmd ls -${TSORT}p1 "${ddir}" | grep "^${interval_m}\..*/$" | head -n 1)" + _techo " Most recent ${interval_m}: '${most_recent}'" + if [ -n "${most_recent}" ] ; then + set -- "$@" -$NEWER "${ddir}/${most_recent}" + fi + m=$((${m}+1)) + done + count=$(pcmd find "$@" -iname "${trial_interval}*" | wc -l) + _techo " Found $count more recent backups of ${trial_interval} (limit: ${c_interval})" + if [ "$count" -lt "${c_interval}" ] ; then + break + fi + n=$((${n}+1)) + done + fi + export INTERVAL="${trial_interval}" + D_FILE_INTERVAL="${intervals_dir}/${INTERVAL}" + D_INTERVAL=$(cat "${D_FILE_INTERVAL}" 2>/dev/null) +} + +# +# need at least interval and one source or --all +# +if [ $# -lt 2 ]; then + if [ "$1" = "-V" -o "$1" = "--version" ]; then + display_version + else + usage + fi +fi + +# +# check for configuraton directory +# +[ -d "${CCOLLECT_CONF}" ] || _exit_err "No configuration found in " \ + "\"${CCOLLECT_CONF}\" (is \$CCOLLECT_CONF properly set?)" + +# +# Filter arguments +# +export INTERVAL="$1"; shift +i=1 +no_sources=0 + +# +# Create source "array" +# +while [ "$#" -ge 1 ]; do + eval arg=\"\$1\"; shift + + if [ "${NO_MORE_ARGS}" = 1 ]; then + eval source_${no_sources}=\"${arg}\" + no_sources=$((${no_sources}+1)) + + # make variable available for subscripts + eval export source_${no_sources} + else + case "${arg}" in + -a|--all) + ALL=1 + ;; + -v|--verbose) + VERBOSE=1 + ;; + -p|--parallel) + PARALLEL=1 + ;; + -h|--help) + usage + ;; + --) + NO_MORE_ARGS=1 + ;; + *) + eval source_${no_sources}=\"$arg\" + no_sources=$(($no_sources+1)) + ;; + esac + fi + + i=$(($i+1)) +done + +# also export number of sources +export no_sources + +# +# be really, really, really verbose +# +if [ "${VERBOSE}" = 1 ]; then + set -x +fi + +# +# Look, if we should take ALL sources +# +if [ "${ALL}" = 1 ]; then + # reset everything specified before + no_sources=0 + + # + # get entries from sources + # + cwd=$(pwd -P) + ( cd "${CSOURCES}" && ls > "${TMP}" ); ret=$? + + [ "${ret}" -eq 0 ] || _exit_err "Listing of sources failed. Aborting." + + while read tmp; do + eval source_${no_sources}=\"${tmp}\" + no_sources=$((${no_sources}+1)) + done < "${TMP}" +fi + +# +# Need at least ONE source to backup +# +if [ "${no_sources}" -lt 1 ]; then + usage +else + _techo "${HALF_VERSION}: Beginning backup using interval ${INTERVAL}" +fi + +# +# Look for pre-exec command (general) +# +if [ -x "${CPREEXEC}" ]; then + _techo "Executing ${CPREEXEC} ..." + "${CPREEXEC}"; ret=$? + _techo "Finished ${CPREEXEC} (return code: ${ret})." + + [ "${ret}" -eq 0 ] || _exit_err "${CPREEXEC} failed. Aborting" +fi + +# +# check default configuration +# + +D_FILE_INTERVAL="${CDEFAULTS}/intervals/${INTERVAL}" +D_INTERVAL=$(cat "${D_FILE_INTERVAL}" 2>/dev/null) + + +# +# Let's do the backup +# +i=0 +while [ "${i}" -lt "${no_sources}" ]; do + + # + # Get current source + # + eval name=\"\$source_${i}\" + i=$((${i}+1)) + + export name + + # + # start ourself, if we want parallel execution + # + if [ "${PARALLEL}" ]; then + "$0" "${INTERVAL}" "${name}" & + continue + fi + +# +# Start subshell for easy log editing +# +( + # + # Stderr to stdout, so we can produce nice logs + # + exec 2>&1 + + # + # Configuration + # + backup="${CSOURCES}/${name}" + c_source="${backup}/source" + c_dest="${backup}/destination" + c_exclude="${backup}/exclude" + c_verbose="${backup}/verbose" + c_vverbose="${backup}/very_verbose" + c_rsync_extra="${backup}/rsync_options" + c_summary="${backup}/summary" + c_pre_exec="${backup}/pre_exec" + c_post_exec="${backup}/post_exec" + f_incomplete="delete_incomplete" + c_incomplete="${backup}/${f_incomplete}" + c_remote_host="${backup}/remote_host" + + # + # Marking backups: If we abort it's not removed => Backup is broken + # + c_marker=".ccollect-marker" + + # + # Times + # + begin_s=$(date +%s) + + # + # unset possible options + # + EXCLUDE="" + RSYNC_EXTRA="" + SUMMARY="" + VERBOSE="" + VVERBOSE="" + DELETE_INCOMPLETE="" + + _techo "Beginning to backup" + + # + # Standard configuration checks + # + if [ ! -e "${backup}" ]; then + _exit_err "Source does not exist." + fi + + # + # configuration _must_ be a directory + # + if [ ! -d "${backup}" ]; then + _exit_err "\"${name}\" is not a cconfig-directory. Skipping." + fi + + # + # first execute pre_exec, which may generate destination or other + # parameters + # + if [ -x "${c_pre_exec}" ]; then + _techo "Executing ${c_pre_exec} ..." + "${c_pre_exec}"; ret="$?" + _techo "Finished ${c_pre_exec} (return code ${ret})." + + if [ "${ret}" -ne 0 ]; then + _exit_err "${c_pre_exec} failed. Skipping." + fi + fi + + # + # Destination is a path + # + if [ ! -f "${c_dest}" ]; then + _exit_err "Destination ${c_dest} is not a file. Skipping." + else + ddir=$(cat "${c_dest}"); ret="$?" + if [ "${ret}" -ne 0 ]; then + _exit_err "Destination ${c_dest} is not readable. Skipping." + fi + fi + + # + # interval definition: First try source specific, fallback to default + # + if [ ${INTERVAL} = "AUTO" ] ; then + auto_interval + _techo "Selected interval: '$INTERVAL'" + fi + c_interval="$(cat "${backup}/intervals/${INTERVAL}" 2>/dev/null)" + + if [ -z "${c_interval}" ]; then + c_interval="${D_INTERVAL}" + + if [ -z "${c_interval}" ]; then + _exit_err "No definition for interval \"${INTERVAL}\" found. Skipping." + fi + fi + + # + # Source checks + # + if [ ! -f "${c_source}" ]; then + _exit_err "Source description \"${c_source}\" is not a file. Skipping." + else + source=$(cat "${c_source}"); ret="$?" + if [ "${ret}" -ne 0 ]; then + _exit_err "Source ${c_source} is not readable. Skipping." + fi + fi + # Verify source is up and accepting connections before deleting any old backups + rsync "$source" >/dev/null || _exit_err "Source ${source} is not readable. Skipping." + + # + # do we backup to a remote host? then set pre-cmd + # + if [ -f "${c_remote_host}" ]; then + # adjust ls and co + remote_host=$(cat "${c_remote_host}"); ret="$?" + if [ "${ret}" -ne 0 ]; then + _exit_err "Remote host file ${c_remote_host} exists, but is not readable. Skipping." + fi + destination="${remote_host}:${ddir}" + else + remote_host="" + destination="${ddir}" + fi + export remote_host + + # + # check for existence / use real name + # + ( pcmd cd "$ddir" ) || _exit_err "Cannot change to ${ddir}. Skipping." + + + # + # Check whether to delete incomplete backups + # + if [ -f "${c_incomplete}" -o -f "${CDEFAULTS}/${f_incomplete}" ]; then + DELETE_INCOMPLETE="yes" + fi + + # NEW method as of 0.6: + # - insert ccollect default parameters + # - insert options + # - insert user options + + # + # rsync standard options + # + + set -- "$@" "--archive" "--delete" "--numeric-ids" "--relative" \ + "--delete-excluded" "--sparse" + + # + # exclude list + # + if [ -f "${c_exclude}" ]; then + set -- "$@" "--exclude-from=${c_exclude}" + fi + + # + # Output a summary + # + if [ -f "${c_summary}" ]; then + set -- "$@" "--stats" + fi + + # + # Verbosity for rsync + # + if [ -f "${c_vverbose}" ]; then + set -- "$@" "-vv" + elif [ -f "${c_verbose}" ]; then + set -- "$@" "-v" + fi + + # + # extra options for rsync provided by the user + # + if [ -f "${c_rsync_extra}" ]; then + while read line; do + set -- "$@" "$line" + done < "${c_rsync_extra}" + fi + + # + # Check for incomplete backups + # + pcmd ls -1 "$ddir/${INTERVAL}"*".${c_marker}" > "${TMP}" 2>/dev/null + + i=0 + while read incomplete; do + eval incomplete_$i=\"$(echo ${incomplete} | sed "s/\\.${c_marker}\$//")\" + i=$(($i+1)) + done < "${TMP}" + + j=0 + while [ "$j" -lt "$i" ]; do + eval realincomplete=\"\$incomplete_$j\" + _techo "Incomplete backup: ${realincomplete}" + if [ "${DELETE_INCOMPLETE}" = "yes" ]; then + _techo "Deleting ${realincomplete} ..." + pcmd rm $VVERBOSE -rf "${ddir}/${realincomplete}" || \ + _exit_err "Removing ${realincomplete} failed." + fi + j=$(($j+1)) + done + + # + # check if maximum number of backups is reached, if so remove + # use grep and ls -p so we only look at directories + # + count="$(pcmd ls -p1 "${ddir}" | grep "^${INTERVAL}\..*/\$" | wc -l \ + | sed 's/^ *//g')" || _exit_err "Counting backups failed" + + _techo "Existing backups: ${count} Total keeping backups: ${c_interval}" + + if [ "${count}" -ge "${c_interval}" ]; then + substract=$((${c_interval} - 1)) + remove=$((${count} - ${substract})) + _techo "Removing ${remove} backup(s)..." + + pcmd ls -${TSORT}p1r "$ddir" | grep "^${INTERVAL}\..*/\$" | \ + head -n "${remove}" > "${TMP}" || \ + _exit_err "Listing old backups failed" + + i=0 + while read to_remove; do + eval remove_$i=\"${to_remove}\" + i=$(($i+1)) + done < "${TMP}" + + j=0 + while [ "$j" -lt "$i" ]; do + eval to_remove=\"\$remove_$j\" + _techo "Removing ${to_remove} ..." + pcmd rm ${VVERBOSE} -rf "${ddir}/${to_remove}" || \ + _exit_err "Removing ${to_remove} failed." + j=$(($j+1)) + done + fi + + + # + # Check for backup directory to clone from: Always clone from the latest one! + # + # Depending on your file system, you may want to sort on: + # 1. mtime (modification time) with TSORT=t, or + # 2. ctime (last change time, usually) with TSORT=tc + last_dir="$(pcmd ls -${TSORT}p1 "${ddir}" | grep '/$' | head -n 1)" || \ + _exit_err "Failed to list contents of ${ddir}." + + # + # clone from old backup, if existing + # + if [ "${last_dir}" ]; then + set -- "$@" "--link-dest=${ddir}/${last_dir}" + _techo "Hard linking from ${last_dir}" + fi + + + # set time when we really begin to backup, not when we began to remove above + destination_date=$(${CDATE}) + destination_dir="${ddir}/${INTERVAL}.${destination_date}.$$" + destination_full="${destination}/${INTERVAL}.${destination_date}.$$" + + # give some info + _techo "Beginning to backup, this may take some time..." + + _techo "Creating ${destination_dir} ..." + pcmd mkdir ${VVERBOSE} "${destination_dir}" || \ + _exit_err "Creating ${destination_dir} failed. Skipping." + + # + # added marking in 0.6 (and remove it, if successful later) + # + pcmd touch "${destination_dir}.${c_marker}" + + # + # the rsync part + # + _techo "Transferring files..." + rsync "$@" "${source}" "${destination_full}"; ret=$? + # Correct the modification time: + pcmd touch "${destination_dir}" + + # + # remove marking here + # + if [ "$ret" -ne 12 ] ; then + pcmd rm "${destination_dir}.${c_marker}" || \ + _exit_err "Removing ${destination_dir}/${c_marker} failed." + fi + + _techo "Finished backup (rsync return code: $ret)." + if [ "${ret}" -ne 0 ]; then + _techo "Warning: rsync exited non-zero, the backup may be broken (see rsync errors)." + fi + + # + # post_exec + # + if [ -x "${c_post_exec}" ]; then + _techo "Executing ${c_post_exec} ..." + "${c_post_exec}"; ret=$? + _techo "Finished ${c_post_exec}." + + if [ ${ret} -ne 0 ]; then + _exit_err "${c_post_exec} failed." + fi + fi + + # Calculation + end_s=$(date +%s) + + full_seconds=$((${end_s} - ${begin_s})) + hours=$((${full_seconds} / 3600)) + seconds=$((${full_seconds} - (${hours} * 3600))) + minutes=$((${seconds} / 60)) + seconds=$((${seconds} - (${minutes} * 60))) + + _techo "Backup lasted: ${hours}:${minutes}:${seconds} (h:m:s)" + +) | add_name +done + +# +# Be a good parent and wait for our children, if they are running wild parallel +# +if [ "${PARALLEL}" ]; then + _techo "Waiting for children to complete..." + wait +fi + +# +# Look for post-exec command (general) +# +if [ -x "${CPOSTEXEC}" ]; then + _techo "Executing ${CPOSTEXEC} ..." + "${CPOSTEXEC}"; ret=$? + _techo "Finished ${CPOSTEXEC} (return code: ${ret})." + + if [ ${ret} -ne 0 ]; then + _techo "${CPOSTEXEC} failed." + fi +fi + +rm -f "${TMP}" +_techo "Finished ${WE}" + +# vim: set shiftwidth=3 tabstop=3 expandtab : diff --git a/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/d.patch b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/d.patch new file mode 100644 index 00000000..7fae4107 --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/d.patch @@ -0,0 +1,17 @@ +--- ccollect-0.7.1-c.sh 2009-05-24 21:39:43.000000000 -0700 ++++ ccollect-0.7.1-d.sh 2009-05-24 21:47:09.000000000 -0700 +@@ -492,12 +492,12 @@ + if [ "${count}" -ge "${c_interval}" ]; then + substract=$((${c_interval} - 1)) + remove=$((${count} - ${substract})) + _techo "Removing ${remove} backup(s)..." + +- pcmd ls -p1 "$ddir" | grep "^${INTERVAL}\..*/\$" | \ +- sort -n | head -n "${remove}" > "${TMP}" || \ ++ pcmd ls -${TSORT}p1r "$ddir" | grep "^${INTERVAL}\..*/\$" | \ ++ head -n "${remove}" > "${TMP}" || \ + _exit_err "Listing old backups failed" + + i=0 + while read to_remove; do + eval remove_$i=\"${to_remove}\" diff --git a/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/e.patch b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/e.patch new file mode 100644 index 00000000..d277c06e --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/e.patch @@ -0,0 +1,19 @@ +--- ccollect-0.7.1-d.sh 2009-05-24 21:47:09.000000000 -0700 ++++ ccollect-0.7.1-e.sh 2009-05-24 22:18:16.000000000 -0700 +@@ -560,12 +560,14 @@ + pcmd touch "${destination_dir}" + + # + # remove marking here + # +- pcmd rm "${destination_dir}.${c_marker}" || \ +- _exit_err "Removing ${destination_dir}/${c_marker} failed." ++ if [ "$ret" -ne 12 ] ; then ++ pcmd rm "${destination_dir}.${c_marker}" || \ ++ _exit_err "Removing ${destination_dir}/${c_marker} failed." ++ fi + + _techo "Finished backup (rsync return code: $ret)." + if [ "${ret}" -ne 0 ]; then + _techo "Warning: rsync exited non-zero, the backup may be broken (see rsync errors)." + fi diff --git a/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/f.patch b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/f.patch new file mode 100644 index 00000000..3bedf34e --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/jlawless-2009-06-03/old/f.patch @@ -0,0 +1,119 @@ +--- ccollect-0.7.1-e.sh 2009-05-24 22:18:16.000000000 -0700 ++++ ccollect-0.7.1-f.sh 2009-05-24 22:19:50.000000000 -0700 +@@ -124,10 +124,64 @@ + echo " Retrieve latest ccollect at http://unix.schottelius.org/ccollect/" + exit 0 + } + + # ++# Select interval if AUTO ++# ++# For this to work nicely, you have to choose interval names that sort nicely ++# such as int1, int2, int3 or a_daily, b_weekly, c_monthly, etc. ++# ++auto_interval() ++{ ++ if [ -d "${backup}/intervals" -a -n "$(ls "${backup}/intervals" 2>/dev/null)" ] ; then ++ intervals_dir="${backup}/intervals" ++ elif [ -d "${CDEFAULTS}/intervals" -a -n "$(ls "${CDEFAULTS}/intervals" 2>/dev/null)" ] ; then ++ intervals_dir="${CDEFAULTS}/intervals" ++ else ++ _exit_err "No intervals are defined. Skipping." ++ fi ++ echo intervals_dir=${intervals_dir} ++ ++ trial_interval="$(ls -1r "${intervals_dir}/" | head -n 1)" || \ ++ _exit_err "Failed to list contents of ${intervals_dir}/." ++ _techo "Considering interval ${trial_interval}" ++ most_recent="$(pcmd ls -${TSORT}p1 "${ddir}" | grep "^${trial_interval}.*/$" | head -n 1)" || \ ++ _exit_err "Failed to list contents of ${ddir}/." ++ _techo " Most recent ${trial_interval}: '${most_recent}'" ++ if [ -n "${most_recent}" ]; then ++ no_intervals="$(ls -1 "${intervals_dir}/" | wc -l)" ++ n=1 ++ while [ "${n}" -le "${no_intervals}" ]; do ++ trial_interval="$(ls -p1 "${intervals_dir}/" | tail -n+${n} | head -n 1)" ++ _techo "Considering interval '${trial_interval}'" ++ c_interval="$(cat "${intervals_dir}/${trial_interval}" 2>/dev/null)" ++ m=$((${n}+1)) ++ set -- "${ddir}" -maxdepth 1 ++ while [ "${m}" -le "${no_intervals}" ]; do ++ interval_m="$(ls -1 "${intervals_dir}/" | tail -n+${m} | head -n 1)" ++ most_recent="$(pcmd ls -${TSORT}p1 "${ddir}" | grep "^${interval_m}\..*/$" | head -n 1)" ++ _techo " Most recent ${interval_m}: '${most_recent}'" ++ if [ -n "${most_recent}" ] ; then ++ set -- "$@" -$NEWER "${ddir}/${most_recent}" ++ fi ++ m=$((${m}+1)) ++ done ++ count=$(pcmd find "$@" -iname "${trial_interval}*" | wc -l) ++ _techo " Found $count more recent backups of ${trial_interval} (limit: ${c_interval})" ++ if [ "$count" -lt "${c_interval}" ] ; then ++ break ++ fi ++ n=$((${n}+1)) ++ done ++ fi ++ export INTERVAL="${trial_interval}" ++ D_FILE_INTERVAL="${intervals_dir}/${INTERVAL}" ++ D_INTERVAL=$(cat "${D_FILE_INTERVAL}" 2>/dev/null) ++} ++ ++# + # need at least interval and one source or --all + # + if [ $# -lt 2 ]; then + if [ "$1" = "-V" -o "$1" = "--version" ]; then + display_version +@@ -344,12 +398,28 @@ + _exit_err "${c_pre_exec} failed. Skipping." + fi + fi + + # ++ # Destination is a path ++ # ++ if [ ! -f "${c_dest}" ]; then ++ _exit_err "Destination ${c_dest} is not a file. Skipping." ++ else ++ ddir=$(cat "${c_dest}"); ret="$?" ++ if [ "${ret}" -ne 0 ]; then ++ _exit_err "Destination ${c_dest} is not readable. Skipping." ++ fi ++ fi ++ ++ # + # interval definition: First try source specific, fallback to default + # ++ if [ ${INTERVAL} = "AUTO" ] ; then ++ auto_interval ++ _techo "Selected interval: '$INTERVAL'" ++ fi + c_interval="$(cat "${backup}/intervals/${INTERVAL}" 2>/dev/null)" + + if [ -z "${c_interval}" ]; then + c_interval="${D_INTERVAL}" + +@@ -371,22 +441,10 @@ + fi + # Verify source is up and accepting connections before deleting any old backups + rsync "$source" >/dev/null || _exit_err "Source ${source} is not readable. Skipping." + + # +- # Destination is a path +- # +- if [ ! -f "${c_dest}" ]; then +- _exit_err "Destination ${c_dest} is not a file. Skipping." +- else +- ddir=$(cat "${c_dest}"); ret="$?" +- if [ "${ret}" -ne 0 ]; then +- _exit_err "Destination ${c_dest} is not readable. Skipping." +- fi +- fi +- +- # + # do we backup to a remote host? then set pre-cmd + # + if [ -f "${c_remote_host}" ]; then + # adjust ls and co + remote_host=$(cat "${c_remote_host}"); ret="$?" diff --git a/software/ccollect/ccollect-0.8/contrib/lucky-2009-07-22/ccollect_logwrapper_destination.patch b/software/ccollect/ccollect-0.8/contrib/lucky-2009-07-22/ccollect_logwrapper_destination.patch new file mode 100644 index 00000000..5fb20b5a --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/lucky-2009-07-22/ccollect_logwrapper_destination.patch @@ -0,0 +1,14 @@ +31c31,41 +< logdir="${LOGCONF}/destination" +--- +> c_dest="${LOGCONF}/destination" +> +> if [ ! -f ${c_dest} ]; then +> _exit_err "Destination ${c_dest} is not a file. Skipping." +> else +> logdir=$(cat "${c_dest}"); ret="$?" +> if [ "${ret}" -ne 0 ]; then +> _exit_err "Destination ${c_dest} is not readable. Skipping." +> fi +> fi +> diff --git a/software/ccollect/ccollect-0.8/contrib/lucky-2009-07-22/ccollect_stats.sh b/software/ccollect/ccollect-0.8/contrib/lucky-2009-07-22/ccollect_stats.sh new file mode 100644 index 00000000..886be092 --- /dev/null +++ b/software/ccollect/ccollect-0.8/contrib/lucky-2009-07-22/ccollect_stats.sh @@ -0,0 +1,26 @@ +#!/bin/sh +# +# 2007 Daniel Aubry +# 2008 Nico Schottelius (added minimal header) +# +# Copying license: GPL2-only +# + +# TODO: +# add variables, add copying, add configuration + +if [ ! -e /tmp/ccollect-stats.lock ] +then + touch /tmp/ccollect-stats.lock + + # changes after license clearify + # for dest in /etc/ccollect/sources/ -type f -name destination | while read line + + find /etc/ccollect/sources/*/destination | while read line + do + d=$(basename $(cat $line)) + echo "====[Backup: $backupname]====" | tee -a /var/log/backup.log + du -sh $line/* | tee -a /var/log/backup.log + done + rm /tmp/ccollect-stats.lock +fi diff --git a/software/ccollect/ccollect-0.8/doc/HACKING b/software/ccollect/ccollect-0.8/doc/HACKING new file mode 100644 index 00000000..00db4d11 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/HACKING @@ -0,0 +1,37 @@ +Hello Hacker, + +I really appreciate your interest in hacking this software, but +I am kind of critical when seeing patches. Thus I created this +file to give you some hints of my thinking quirks. + + +Submitting patches +------------------ +Make my life easier, make your life easier, use a version control system (vcs). +For this software the preferred vcs is git. Clone the latest repo, create +a new local branch (git checkout -b ) write down your ideas. + +When you're done, push all your stuff out to some public repo and drop a +mail to the mailinglist, what you did and where to get it. + + +Introduce a feature or change behaviour +--------------------------------------- +Uhh, fancy! You have had a great idea, then it's time to change +the major version, so others know that something changed. + +If the configuration format is changed, add a script to tools/ +to allow users upgrade their configuration to this major version. + +And now comes the most difficult part: Add documentation. Nobody +benefits from your cool feature, if it is not known. I know, writing +documentation is not so much fun, but you also expect good documentation +for this software, don't you? + + +If you think my thinking quirks must be corrected +------------------------------------------------- +See above ("Submitting patches") and submit a patch to this file. + + +Thanks for reading. diff --git a/software/ccollect/ccollect-0.8/doc/braindumps/LOCAL_vs._REMOTE b/software/ccollect/ccollect-0.8/doc/braindumps/LOCAL_vs._REMOTE new file mode 100644 index 00000000..f2a40b70 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/braindumps/LOCAL_vs._REMOTE @@ -0,0 +1,35 @@ + to Local to Remote + backup destination is exiting + pre/postexec runs locally + --link-dest? + /delete_incomplete - can chech ddir + + can check destination dir + -> dooooooo it before! + + + remote_host! + => rddir_ls: + incomplete: ls -1 "${INTERVAL}"*".${c_marker}" + + host support? + ssh-host-support? + + => ssh_host => save to host + execute commands there! + + rm! + + --link-dest? + + --link-dest=DIR + => remote dirs, rsync remote + => works!!!! + + local_destination + remote_destination + => remote_* + + both + configuration is local (what to where) + diff --git a/software/ccollect/ccollect-0.8/doc/braindumps/README b/software/ccollect/ccollect-0.8/doc/braindumps/README new file mode 100644 index 00000000..973addcb --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/braindumps/README @@ -0,0 +1 @@ +Do not read the files in this directory diff --git a/software/ccollect/ccollect-0.8/doc/ccollect-restoring.text b/software/ccollect/ccollect-0.8/doc/ccollect-restoring.text new file mode 100644 index 00000000..7bb29eae --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/ccollect-restoring.text @@ -0,0 +1,196 @@ +ccollect - Restoring backups +============================ +Nico Schottelius +0.1, for all ccollect version, Initial Version from 2008-07-04 +:Author Initials: NS + + +Having backups is half the way to success on a failure. +Knowing how to restore the systems is the other half. + + +Introduction +------------ +You made your backup and now you want to restore your +data. If you backuped only parts of a computer and need +only to restore them, it is pretty easy to achieve. +Restoring a whole system is a little bit more +difficult and needs some knowledge of the operating system. + + +Restoring parts of a system +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Log into your backupserver. Change into the +backup directory you want to restore from. +Do `rsync -av './files/to/be/recovered/' 'sourcehost:/files/to/be/recovered/'. + +Restoring a complete system (general) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Boot the system to be rescued from a media that contains low level tools +for your OS (like partitioning, formatting) and the necessary tools +(ssh, tar or rsync). +Use +- create the necessary partition table (or however it is called + +Get a live-cd, that ships with +- rsync / tar +- ssh (d) -> from backupserver +- support for the filesystems + +Restoring a complete FreeBSD system +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Get a FreeBSD-live-cd (I used the FreeBSD 7.0 live CD, +but FreeSBIE (http://www.freesbie.org/), +Frenzy (http://frenzy.org.ua/en/) or the +FreeBSD LiveCD (http://livecd.sourceforge.net/) +may also be helpful. The following way uses the FreeBSD 7.0 +live cd. + +So boot it up, select your language. After that select +*Custom* then *Partition*. Create the slice like you want +to have it. Then let the installer write into the MBR, +select *BootMgr*. + +After that create the necessary labels, select *Label* and +make sure "Newfs" flag is set to "Y". + +Finally, select *Commit* and choose an installation type +that must fail, because we want the installer only to write +the partitions and labels, but not to install anything on it. + +At this point we have created the base for restoring the whole +system. Move back to the main menu and select *Fixit*, then +*CDROM/DVD*. This starts a shell on TTY4, which can be reached +by pressing *ALT+F4*. Then enter the following data: + +-------------------------------------------------------------------------------- + rootdir=/ccollect + rootdev=/dev/ad0s1a + backupserver=192.42.23.5 + + # create destination directory + mkdir "$rootdir" + + # mount root; add other mounts if you created more labels + mount "$rootdev" "$rootdir" + + # find out which network devices exist + ifconfig + + # create the directory, because dhclient needs it + mkdir /var/db + + # retrieve an ip address + dhclient fxp0 + + # test connection + ssh "$backupserver" + + # go back + backupserver% exit +-------------------------------------------------------------------------------- + +Now we've prepared everything for the real backup. The next problem maybe, +that we cannot (should not) be able to login as root to the backup server. +Additionally the system to be restored may not reachable from the backup server, +because it is behind a firewall or nat. +Thus I describe a way, that is a little bit more complicated for those, that +do not have these limitations, but works in both scenarios. + +I just start netcat on the local machine, pipe its output to tar and put +both into the background. Then I create a ssh tunnel to the backupserver, +which is then able to connect to my netcat "directly". + +-------------------------------------------------------------------------------- + # user to connect to the backupserver + myuser=nico + + # our name in the backup + restorehost=server1 + + # the instance to be used + backup="weekly.20080718-2327.23053" + + # Need to setup lo0 first, the livecd did not do it for me + ifconfig lo0 127.0.0.1 up + + # change to the destination directory + cd "$rootdir" + + # start listener + ( nc -l 127.0.0.1 4242 | tar xvf - ) & + + # verify that it runs correctly + sockstat -4l + + # connect as a normal user to the backupserver + ssh -R4242:127.0.0.1:4242 "$myuser@$backupserver" + + # become root + backupserver% su - + + # change to the source directory + backupserver# cd /home/server/backup/$restorehost/$backup + + # begin the backup + backup # tar cf - . | nc 127.0.0.1 4242 + + # wait until it finishes, press ctrl-c to kill netcat + # logoff the backupserver + backupserver# exit + backupserver% exit +-------------------------------------------------------------------------------- + +Now we are just right next to be finished. Still, we have to take care about +some things: + +- Do the block devices still have the same names? If not, correct /etc/fstab. +- Do the network devices still have the same names? If not, correct /etc/rc.conf. + +If everything is fixed, let us finish the restore: + +-------------------------------------------------------------------------------- + # cleanly umount it + umount "$rootdir" + + # reboot, remove the cd and bootup the restored system + reboot +-------------------------------------------------------------------------------- + +Restoring a complete Linux system +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Knoppix + knoppix 2 at boot prompt + + rootdir=/ccollect + dev=/dev/hda + rootdev="${dev}1" + fs=jfs + tar + + # create the needed partitions + cfdisk $dev + + mkfs.$fs $rootdev + + mkdir $rootdir + + mount $rootdev $rootdir + + cd $rootdir + + pump + ifconfig + + # start listener (from now on it is the same as + ( nc -l 127.0.0.1 4242 | tar xvf - ) & + + + +TO BE DONE + +Future +------ +I think about automating full system recoveries in the future. +I think it could be easily done and here are some hints for +people who would like to implement it. diff --git a/software/ccollect/ccollect-0.8/doc/ccollect.text b/software/ccollect/ccollect-0.8/doc/ccollect.text new file mode 100644 index 00000000..720bcc8b --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/ccollect.text @@ -0,0 +1,1174 @@ +ccollect - Installing, Configuring and Using +============================================ +Nico Schottelius +0.8, for ccollect 0.8, Initial Version from 2006-01-13 +:Author Initials: NS + + +(pseudo) incremental backup +with different exclude lists +using hardlinks and `rsync` + + +Introduction +------------ +`ccollect` is a backup utility written in the sh-scripting language. +It does not depend on a specific shell, only `/bin/sh` needs to be +bourne shell compatible (like 'dash', 'ksh', 'zsh', 'bash', ...). + + +Supported and tested operating systems and architectures +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +`ccollect` was successfully tested on the following platforms: + +- GNU/Linux on amd64/hppa/i386/ppc/ARM +- FreeBSD on amd64/i386 +- Mac OS X 10.5 +- NetBSD on alpha/amd64/i386/sparc/sparc64 +- OpenBSD on amd64 + +It *should* run on any Unix that supports `rsync` and has a POSIX-compatible +bourne shell. If your platform is not listed above and you have it successfully +running, please drop me a mail. + + +Why you COULD only backup from remote hosts, not to them +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +While considering the design of ccollect, I thought about enabling +backup to *remote* hosts. Though this sounds like a nice feature +('"Backup my notebook to the server now."'), in my opinion it is a +bad idea to backup to a remote host. + +But as more and more people requested this feature, it was implemented, +so you have the choice whether you want to use it or not. + + +Reason +^^^^^^ +If you want to backup *TO* a remote host, you have to loosen security on it. + +Imagine the following situation: You backup your farm of webservers *TO* +a backup host somewhere else. +Now one of your webservers which has access to your backup host gets +compromised. + +Your backup server will be compromised, too. + +And the attacker will have access to all data on the other webservers. + + +Doing it securely +^^^^^^^^^^^^^^^^^ +Think of it the other way round: The backup server (now behind a +firewall, not accessable from outside) connects to the +webservers and pulls the data *from* them. If someone gets access to one +of the webservers, this person will perhaps not even see your machine. If +the attacker sees connections from a host to the compromised +machine, she will not be able to log in on the backup machine. +All other backups are still secure. + + +Incompatibilities and changes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +Versions 0.7 and 0.8 +^^^^^^^^^^^^^^^^^^^^^ + +.The argument order changed: +- Old: " [args] " +- New: "[args] " + +If you did not use arguments (most people do not), nothing will +change for you. + +.Deletion of incomplete backups using the 'delete_incomplete' option +- Old: Only incomplete backups from the current interval have been removed +- New: All incomplete backups are deleted + +.Support for standard values +- Old: no support +- New: Options in $CCOLLECT_CONF/defaults are used as defaults (see below) + +Versions 0.6 and 0.7 +^^^^^^^^^^^^^^^^^^^^^ +.The format of `destination` changed: +- Before 0.7 it was a (link to a) directory +- As of 0.7 it is a textfile containing the destination + +You can update your configuration using `tools/config-pre-0.7-to-0.7.sh`. + +.Added 'remote_host' +- As of 0.7 it is possible to backup *to* hosts (see section remote_host below). + + +Versions 0.5 and 0.6 +^^^^^^^^^^^^^^^^^^^^^ +.The format of `rsync_options` changed: +- Before 0.6 it was whitespace delimeted +- As of 0.6 it is newline seperated (so you can pass whitespaces to `rsync`) + +You can update your configuration using `tools/config-pre-0.6-to-0.6.sh`. + +.The name of the backup directories changed: +- Before 0.6: "date +%Y-%m-%d-%H%M" +- As of 0.6: "date +%Y%m%d-%H%M" (better readable, date is closer together) + +For the second change there is no updated needed, as XXXX- is always before +XXXXX (- comes before digit). + + + +Versions 0.4 and 0.5 +^^^^^^^^^^^^^^^^^^^^^ +Not a real incompatibilty, but seems to fit in this section: + +.0.5 does *NOT* require +- PaX +- bc + +anymore! + + +Versions < 0.4 and 0.4 +^^^^^^^^^^^^^^^^^^^^^^ + +Since `ccollect` 0.4 there are several incompatibilities with earlier +versions: + +.List of incompatibilities +- `pax` (Posix) is now required, `cp -al` (GNU specific) is removed +- "interval" was written with two 'l' (ell), which is wrong in English +- Changed the name of backup directories, removed the colon in the interval +- ccollect will now exit when preexec returns non-zero +- ccollect now reports when postexec returns non-zero + +You can convert your old configuration directory using +`config-pre-0.4-to-0.4.sh`, which can be found in the *tools/* +subdirectory: + +-------------------------------------------------------------------------------- +[10:05] hydrogenium:ccollect-0.4# ./tools/config-pre-0.4-to-0.4.sh /etc/ccollect +-------------------------------------------------------------------------------- + + +Quick start +----------- +For those who do not want to read the whole long document: + +-------------------------------------------------------------------------------- +# get latest ccollect tarball from http://www.nico.schottelius.org/software/ccollect/ +# replace value for CCV with the current version +export CCV=0.8 + +# +# replace 'wget' with 'fetch' on bsd +# +holen=wget +"$holen" http://www.nico.schottelius.org/software/ccollect/ccollect-${CCV}.tar.bz2 + +# extract the tarball, change to the newly created directory +tar -xvjf ccollect-${CCV}.tar.bz2 +cd ccollect-${CCV} + +# create mini-configuration +# first create directory structure +mkdir -p miniconfig/defaults/intervals +mkdir miniconfig/sources + +# create sample intervals +echo 2 > miniconfig/defaults/intervals/testinterval +echo 3 > miniconfig/defaults/intervals/testinterval2 + +# create destination directory, where the backups will be kept +mkdir ~/DASI + +# create sample source, which will be saved +mkdir miniconfig/sources/testsource + +# We will save '/bin' to the directory '~/DASI' +echo '/bin' > miniconfig/sources/testsource/source + +# configure ccollect to use ~/DASI as destination +echo ~/DASI > miniconfig/sources/testsource/destination + +# We want to see what happens and also a small summary at the end +touch miniconfig/sources/testsource/verbose +touch miniconfig/sources/testsource/summary + +echo "do the backup, twice" +CCOLLECT_CONF=./miniconfig ./ccollect.sh testinterval testsource +CCOLLECT_CONF=./miniconfig ./ccollect.sh testinterval testsource + +echo "the third time ccollect begins to remove old backups" +echo -n "Hit enter to see it" +read +CCOLLECT_CONF=./miniconfig ./ccollect.sh testinterval testsource + +echo "Now we add another interval, ccollect should clone from existent ones" +echo -n "Hit enter to see it" +read +CCOLLECT_CONF=./miniconfig ./ccollect.sh testinterval2 testsource + +echo "Let's see how much space we used with two backups and compare it to /bin" +du -s ~/DASI /bin + +# report success +echo "Please report success using ./tools/report_success.sh" + +-------------------------------------------------------------------------------- + +Cutting and pasting the complete section above to your shell will result in +the download of ccollect, the creation of a sample configuration and the +execution of some backups. + + +Requirements +------------ + +Installing ccollect +~~~~~~~~~~~~~~~~~~~ +For the installation you need at least + + - the latest ccollect package (http://www.nico.schottelius.org/software/ccollect/) + - either `cp` and `chmod` or `install` + - for more comfort: `make` + - for rebuilding the generated documentation: additionally `asciidoc` + + +Using ccollect +~~~~~~~~~~~~~~ +.Running ccollect requires the following tools to be installed: + - `date` + - `rsync` + - `ssh` (if you want to use rsync over ssh, which is recommened for security) + + +Installing +---------- +Either type 'make install' or simply copy it to a directory in your +$PATH and execute 'chmod *0755* /path/to/ccollect.sh'. If you like +to use the new management scripts (available since 0.6), copy the +following scripts to a directory in $PATH: + +- `tools/ccollect_add_source.sh` +- `tools/ccollect_analyse_logs.sh.sh` +- `tools/ccollect_delete_source.sh` +- `tools/ccollect_list_intervals.sh` +- `tools/ccollect_logwrapper.sh` + +After having installed and used ccollect, report success using +'./tools/report_success.sh'. + + +Configuring +----------- +For configuration aid have a look at the above mentioned tools, which can assist +you quite well. When you are successfully using `ccollect`, report success using +`tools/report_success.sh`. + + +Runtime options +~~~~~~~~~~~~~~~ +`ccollect` looks for its configuration in '/etc/ccollect' or, if set, in +the directory specified by the variable '$CCOLLECT_CONF': +-------------------------------------------------------------------------------- +# sh-compatible (dash, zsh, mksh, ksh, bash, ...) +$ CCOLLECT_CONF=/your/config/dir ccollect.sh ... + +# csh +$ ( setenv CCOLLECT_CONF /your/config/dir ; ccollect.sh ... ) +-------------------------------------------------------------------------------- + +When you start `ccollect`, you have to specify in which interval +to backup (daily, weekly, yearly; you can specify the names yourself, see below) +and which sources to backup (or -a to backup all sources). + +The interval specifies how many backups are kept. + +There are also some self-explanatory parameters you can pass to ccollect, +simply use `ccollect.sh --help` for info. + + +General configuration +~~~~~~~~~~~~~~~~~~~~~ +The general configuration can be found in $CCOLLECT_CONF/defaults or +/etc/ccollect/defaults. All options specified there are generally valid for +all source definitions, although the values can be overwritten in the source +configuration. + +All configuration entries are plain-text files +(use UTF-8 for non-ascii characters). + + +Interval definition +^^^^^^^^^^^^^^^^^^^^ +The interval definition can be found in +'$CCOLLECT_CONF/defaults/intervals/' or '/etc/ccollect/defaults/intervals'. +Each file in this directory specifies an interval. The name of the file is +the same as the name of the interval: `intervals/''`. + +The content of this file should be a single line containing a number. +This number defines how many versions of this interval are kept. + +Example: +------------------------------------------------------------------------- + [10:23] zaphodbeeblebrox:ccollect-0.2% ls -l conf/defaults/intervals/ + insgesamt 12 + -rw-r--r-- 1 nico users 3 2005-12-08 10:24 daily + -rw-r--r-- 1 nico users 3 2005-12-08 11:36 monthly + -rw-r--r-- 1 nico users 2 2005-12-08 11:36 weekly + [10:23] zaphodbeeblebrox:ccollect-0.2% cat conf/defaults/intervals/* + 28 + 12 + 4 +-------------------------------------------------------------------------------- +This means to keep 28 daily backups, 12 monthly backups and 4 weekly. + + +General pre- and post-execution +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +If you add '$CCOLLECT_CONF/defaults/`pre_exec`' or +'/etc/ccollect/defaults/`pre_exec`' (same with `post_exec`), `ccollect` +will start `pre_exec` before the whole backup process and +`post_exec` after backup of all sources is done. + +The following example describes how to report free disk space in +human readable format before and after the whole backup process: +------------------------------------------------------------------------- +[13:00] hydrogenium:~# mkdir -p /etc/ccollect/defaults/ +[13:00] hydrogenium:~# echo '#!/bin/sh' > /etc/ccollect/defaults/pre_exec +[13:01] hydrogenium:~# echo '' >> /etc/ccollect/defaults/pre_exec +[13:01] hydrogenium:~# echo 'df -h' >> /etc/ccollect/defaults/pre_exec +[13:01] hydrogenium:~# chmod 0755 /etc/ccollect/defaults/pre_exec +[13:01] hydrogenium:~# ln -s /etc/ccollect/defaults/pre_exec /etc/ccollect/defaults/post_exec +------------------------------------------------------------------------- + + +Source configuration +~~~~~~~~~~~~~~~~~~~~ +Each source configuration exists in '$CCOLLECT_CONF/sources/$name' or +'/etc/ccollect/sources/$name'. + +The name you choose for the subdirectory describes the source. + +Each source contains at least the following files: + + - `source` (a text file containing the `rsync` compatible path to backup) + - `destination` (a text file containing the directory we should backup to) + +Additionally a source may have the following files: + + - `pre_exec` program to execute before backing up *this* source + - `post_exec` program to execute after backing up *this* source + + - `verbose` whether to be verbose (passes -v to `rsync`) + - `very_verbose` be very verbose (`mkdir -v`, `rm -v` and `rsync -vv`) + - `summary` create a transfer summary when `rsync` finished + + - `exclude` exclude list for `rsync`. newline seperated list. + - `rsync_options` extra options for `rsync`. newline seperated list. + + - `delete_incomplete` delete incomplete backups + - `remote_host` host to backup to + - `rsync_failure_codes` list of rsync exit codes that indicate complete failure + - `mtime` Sort backup directories based on their modification time + - `quiet_if_down` Suppress error messages if source is not connectable + + +Example: +-------------------------------------------------------------------------------- + [10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2 + insgesamt 12 + lrwxrwxrwx 1 nico users 20 2005-11-17 16:44 destination + -rw-r--r-- 1 nico users 62 2005-12-07 17:43 exclude + drwxr-xr-x 2 nico users 4096 2005-12-07 17:38 intervals + -rw-r--r-- 1 nico users 15 2005-11-17 16:44 source + [10:47] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude + openvpn-2.0.1.tar.gz + nicht_reinnehmen + etwas mit leerzeichenli + [10:47] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervals + insgesamt 4 + -rw-r--r-- 1 nico users 2 2005-12-07 17:38 daily + [10:48] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/intervals/daily + 5 + [10:48] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/source + /home/nico/vpn +-------------------------------------------------------------------------------- + +Default options +^^^^^^^^^^^^^^^ +If you add '$CCOLLECT_CONF/defaults/`option_name`', the value will +be used in abscence of the option in a source. If you want to prevent +the default value to be used in a source, you can create the file +'$CCOLLECT_CONF/sources/$name/`no_option_name`' (i.e. prefix it with +'no_'. + +Example: +-------------------------------------------------------------------------------- + [9:04] ikn2:ccollect% touch conf/defaults/verbose + [9:04] ikn2:ccollect% touch conf/sources/local/no_verbose +-------------------------------------------------------------------------------- +This enables the verbose option for all sources, but disables it for the +source 'local'. + +If an option is specified in the defaults folder and in the source, +the source specific version overrides the default one: + +Example: +-------------------------------------------------------------------------------- + [9:05] ikn2:ccollect% echo "backup-host" > conf/defaults/remote_host + [9:05] ikn2:ccollect% echo "different-host" > conf/sources/local/remote_host +-------------------------------------------------------------------------------- + +You can use all source options as defaults, with the exception of + +- `source` +- `destination` +- `pre_exec` +- `post_exec` + +Detailed description of "source" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +`source` describes a `rsync` compatible source (one line only). + +For instance 'backup_user@foreign_host:/home/server/video'. +To use the `rsync` protocol without the `ssh`-tunnel, use +'rsync::USER@HOST/SRC'. For more information have a look at the manpage +of `rsync`(1). + + +Detailed description of "destination" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +`destination` must be a text file containing the destination directory. +`destination` *USED* to be a link to the destination directory in +earlier versions, so do not be confused if you see such examples. + + +Example: +-------------------------------------------------------------------------------- + [11:36] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/destination + /home/nico/backupdir +-------------------------------------------------------------------------------- + + +Detailed description of "remote_host" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +`remote_host` must be a text file containing the destination host. +If this file is existing, you are backing up your data *TO* this host +and *not* to you local host. + +*Warning*: You need to have `ssh` access to the remote host. `rsync` and +`ccollect` will connect to that host via `ssh`. `ccollect` needs the shell +access, because it needs to find out how many backups exist on the remote +host and to be able to delete them. + + +Example: +-------------------------------------------------------------------------------- + [10:17] denkbrett:ccollect-0.7.0% cat conf/sources/remote1/remote_host + home.schottelius.org +-------------------------------------------------------------------------------- + +It may contain all the ssh-specific values like 'myuser@yourhost.ch'. + + +Detailed description of "verbose" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +`verbose` tells `ccollect` that the log should contain verbose messages. + +If this file exists in the source specification *-v* will be passed to `rsync`. + + +Example: +-------------------------------------------------------------------------------- + [11:35] zaphodbeeblebrox:ccollect-0.2% touch conf/sources/testsource1/verbose +-------------------------------------------------------------------------------- + + +Detailed description of "very_verbose" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +`very_verbose` tells `ccollect` that it should log very verbosely. + +If this file exists in the source specification *-v* will be passed to +`rsync`, `rm` and `mkdir`. + + +Example: +-------------------------------------------------------------------------------- + [23:67] nohost:~% touch conf/sources/testsource1/very_verbose +-------------------------------------------------------------------------------- + + +Detailed description of "summary" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +If you create the file `summary` in the source definition, +`ccollect` will present you a nice summary at the end. + +------------------------------------------------------------------------------- +backup:~# touch /etc/ccollect/sources/root/summary +backup:~# ccollect.sh werktags root +==> ccollect.sh: Beginning backup using interval werktags <== +[root] Beginning to backup this source ... +[root] Currently 3 backup(s) exist, total keeping 50 backup(s). +[root] Beginning to backup, this may take some time... +[root] Hard linking... +[root] Transferring files... +[root] +[root] Number of files: 84183 +[root] Number of files transferred: 32 +[root] Total file size: 26234080536 bytes +[root] Total transferred file size: 9988252 bytes +[root] Literal data: 9988252 bytes +[root] Matched data: 0 bytes +[root] File list size: 3016771 +[root] File list generation time: 1.786 seconds +[root] File list transfer time: 0.000 seconds +[root] Total bytes sent: 13009119 +[root] Total bytes received: 2152 +[root] +[root] sent 13009119 bytes received 2152 bytes 2891393.56 bytes/sec +[root] total size is 26234080536 speedup is 2016.26 +[root] Successfully finished backup. +==> Finished ccollect.sh <== +------------------------------------------------------------------------------- + +You could also combine it with `verbose` or `very_verbose`, but these +already print some statistics (though not all / the same as presented by +`summary`). + + +Detailed description of "exclude" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +`exclude` specifies a list of paths to exclude. The entries are seperated by a newline (\n). + + +Example: +-------------------------------------------------------------------------------- + [11:35] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/exclude + openvpn-2.0.1.tar.gz + nicht_reinnehmen + etwas mit leerzeichenli + something with spaces is not a problem +-------------------------------------------------------------------------------- + + +Detailed description of "intervals/" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +When you create the subdirectory `intervals/` in your source configuration +directory, you can specify individiual intervals for this specific source. +Each file in this directory describes an interval. + + +Example: +-------------------------------------------------------------------------------- + [11:37] zaphodbeeblebrox:ccollect-0.2% ls -l conf/sources/testsource2/intervals/ + insgesamt 8 + -rw-r--r-- 1 nico users 2 2005-12-07 17:38 daily + -rw-r--r-- 1 nico users 3 2005-12-14 11:33 yearly + [11:37] zaphodbeeblebrox:ccollect-0.2% cat conf/sources/testsource2/intervals/* + 5 + 20 +-------------------------------------------------------------------------------- + + +Detailled description of "rsync_options" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +When you create the file `rsync_options` in your source configuration, +all the parameters in this file will be passed to rsync. This +way you can pass additional options to rsync. For instance you can tell rsync +to show progress ("--progress"), or which -password-file ("--password-file") +to use for automatic backup over the rsync-protocol. + + +Example: +-------------------------------------------------------------------------------- + [23:42] hydrogenium:ccollect-0.2% cat conf/sources/test_rsync/rsync_options + --password-file=/home/user/backup/protected_password_file +-------------------------------------------------------------------------------- + + +Detailled description of "pre_exec" and "post_exec" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +When you create `pre_exec` and / or `post_exec` in your source +configuration, `ccollect` will execute this command before and +respectively after doing the backup for *this specific* source. +If you want to have pre-/post-exec before and after *all* +backups, see above for general configuration. + + +Example: +-------------------------------------------------------------------------------- +[13:09] hydrogenium:ccollect-0.3% cat conf/sources/with_exec/pre_exec +#!/bin/sh + +# Show whats free before +df -h +[13:09] hydrogenium:ccollect-0.3% cat conf/sources/with_exec/post_exec +#!/bin/sh + +# Show whats free after +df -h +-------------------------------------------------------------------------------- + + +Detailed description of "delete_incomplete" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +If you create the file `delete_incomplete` in a source specification directory, +`ccollect` will look for incomplete backups (when the whole `ccollect` process +was interrupted) and remove them. Without this file `ccollect` will only warn +the user. + +Detailed description of "rsync_failure_codes" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +If you have the file `rsync_failure_codes` in your source configuration +directory, it should contain a newline-separated list of numbers representing +rsync exit codes. If rsync exits with any code in this list, a marker will +be left in the destination directory indicating failure of this backup. If +you have enabled delete_incomplete, then this backup will be deleted during +the next ccollect run on the same interval. + +Detailed description of "mtime" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +By default, ccollect.sh chooses the most recent backup directory for cloning or +the oldest for deletion based on the directory's last change time (ctime). +With this option, the sorting is done based on modification time (mtime). With +this version of ccollect, the ctime and mtime of your backups will normally +be the same and this option has no effect. However, if you, for example, move +your backups to another hard disk using cp -a or rsync -a, you should use this +option because the ctimes are not preserved during such operations. + +If you have any backups in your repository made with ccollect version 0.7.1 or +earlier, do not use this option. + +Detailed description of "quiet_if_down" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +By default, ccollect.sh emits a series of error messages if a source is not +connectable. With this option enabled, ccollect still reports that the +source is not connectable but the associated error messages generated by +rsync or ssh are suppressed. You may want to use this option for sources, +like notebook PCs, that are often disconnected. + + +Hints +----- + +Smart logging +~~~~~~~~~~~~~ +Since ccollect-0.6.1 you can use the ccollect-logwrapper.sh(1) for logging. +You call it the same way you call ccollect.sh and it will create a +logfile containing the output of ccollect.sh. For more information look +at the manpage 'ccollect-logwrapper'. The following is an example running +ccollect-logwrapper.sh: + +-------------------------------------------------------------------------------- +u0219 ~ # ~chdscni9/ccollect-logwrapper.sh daily u0160.nshq.ch.netstream.com +ccollect-logwrapper.sh (11722): Starting with arguments: daily u0160.nshq.ch.netstream.com +ccollect-logwrapper.sh (11722): Finished. +-------------------------------------------------------------------------------- + + +Using a different ssh port +~~~~~~~~~~~~~~~~~~~~~~~~~~ +Mostly easy is to use your ~/.ssh/config file: + +-------------------------------------------------------------------------------- +host mx2.schottelius.org + Port 2342 +-------------------------------------------------------------------------------- + +If you only use that port for backup only and normally want to use another port, +you can add 'HostName' and "HostKeyAlias" (if you also have different +keys on the different ports): + +-------------------------------------------------------------------------------- +Host hhydrogenium + Hostname bruehe.schottelius.org + Port 666 + HostKeyAlias hydrogenium + +Host bruehe + Hostname bruehe.schottelius.org + Port 22 + HostKeyAlias bruehe.schottelius.org +-------------------------------------------------------------------------------- + + +Using source names or interval in pre_/post_exec scripts +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The pre-/post_exec scripts can access some internal variables from `ccollect`: + +- INTERVAL: The interval specified on the command line +- no_sources: number of sources +- source_$NUM: the name of the source +- name: the name of the currently being backuped source (not available for + generic pre_exec script) + + +Using rsync protocol without ssh +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When you have a computer with little computing power, it may be useful to use +rsync without ssh, directly using the rsync protocol +(specify 'user@host::share' in `source`). You may wish to use +`rsync_options` to specify a password file to use for automatic backup. + + +Example: +-------------------------------------------------------------------------------- +backup:~# cat /etc/ccollect/sources/sample.backup.host.org/source +backup@webserver::backup-share + +backup:~# cat /etc/ccollect/sources/sample.backup.host.org/rsync_options +--password-file=/etc/ccollect/sources/sample.backup.host.org/rsync_password + +backup:~# cat /etc/ccollect/sources/sample.backup.host.org/rsync_password +this_is_the_rsync_password +-------------------------------------------------------------------------------- +This hint was reported by Daniel Aubry. + + +Not excluding top-level directories +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When you exclude "/proc" or "/mnt" from your backup, you may run into +trouble when you restore your backup. When you use "/proc/\*" or "/mnt/\*" +instead, `ccollect` will backup empty directories. + + +[NOTE] +=========================================== +When those directories contain hidden files +(those beginning with a dot (*.*)), +they will still be transferred! +=========================================== +This hint was reported by Marcus Wagner. + + +Re-using already created rsync-backups +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +If you used `rsync` directly before you use `ccollect`, you can +use this old backup as initial backup for `ccollect`: You +simply move it into a directory below the destination directory +and name it "'interval'.0". + + +Example: +------------------------------------------------------------------------------- +backup:/home/backup/web1# ls +bin dev etc initrd lost+found mnt root srv usr vmlinuz +boot doc home lib media opt sbin tmp var vmlinuz.old + +backup:/home/backup/web1# mkdir daily.0 + +# ignore error about copying to itself +backup:/home/backup/web1# mv * daily.0 2>/dev/null + +backup:/home/backup/web1# ls +daily.0 +------------------------------------------------------------------------------- +Now you can use /home/backup/web1 as the `destination` for the backup. + +[NOTE] +=============================================================================== +It does not matter anymore how you name your directory, as `ccollect` uses +the -c option from `ls` to find out which directory to clone from. +=============================================================================== + + +[NOTE] +=============================================================================== +Older versions (pre 0.6, iirc) had a problem, if you named the first backup +something like "daily.initial". It was needed to use the "*0*" (or some +number that is lower than the current year) as extension. `ccollect` +used `sort` to find the latest backup. `ccollect` itself uses +'interval.YEARMONTHDAY-HOURMINUTE.PID'. This notation was *always* before +"daily.initial", as numbers are earlier in the list +which is produced by `sort`. So, if you had a directory named "daily.initial", +`ccollect` always diffed against this backup and transfered and deleted +files which where deleted in previous backups. This means you simply +wasted resources, but your backup had beer complete anyway. +=============================================================================== + + +Using pre_/post_exec +~~~~~~~~~~~~~~~~~~~~ +Your pre_/post_exec script does not need to be a script, you can also +use a link to + + - an existing program + - an already written script + +The only requirement is that it is executable. + + +Using source specific interval definitions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When you are backing up multiple hosts via cron each night, it may be +a problem that host "big_server" may only have 4 daily backups, because +otherwise its backup device will be full. But for all other hosts +you want to keep 20 daily backups. In this case you would create +`/etc/ccollect/default/intervals/daily` containing "20" and +`/etc/ccollect/sources/big_server/intervals/daily` containing "4". + +Source specific intervals always overwrite the default values. +If you have to specify it individually for every host, because +of different requirements, you can even omit creating +`/etc/ccollect/default/intervals/daily`. + + +Comparing backups +~~~~~~~~~~~~~~~~~ +If you want to see what changed between two backups, you can use +`rsync` directly: + +-------------------------------------------------------------------------------- +[12:00] u0255:ddba034.netstream.ch# rsync -n -a --delete --stats --progress daily.20080324-0313.17841/ daily.20080325-0313.31148/ +-------------------------------------------------------------------------------- +This results in a listing of changes. Because we pass -n to rsync no transfer +is made (i.e. report only mode)" + +This hint was reported by Daniel Aubry. + + +Testing for host reachabilty +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +If you want to test whether the host you try to backup is reachable, you can use +the following script as source specific pre-exec: + +-------------------------------------------------------------------------------- +#!/bin/sh +# ping -c1 -q `cat "/etc/ccollect/sources/$name/source" | cut -d"@" -f2 | cut -d":" -f1` +-------------------------------------------------------------------------------- + +This prevents the deletion of old backups, if the host is not reachable. + +This hint was reported by Daniel Aubry. + + +Easy check for errors +~~~~~~~~~~~~~~~~~~~~~ +If you want to see whether there have been any errors while doing the backup, +you can run `ccollect` together with `ccollect_analyse_logs.sh`: +-------------------------------------------------------------------------------- +$ ccollect | ccollect_analyse_logs.sh e +-------------------------------------------------------------------------------- + + +F.A.Q. +------ + +What happens if one backup is broken or empty? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Let us assume that one backup failed (connection broke or the source +hard disk had some failures). Therefore we've got one incomplete backup in our history. + +`ccollect` will transfer the missing files the next time you use it. +This leads to + + - more transferred files + - much greater disk space usage, as no hardlinks can be used + +If the whole `ccollect` process was interrupted, `ccollect` (since 0.6) can +detect that and remove the incomplete backups, so you can clone from a complete +backup instead + + +When backing up from localhost the destination is also included. Is this a bug? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +No. `ccollect` passes your source definition directly to `rsync`. It +does not try to analyze it. So it actually does not know if a source +comes from local harddisk or from a remote server. And it does not want +to. When you backup from the local harddisk (which is perhaps not +even a good idea when thinking of security), add the `destination` +to 'source/exclude'. (Daniel Aubry reported this problem) + + +Why does ccollect say "Permission denied" with my pre-/postexec script? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The most common error is that you have not given your script the correct +permissions. Try `chmod 0755 /etc/ccollect/sources/'yoursource'/*_exec``. + + +Why does the backup job fail when part of the source is a link? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When a part of your path you specified in the source is a +(symbolic, hard links are not possible for directories) link, +the backup *must* fail. + +First of all, let us have a look at how it looks like: + +------------------------------------------------------------------------------- +==> ccollect 0.4: Beginning backup using interval taeglich <== +[testsource] Sa Apr 29 00:01:55 CEST 2006 Beginning to backup +[testsource] Currently 0 backup(s) exist(s), total keeping 10 backup(s). +[testsource] Beginning to backup, this may take some time... +[testsource] Creating /etc/ccollect/sources/testsource/destination/taeglich.2006-04-29-0001.3874 ... +[testsource] Sa Apr 29 00:01:55 CEST 2006 Transferring files... +[testsource] rsync: recv_generator: mkdir "/etc/ccollect/sources/testsource/destination/taeglich.2006-04-29-0001.3874/home/user/nico/projekte/ccollect" failed: No such file or directory (2) +[testsource] rsync: stat "/etc/ccollect/sources/testsource/destination/taeglich.2006-04-29-0001.3874/home/user/nico/projekte/ccollect" failed: No such file or directory (2) +[...] +------------------------------------------------------------------------------- + +So what is the problem? It is very obvious when you look deeper into it: + +------------------------------------------------------------------------------- +% cat /etc/ccollect/sources/testsource/source +/home/user/nico/projekte/ccollect/ccollect-0.4 +% ls -l /home/user/nico/projekte +lrwxrwxrwx 1 nico nico 29 2005-12-02 23:28 /home/user/nico/projekte -> oeffentlich/computer/projekte +% ls -l /etc/ccollect/sources/testsource/destination/taeglich.2006-04-29-0001.3874/home/user/nico +lrwxrwxrwx 1 nico nico 29 2006-04-29 00:01 projekte -> oeffentlich/computer/projekte +------------------------------------------------------------------------------- + +`rsync` creates the directory structure before it creates the symbolic link. +This link now links to something not reachable (dead link). It is +impossible to create subdirectories under the broken link. + +In conclusion you cannot use paths with a linked part. + +However, you can backup directories containing symbolic links +(in this case you could backup /home/user/nico, which contains +/home/user/nico/projekte and oeffentlich/computer/projekte). + + +How can I prevent missing the right time to enter my password? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +As `ccollect` first deletes the old backups, it may take some time +until `rsync` requests the password for the `ssh` session from you. + +The easiest way not to miss that point is running `ccollect` in `screen`, +which has the ability to monitor the output for activity. So as soon as +your screen beeps, after `ccollect` began to remove the last directory, +you can enter your password (have a look at screen(1), especially "C-a M" +and "C-a _", for more information). + + +Examples +-------- + +A backup host configuration from scratch +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +-------------------------------------------------------------------------------- +srwali01:~# mkdir /etc/ccollect +srwali01:~# mkdir -p /etc/ccollect/defaults/intervals/ +srwali01:~# echo 28 > /etc/ccollect/defaults/intervals/taeglich +srwali01:~# echo 52 > /etc/ccollect/defaults/intervals/woechentlich +srwali01:~# cd /etc/ccollect/ +srwali01:/etc/ccollect# mkdir sources +srwali01:/etc/ccollect# cd sources/ +srwali01:/etc/ccollect/sources# ls +srwali01:/etc/ccollect/sources# mkdir local-root +srwali01:/etc/ccollect/sources# cd local-root/ +srwali01:/etc/ccollect/sources/local-root# echo / > source +srwali01:/etc/ccollect/sources/local-root# cat > exclude << EOF +> /proc +> /sys +> /mnt +> EOF +srwali01:/etc/ccollect/sources/local-root# echo /mnt/hdbackup/local-root > destination +srwali01:/etc/ccollect/sources/local-root# mkdir /mnt/hdbackup/local-root +srwali01:/etc/ccollect/sources/local-root# ccollect.sh taeglich local-root +/o> ccollect.sh: Beginning backup using interval taeglich +/=> Beginning to backup "local-root" ... +|-> 0 backup(s) already exist, keeping 28 backup(s). +-------------------------------------------------------------------------------- + + +After that, I added some more sources: +-------------------------------------------------------------------------------- +srwali01:~# cd /etc/ccollect/sources +srwali01:/etc/ccollect/sources# mkdir windos-wl6 +srwali01:/etc/ccollect/sources# cd windos-wl6/ +srwali01:/etc/ccollect/sources/windos-wl6# echo /mnt/win/SYS/WL6 > source +srwali01:/etc/ccollect/sources/windos-wl6# echo /mnt/hdbackup/wl6 > destination +srwali01:/etc/ccollect/sources/windos-wl6# mkdir /mnt/hdbackup/wl6 +srwali01:/etc/ccollect/sources/windos-wl6# cd .. +srwali01:/etc/ccollect/sources# mkdir windos-daten +srwali01:/etc/ccollect/sources/windos-daten# echo /mnt/win/Daten > source +srwali01:/etc/ccollect/sources/windos-daten# echo /mnt/hdbackup/windos-daten > destination +srwali01:/etc/ccollect/sources/windos-daten# mkdir /mnt/hdbackup/windos-daten + +# Now add some remote source +srwali01:/etc/ccollect/sources/windos-daten# cd .. +srwali01:/etc/ccollect/sources# mkdir srwali03 +srwali01:/etc/ccollect/sources# cd srwali03/ +srwali01:/etc/ccollect/sources/srwali03# cat > exclude << EOF +> /proc +> /sys +> /mnt +> /home +> EOF +srwali01:/etc/ccollect/sources/srwali03# echo 'root@10.103.2.3:/' > source +srwali01:/etc/ccollect/sources/srwali03# echo /mnt/hdbackup/srwali03 > destination +srwali01:/etc/ccollect/sources/srwali03# mkdir /mnt/hdbackup/srwali03 +-------------------------------------------------------------------------------- + + +Using hard-links requires less disk space +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +------------------------------------------------------------------------- +# du (coreutils) 5.2.1 +[10:53] srsyg01:sources% du -sh ~/backupdir +4.6M /home/nico/backupdir +[10:53] srsyg01:sources% du -sh ~/backupdir/* +4.1M /home/nico/backupdir/daily.2005-12-08-10:52.28456 +4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28484 +4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28507 +4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28531 +4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28554 +4.1M /home/nico/backupdir/daily.2005-12-08-10:53.28577 + +srwali01:/etc/ccollect/sources# du -sh /mnt/hdbackup/wl6/ +186M /mnt/hdbackup/wl6/ +srwali01:/etc/ccollect/sources# du -sh /mnt/hdbackup/wl6/* +147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:42.312 +147M /mnt/hdbackup/wl6/taeglich.2005-12-08-14:45.588 +------------------------------------------------------------------------- + +The backup of our main fileserver: +------------------------------------------------------------------------- +backup:~# df -h /home/backup/srsyg01/ +Filesystem Size Used Avail Use% Mounted on +/dev/mapper/backup--01-srsyg01 + 591G 451G 111G 81% /home/backup/srsyg01 +backup:~# du -sh /home/backup/srsyg01/* +432G /home/backup/srsyg01/daily.2006-01-24-01:00.15990 +432G /home/backup/srsyg01/daily.2006-01-26-01:00.30152 +434G /home/backup/srsyg01/daily.2006-01-27-01:00.4596 +435G /home/backup/srsyg01/daily.2006-01-28-01:00.11998 +437G /home/backup/srsyg01/daily.2006-01-29-01:00.19115 +437G /home/backup/srsyg01/daily.2006-01-30-01:00.26405 +438G /home/backup/srsyg01/daily.2006-01-31-01:00.1148 +439G /home/backup/srsyg01/daily.2006-02-01-01:00.8321 +439G /home/backup/srsyg01/daily.2006-02-02-01:00.15383 +439G /home/backup/srsyg01/daily.2006-02-03-01:00.22567 +16K /home/backup/srsyg01/lost+found +backup:~# du --version | head -n1 +du (coreutils) 5.2.1 +------------------------------------------------------------------------- + +Newer versions of du also detect the hardlinks, so we can even compare +the sizes directly with du: +------------------------------------------------------------------------- +[8:16] eiche:~# du --version | head -n 1 +du (GNU coreutils) 5.93 +[8:17] eiche:schwarzesloch# du -slh hydrogenium/* +19G hydrogenium/durcheinander.0 +18G hydrogenium/durcheinander.2006-01-17-00:27.13820 +19G hydrogenium/durcheinander.2006-01-25-23:18.31328 +19G hydrogenium/durcheinander.2006-01-26-00:11.3332 +[8:22] eiche:schwarzesloch# du -sh hydrogenium/* +19G hydrogenium/durcheinander.0 +12G hydrogenium/durcheinander.2006-01-17-00:27.13820 +1.5G hydrogenium/durcheinander.2006-01-25-23:18.31328 +200M hydrogenium/durcheinander.2006-01-26-00:11.3332 +------------------------------------------------------------------------- + +In the second report (without -l) the sizes include the space the inodes of +the hardlinks allocate. + + +A collection of backups on the backup server +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +All the data of my important hosts is backuped to eiche into +/mnt/schwarzesloch/backup: + +------------------------------------------------------------------------- +[9:24] eiche:backup# ls * +creme: +woechentlich.2006-01-26-22:22.4153 woechentlich.2006-02-12-11:48.2461 +woechentlich.2006-01-26-22:23.4180 woechentlich.2006-02-18-23:00.7898 +woechentlich.2006-02-05-02:43.14281 woechentlich.2006-02-25-23:00.13480 +woechentlich.2006-02-06-00:24.15509 woechentlich.2006-03-04-23:00.25439 + +hydrogenium: +durcheinander.2006-01-27-11:16.6391 durcheinander.2006-02-13-01:07.2895 +durcheinander.2006-01-30-19:29.9505 durcheinander.2006-02-17-08:20.6707 +durcheinander.2006-01-30-22:27.9623 durcheinander.2006-02-24-16:24.12461 +durcheinander.2006-02-03-09:52.12885 durcheinander.2006-03-03-19:17.18075 +durcheinander.2006-02-05-23:00.15068 durcheinander.2006-03-17-22:41.5007 + +scice: +woechentlich.2006-02-04-10:32.13766 woechentlich.2006-02-16-23:00.6185 +woechentlich.2006-02-05-23:02.15093 woechentlich.2006-02-23-23:00.11783 +woechentlich.2006-02-06-08:22.15994 woechentlich.2006-03-02-23:00.17346 +woechentlich.2006-02-06-19:40.16321 woechentlich.2006-03-09-23:00.29317 +woechentlich.2006-02-12-11:51.2514 woechentlich.2006-03-16-23:00.4218 +------------------------------------------------------------------------- + +And this incremental backup and the archive are copied to an external +usb harddisk (attention: you *should* really use -H to backup the backup): + +------------------------------------------------------------------------- +[9:23] eiche:backup# df -h +Filesystem Size Used Avail Use% Mounted on +rootfs 14G 8.2G 4.9G 63% / +/dev/root 14G 8.2G 4.9G 63% / +/dev/root 14G 8.2G 4.9G 63% /dev/.static/dev +tmpfs 10M 444K 9.6M 5% /dev +/dev/hdh 29G 3.7M 29G 1% /mnt/datenklo +tmpfs 110M 4.0K 110M 1% /dev/shm +/dev/mapper/nirvana 112G 90G 23G 81% /mnt/datennirvana +/dev/mapper/schwarzes-loch + 230G 144G 86G 63% /mnt/schwarzesloch +/dev/mapper/archiv 38G 20G 19G 52% /mnt/archiv +/dev/mapper/usb-backup + 280G 36M 280G 1% /mnt/usb/backup +[9:24] eiche:backup# cat ~/bin/sync-to-usb +DDIR=/mnt/usb/backup + +rsync -av -H --delete /mnt/schwarzesloch/ "$DDIR/schwarzesloch/" + +rsync -av -H --delete /mnt/archiv/ "$DDIR/archiv/" +------------------------------------------------------------------------- + + +Processes running when doing ccollect -p +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Truncated output from `ps axuwwwf`: + +------------------------------------------------------------------------- + S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily -p ddba034 ddba045 ddba046 ddba047 ddba049 ddna010 ddna011 + S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba034 + S+ 11:40 0:00 | | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba034 + R+ 11:40 23:40 | | | | | \_ rsync -a --delete --numeric-ids --relative --delete-excluded --link-dest=/home/server/backup/ddba034 + S+ 11:40 0:00 | | | | | \_ ssh -l root ddba034.netstream.ch rsync --server --sender -vlogDtprR --numeric-ids . / + S+ 11:41 0:11 | | | | | \_ rsync -a --delete --numeric-ids --relative --delete-excluded --link-dest=/home/server/backup/ddb + S+ 11:40 0:00 | | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba034 + S+ 11:40 0:00 | | | | \_ sed s:^:\[ddba034\] : + S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba045 + S+ 11:40 0:00 | | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba045 + R+ 11:40 0:02 | | | | | \_ rm -rf /etc/ccollect/sources/ddba045/destination/daily.2006-10-19-1807.6934 + S+ 11:40 0:00 | | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba045 + S+ 11:40 0:00 | | | | \_ sed s:^:\[ddba045\] : + S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba046 + S+ 11:40 0:00 | | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba046 + R+ 11:40 0:02 | | | | | \_ rm -rf /etc/ccollect/sources/ddba046/destination/daily.2006-10-19-1810.7072 + S+ 11:40 0:00 | | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba046 + S+ 11:40 0:00 | | | | \_ sed s:^:\[ddba046\] : + S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba047 + S+ 11:40 0:00 | | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba047 + R+ 11:40 0:03 | | | | | \_ rm -rf /etc/ccollect/sources/ddba047/destination/daily.2006-10-19-1816.7268 + S+ 11:40 0:00 | | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba047 + S+ 11:40 0:00 | | | | \_ sed s:^:\[ddba047\] : + S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba049 + S+ 11:40 0:00 | | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba049 + D+ 11:40 0:03 | | | | | \_ rm -rf /etc/ccollect/sources/ddba049/destination/daily.2006-10-19-1821.7504 + S+ 11:40 0:00 | | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddba049 + S+ 11:40 0:00 | | | | \_ sed s:^:\[ddba049\] : + S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddna010 + S+ 11:40 0:00 | | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddna010 + R+ 11:40 0:03 | | | | | \_ rm -rf /etc/ccollect/sources/ddna010/destination/daily.2006-10-19-1805.6849 + S+ 11:40 0:00 | | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddna010 + S+ 11:40 0:00 | | | | \_ sed s:^:\[ddna010\] : + S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddna011 + S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddna011 + R+ 12:08 0:00 | | | | \_ rm -rf /etc/ccollect/sources/ddna011/destination/daily.2006-10-20-1502.7824 + S+ 11:40 0:00 | | | \_ /bin/sh /usr/local/bin/ccollect.sh daily ddna011 + S+ 11:40 0:00 | | | \_ sed s:^:\[ddna011\] : + +------------------------------------------------------------------------- +As you can see, six processes are deleting old backups, while one backup +(ddba034) is already copying data. diff --git a/software/ccollect/ccollect-0.8/doc/changes/0.7.1 b/software/ccollect/ccollect-0.8/doc/changes/0.7.1 new file mode 100644 index 00000000..adf9ff43 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/changes/0.7.1 @@ -0,0 +1,9 @@ +* Added support for global delete_incomplete option +* Updated tools/ccollect_analyse_logs.sh: Added more error strings to find +* Removed use of 'basename': Replaced it with standard variables from cconf +* Updated documentation + * More hints + * Updated remote_host description +* Bugfix in shell artihmetic (Jeroen Bruijning) +* Bugfix: Allow "&" in sourcename (Reported by Tiziano Müller) +* Added ccollect_list_intervals.sh to list intervals with values diff --git a/software/ccollect/ccollect-0.8/doc/changes/0.8 b/software/ccollect/ccollect-0.8/doc/changes/0.8 new file mode 100644 index 00000000..75f9a63b --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/changes/0.8 @@ -0,0 +1,14 @@ +* Introduce consistenst time sorting (John Lawless) +* Check for source connectivity before trying backup (John Lawless) +* Defensive programming patch (John Lawless) +* Some code cleanups (argument parsing, usage) (Nico Schottelius) +* Only consider directories as sources when using -a (Nico Schottelius) +* Fix general parsing problem with -a (Nico Schottelius) +* Fix potential bug when using remote_host, delete_incomplete and ssh (Nico Schottelius) +* Improve removal performance: minimised number of 'rm' calls (Nico Schottelius) +* Support sorting by mtime (John Lawless) +* Improve option handling (John Lawless) +* Add support for quiet operation for dead devices (quiet_if_down) (John Lawless) +* Add smart option parsing, including support for default values (John Lawless) +* Updated and cleaned up documentation (Nico Schottelius) +* Fixed bug "removal of current directory" in ccollect_delete_source.sh (Found by Günter Stöhr, fixed by Nico Schottelius) diff --git a/software/ccollect/ccollect-0.8/doc/changes/pre-0.7.1 b/software/ccollect/ccollect-0.8/doc/changes/pre-0.7.1 new file mode 100644 index 00000000..2c3f5228 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/changes/pre-0.7.1 @@ -0,0 +1,100 @@ +0.6.2 to 0.7.0: + * Added tools/report_success.sh (try it out!) + * Added ccollect_analyse_logs.sh (see ccollect_analyse_logs(1)) + * Add capability to backup to a host + - Also updated documentation + * Changed "destination" format + - Use tools/config-pre-0.7-to-0.7.sh to convert pre 0.7.x configurations + * Renamed all tools to begin with "ccollect_" + * Updated ccollect_add_source.sh and added manpage + * Updated todos + * Changed license to GPLv3 (from GPLv2) + + +0.6.1 to 0.6.2: + * Added analyse-ccollect-logs.sh + * Fixed bug: Removing of backups was broken since update to 0.6 + (forgot to prepend path...) + * Fixed bug: The marker was always deleted, because rsync deleted it. + Create it outside of the backup destination now. + +0.6 to 0.6.1: + * Added check for destination_base in add_ccollect_source.sh + * Added support for -V and --version + * Added ccollect-logwrapper.sh (and a manpage ;-) + * Changed behaviour: ccollect now clones from the latest existing backup, + independent of the interval. This way different intervals do not + diverge. ccollect uses ls -c to determine latest backup. + +0.5.2 to 0.6: + * Always print return code of rsync + * Add much more timing information + * One option per line in rsync_options now (NOT space seperated) + * Added --sparse as default option + * Added management tools (including manpages): + * add_ccollect_source.sh + * delete_ccollect_source.sh + * list_ccollect_intervals.sh + * Cleaned up exit calls (now always cleanly removes temporary files) + * In theory, added pdf documentation (though, was unable to do it with fop) + * Changed license to GPLv3 (from GPLv2) + +0.5.1 to 0.5.2: + * Display correct error code, if rsync returns non-zero + * Unify messages + * Remove some potential quoting problems + +0.5 to 0.5.1: + * Remove always printed debug information + +0.4.3 to 0.5: + * Removed requirement PaX + * Removed requirement bc + +0.4.2 to 0.4.3: + * Display error code of rsync, if non-zero (for further analysis) + * Fix Makefile, so 'make install' works on others OS + * reorder $RSYNC_EXTRA, so it can be overriden by users + +0.4.1 to 0.4.2: + * fixed bug when $CCOLLECT_CONF is relative + * added Quickstart to documentation + +0.4 to 0.4.1: + * updated documentation, fixed some English related problems + * added Texinfo documentation + * added a manpage (English) + * fixed problem with 'make install' (strip was used) + * fixed possible problem with pre_exec beeing executed to late + * fixed small bug in sed expression: using 'source/' made it fail + +0.3.3 to 0.4: + * `pax` (Posix) is now required, `cp -al` (GNU specific) is removed + * "interval" was written with two 'l' (ell), which is wrong in English + * Changed the name of backup directories, removed the colon in the interval + * ccollect will now exit, when preexec returns non-zero + * ccollect now reports when postexec returns non-zero + +0.3.2 to 0.3.3: + * Fix a small bug, which suppressed information when rsync exits non-zero + +0.3.1 to 0.3.2: + * ccollect now prints the start time, end time and duration of the backup + +0.3 to 0.3.1: + * added support for printing a summary + * some cosmetic changes + +0.2 to 0.3: + * added "very_verbose" + * normal "verbose" is now less verbose + * added general 'pre_exec' and 'post_exec' support + * added source specfifc 'pre_exec' and 'post_exec' support + +0.1 to 0.2: + * Added plausibility check + * Updated and made documentation readable + * implemented verbose option + * Fixed double exclude parameter bug + * Added much better documentation (asciidoc) + * added rsync extra parameter option diff --git a/software/ccollect/ccollect-0.8/doc/gpl3-header b/software/ccollect/ccollect-0.8/doc/gpl3-header new file mode 100644 index 00000000..5a1663dc --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/gpl3-header @@ -0,0 +1,19 @@ +#!/bin/sh +# +# 2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# diff --git a/software/ccollect/ccollect-0.8/doc/logwrapper.text b/software/ccollect/ccollect-0.8/doc/logwrapper.text new file mode 100644 index 00000000..ec12db48 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/logwrapper.text @@ -0,0 +1,29 @@ +ccollect-logwrapper: Logging backup output +=========================================== +Nico Schottelius +0.1, for ccollect-logwrapper 0.1, Initial Version from 2007-06-08 +:Author Initials: NS + +This wrapper makes it easy to have logs of ccollect output. + + +Introduction +------------ +/etc/ccollect/logwrapper/ (also uses $CCOLLECT_CONF). + + + pipe: will pipe to a program + staticfile: link to a file + datefile: + contains a string that is passed to date that returns + dynamicfile: + is a program, that returns some string that we use as + a filename + syslog: + syslog-facility + syslog-level + only-stderr: + omit stdout output + +The logger will output to which destinations it logs and with which +parameters it was started. diff --git a/software/ccollect/ccollect-0.8/doc/man/ccollect.text b/software/ccollect/ccollect-0.8/doc/man/ccollect.text new file mode 100644 index 00000000..84e95bb0 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/man/ccollect.text @@ -0,0 +1,61 @@ +ccollect(1) +=========== +Nico Schottelius + + +NAME +---- +ccollect - (pseudo) incremental backup with different exclude lists using hardlinks and rsync + + +SYNOPSIS +-------- +'ccollect.sh' [args] + + +DESCRIPTION +----------- +`ccollect` is a backup utility written in the sh-scripting language. +It does not depend on a specific shell, only `/bin/sh` needs to be +bourne shell compatibel (like 'dash', 'ksh', 'zsh', 'bash', ...). + +For more information refer to the manual titled +"ccollect - Installing, Configuring and Using" (available as text (asciidoc), +texinfo or html). + + +OPTIONS +------- +-h, --help:: + Show the help screen + +-p, --parallel:: + Parallelise backup processes + +-a, --all:: + Backup all sources specified in /etc/ccollect/sources + +-v, --verbose:: + Be very verbose (uses set -x). + + +SEE ALSO +-------- +ccollect_add_source(1), ccollect_analyse_logs(1), ccollect_logwrapper(1) +ccollect_delete_source(1), ccollect_list_intervals(1) + + +AUTHOR +------ +Nico Schottelius + + +RESOURCES +--------- +Main web site: http://www.nico.schottelius.org/software/ccollect/[] + + +COPYING +------- +Copyright \(C) 2006-2008 Nico Schottelius. Free use of this software is +granted under the terms of the GNU General Public License Version 3 (GPLv3). diff --git a/software/ccollect/ccollect-0.8/doc/man/ccollect_add_source.text b/software/ccollect/ccollect-0.8/doc/man/ccollect_add_source.text new file mode 100644 index 00000000..e3b9f83c --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/man/ccollect_add_source.text @@ -0,0 +1,75 @@ +ccollect_add_source(1) +====================== +Nico Schottelius + + +NAME +---- +ccollect_add_source - create new source for ccollect(1) + + +SYNOPSIS +-------- + +'ccollect_add_source.sh' + + +DESCRIPTION +----------- +ccollect_add_source.sh creates a new backup source for use with ccollect(1). +It copies the files from to the source directory with the hostname below +'$CCOLLECT_CONF/sources'. It is designed to run on a backup server to create +new directories for new hosts. + + +FILES +----- +$CCOLLECT_CONF/defaults/sources:: + Main configuration directory. $CCOLLECT_CONF is '/etc/ccollect', if unset. + All the following files reside below this directory. + +exclude:: +summary:: +intervals:: +pre_exec:: +post_exec:: +rsync_options:: +verbose:: +very_verbose:: + Those are the standard configuration files known by ccollect(1). + If the file exist it will be copied to the newly created source. + Directories ('intervals') are copied recursively. + +destination_base:: + A link to the directory where to store the backups. Below this directory + `ccollect_add_source.sh` will create a directory with the hostname you + specified on the command line. A common valua for `destination_base` is + '/home/server/backup'. + +source_prefix:: +source_postfix:: + `source_prefix` is put before the hostname, `source_postfix` is appended + after it. A common value for `source_prefix` maybe 'root@' and ':/' + for `source_postfix`. + + +SEE ALSO +-------- +ccollect(1), ccollect_analyse_logs.sh, ccollect_delete_source(1), +ccollect_list_intervals(1), ccollect_logwrapper(1), + + +AUTHOR +------ +Nico Schottelius + + +RESOURCES +--------- +Main web site: http://www.nico.schottelius.org/software/ccollect/[] + + +COPYING +------- +Copyright \(C) 2007-2008 Nico Schottelius. Free use of this software is +granted under the terms of the GNU General Public License Version 3 (GPLv3). diff --git a/software/ccollect/ccollect-0.8/doc/man/ccollect_analyse_logs.text b/software/ccollect/ccollect-0.8/doc/man/ccollect_analyse_logs.text new file mode 100644 index 00000000..659ddbf4 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/man/ccollect_analyse_logs.text @@ -0,0 +1,56 @@ +ccollect_analyse_logs(1) +======================== +Nico Schottelius + + +NAME +---- +ccollect_analyse_logs - analyse logs produced by ccollect(1) + + +SYNOPSIS +-------- + +'ccollect_analyse_logs.sh' [iwe] + + +DESCRIPTION +----------- +ccollect_analyse_logs.sh reads the logfiles from stdin. You have to specify +at least one of the three loglevels (*i*nformational, *w*arning, *e*rror). Any +combination of them is allowed. + + +FILES +----- +ccollect log files:: + Are read from stdin + +EXAMPLES +-------- +cat /var/log/ccollect/single/* | ccollect_analyse_logs.sh iw:: + Displays warnings and informational parts +ccollect_analyse_logs.sh iw < /var/log/ccollect/all_together:: + Displays only error messages (useful for the morning mail) + + +SEE ALSO +-------- +ccollect(1), ccollect_add_source.sh, ccollect_delete_source(1), +ccollect_list_intervals(1), ccollect_logwrapper(1), + + +AUTHOR +------ +Nico Schottelius + + +RESOURCES +--------- +Main web site: http://www.nico.schottelius.org/software/ccollect/[] + + +COPYING +------- +Copyright \(C) 2007-2008 Nico Schottelius. Free use of this software is +granted under the terms of the GNU General Public License Version 3 (GPLv3). diff --git a/software/ccollect/ccollect-0.8/doc/man/ccollect_delete_source.text b/software/ccollect/ccollect-0.8/doc/man/ccollect_delete_source.text new file mode 100644 index 00000000..2b7f3000 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/man/ccollect_delete_source.text @@ -0,0 +1,57 @@ +ccollect_delete_source(1) +========================= +Nico Schottelius + + +NAME +---- +ccollect_delete_source - delete sources from ccollect(1) + + +SYNOPSIS +-------- +'ccollect_delete_source.sh' [-d] [-f] + + +DESCRIPTION +----------- +ccollect_delete_source.sh deletes backup sources from ccollect(1) and optional +also the backups created for that source. + + +OPTIONS +------- +-d: + Delete also the destination directory. `add_ccollect_source.sh` will change + to the source/'name'/destination directory, get the absolute name and delete + it recursively. +-f: + Force deletion. Do not ask. Very handy for people who know what they do. + Very dangerous for everyone else. + + +FILES +----- +$CCOLLECT_CONF/sources:: + Directory containing the sources. $CCOLLECT_CONF is '/etc/ccollect', if unset. + +SEE ALSO +-------- +ccollect(1), ccollect_add_source(1), ccollect_add_source(1), +ccollect_logwrapper(1), ccollect_list_intervals(1) + + +AUTHOR +------ +Nico Schottelius + + +RESOURCES +--------- +Main web site: http://www.nico.schottelius.org/software/ccollect/[] + + +COPYING +------- +Copyright \(C) 2007-2008 Nico Schottelius. Free use of this software is +granted under the terms of the GNU General Public License Version 3 (GPLv3). diff --git a/software/ccollect/ccollect-0.8/doc/man/ccollect_list_intervals.text b/software/ccollect/ccollect-0.8/doc/man/ccollect_list_intervals.text new file mode 100644 index 00000000..9df4121b --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/man/ccollect_list_intervals.text @@ -0,0 +1,48 @@ +ccollect_list_intervals(1) +========================== +Nico Schottelius + + +NAME +---- +ccollect_list_intervals - list available intervals from ccollect(1) + + +SYNOPSIS +-------- +'ccollect_list_intervals.sh' + + +DESCRIPTION +----------- +ccollect_list_intervals.sh shows intervals specified in the configuration +for ccollect(1). It displays the name of each interval, followed by a colon +followed by the number backups to keep. + + +FILES +----- +$CCOLLECT_CONF/intervals:: + Directory containing the intervals. $CCOLLECT_CONF is '/etc/ccollect', if unset. + + +SEE ALSO +-------- +collect(1), ccollect_add_source(1), ccollect_analyse_logs(1), +ccollect_delete_source(1), ccollect_logwrapper(1) + + +AUTHOR +------ +Nico Schottelius + + +RESOURCES +--------- +Main web site: http://www.nico.schottelius.org/software/ccollect/[] + + +COPYING +------- +Copyright \(C) 2007-2008 Nico Schottelius. Free use of this software is +granted under the terms of the GNU General Public License Version 3 (GPLv3). diff --git a/software/ccollect/ccollect-0.8/doc/man/ccollect_logwrapper.text b/software/ccollect/ccollect-0.8/doc/man/ccollect_logwrapper.text new file mode 100644 index 00000000..8084c106 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/man/ccollect_logwrapper.text @@ -0,0 +1,56 @@ +ccollect_logwrapper(1) +====================== +Nico Schottelius + + +NAME +---- +ccollect_logwrapper - start ccollect(1) and create a unique logfile + + +SYNOPSIS +-------- +'ccollect_logwrapper.sh' + + +DESCRIPTION +----------- +ccollect_logwrapper.sh creates a unique logfile below +'$CCOLLECT_CONF/logwrapper' and redirects ccollect(1) output +(stdout and stderr) to it. + + +OPTIONS +------- +Options are passed directly to ccollect(1). + + +FILES +----- +$CCOLLECT_CONF/logwrapper:: + Directory containing the configuration. $CCOLLECT_CONF is '/etc/ccollect', if unset. + +$CCOLLECT_CONF/logwrapper/destination:: + Link to the destination directory for the logfiles + + +SEE ALSO +-------- +ccollect(1), ccollect_add_source(1), ccollect_analyse_logs(1), +ccollect_delete_source(1), ccollect_list_intervals(1) + + +AUTHOR +------ +Nico Schottelius + + +RESOURCES +--------- +Main web site: http://www.nico.schottelius.org/software/ccollect/[] + + +COPYING +------- +Copyright \(C) 2007-2008 Nico Schottelius. Free use of this software is +granted under the terms of the GNU General Public License Version 3 (GPLv3). diff --git a/software/ccollect/ccollect-0.8/doc/release-checklist b/software/ccollect/ccollect-0.8/doc/release-checklist new file mode 100644 index 00000000..7cba30cd --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/release-checklist @@ -0,0 +1,9 @@ +* Change version and date in ccollect.sh +* Change version in documentation/ccollect.text +* Regenerate documentation +* Create tarball +* Transfer to home.schottelius.org +* Extract files +* Update website +* Announce on freshmeat +* Announce on announce@ diff --git a/software/ccollect/ccollect-0.8/doc/todo/0.5.2 b/software/ccollect/ccollect-0.8/doc/todo/0.5.2 new file mode 100644 index 00000000..0c1c2250 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/todo/0.5.2 @@ -0,0 +1,4 @@ +x Fix $? problem +x Check last dir searching +x Check count generation +x Check general functionality (remove testing) diff --git a/software/ccollect/ccollect-0.8/doc/todo/0.5.3 b/software/ccollect/ccollect-0.8/doc/todo/0.5.3 new file mode 100644 index 00000000..00195812 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/todo/0.5.3 @@ -0,0 +1,34 @@ +Done: +==> screenshot +u0219 zrha166.netstream.ch # ~chdscni9/ccollect.sh weekly zrha166.netstream.ch +2007-08-16-21:49:44: ccollect 0.6: Beginning backup using interval weekly +[zrha166.netstream.ch] 2007-08-16-21:49:44: Beginning to backup +[zrha166.netstream.ch] 2007-08-16-21:49:45: Existing backups: 0 Total keeping backups: 8 +[zrha166.netstream.ch] 2007-08-16-21:49:45: Did not find existing backups for interval weekly. +[zrha166.netstream.ch] 2007-08-16-21:49:45: Using backup from daily. +[zrha166.netstream.ch] 2007-08-16-21:49:45: Beginning to backup, this may take some time... +[zrha166.netstream.ch] 2007-08-16-21:49:45: Creating /etc/ccollect/sources/zrha166.netstream.ch/destination/weekly.20070816-2149.22188 ... +[zrha166.netstream.ch] 2007-08-16-21:49:45: Transferring files... + +- beep-after-delete-hack? + -> tonnerre / #cLinux + +- replace nico-linux-ccollect with nico-ccollect + * ccollect is not Linux specific + +- remove exit-calls + * will leave behind unused temporary file! + * use _exit_err + +X join todos + +- fix possible quoting problems + * rsync_extra (redefine format) + * exclude + * other + * create _rsync, filtering args, creating new $@ + +- check and export variables for use in scripts! + +Contact Julian later: + * integrate updated german documentation diff --git a/software/ccollect/ccollect-0.8/doc/todo/0.5.3.tonnerre b/software/ccollect/ccollect-0.8/doc/todo/0.5.3.tonnerre new file mode 100644 index 00000000..4e7b7db5 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/todo/0.5.3.tonnerre @@ -0,0 +1,11 @@ +NetBSD/i386,amd64,sparc,sparc64 + +13:13 < Tonnerre> telmich, die kleine wä, 2 Variablen aus $(CCOLLECT) zu + machen +13:13 < Tonnerre> telmich, eine fü Sourcepfad und eine fü +Destinationfpad +13:13 < Tonnerre> pfa +13:13 < Tonnerre> d +13:14 < Tonnerre> telmich, die gröre wä die $() durch ${} zu ersetzen, so +dass der Kram auch mit !GNU-Make geht + diff --git a/software/ccollect/ccollect-0.8/doc/todo/0.6 b/software/ccollect/ccollect-0.8/doc/todo/0.6 new file mode 100644 index 00000000..ee8e97fd --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/todo/0.6 @@ -0,0 +1,63 @@ +not did, did not remember why I wanted to do that: + +- do sed-time check: + +1.2. replace sed? +compare timing: +_echo () { echo "$name $msg" } +and create +_techo () { echo "$timestamp $name $msg" } +perhaps create +_eecho () { _techo "ERROR $msg" } +? + + + +done: +add note for + 09:24 < telmich> Obri: ~/.ssh/config! + +- implement use of different intervals as source for cloning + * use 'woechentlich' if available and no backup exists for 'daily' + * add 'prefer_latest' to prefer different interval, that's newer than + ours + * or add 'prefer_same_interval' instead? +- implement detection of partial backups +3. detect unfinished backups +sven / markierung + - wie seht der Marker aus? + -> .ccollect-YEAR-MM-DD.HHmm.pid (like the directory) + --> assume incomplete, when we did not finish. + --> means it's complete,when rsync was successful +---> partial implemented in 0.5.2 (commented out) +- do not check by name, but by time + * is ls -t posix? + * also check: -p -1 +- ccollect Zeitausgabe verbessern + - Wofuer wie lange gebraucht + * rm + * klonen (gibt es so ja nicht) + Wenn Summary angegeben ist am Ende ausgeben +- do we want rsync -H by default? + * no: takes much more memory +ssh / port change: + change rsync_extra format +- Variables: + source_$n + no_sources + name +- changed naming of sources: + YYYYMMDD-HHMM.PID (better readable) + => source / backup converter! => not needed! +config: + set destination-base + /etc/ccollect/defaults/source_config + +Documentation: +- hint: backuping backup +- update documentation: + - german doc? + - exit pre/post exec -> error codes (after implementation!) (in 0.4) + - write about fast changing fs +- delete_incomplete + diff --git a/software/ccollect/ccollect-0.8/doc/todo/0.6.1 b/software/ccollect/ccollect-0.8/doc/todo/0.6.1 new file mode 100644 index 00000000..99a17eba --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/todo/0.6.1 @@ -0,0 +1,23 @@ +- Add filter support + * filter +- add source/ignore_failed_pre_exec +- add source/ignore_failed_post_exec +- do not delete_incomplete, when there's only one backup left! +- .ccollect-marker is deleted by rsync at the beginning! + - fix marking +- add logwrapper +- add loganalyser + speedup is + error codes + vanished files (see netstream)!!! + +Done: +- Improve finding backup from another interval: + o strip of interval name + o run sort -n + o use the last entry +- add --version, -V + +not needed: +- think about 'prefer_same_interval' / 'prefer_latest' + diff --git a/software/ccollect/ccollect-0.8/doc/todo/0.6.2 b/software/ccollect/ccollect-0.8/doc/todo/0.6.2 new file mode 100644 index 00000000..05798ffd --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/todo/0.6.2 @@ -0,0 +1 @@ +- fix delete_incomplete marker diff --git a/software/ccollect/ccollect-0.8/doc/todo/0.7.1 b/software/ccollect/ccollect-0.8/doc/todo/0.7.1 new file mode 100644 index 00000000..49df1545 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/todo/0.7.1 @@ -0,0 +1,35 @@ + + +-------------------------------------------------------------------------------- +done: +- tools in makefile +14:15 also alle tools/ccollect_* +14:15 mach mal n besseres Makefile :) +14:15 hatte die extra deswegen umbenannt +14:15 nehme ich mal als hinweis für 0.7.1 + + +- add global delete_incomplete (/etc/ccollect/defaults/delete_incomplete) + +09:31 < Obri> telmich: hab nen kleinen tipp für ccollect +09:32 < Obri> telmich: ich habe hier hosts die nicht immer online sind, das ist dumm weil so das backup + kaputtgeht... +09:32 < Obri> telmich: ich habe jetzt das ein preexec script gebastelt: +09:32 < Obri> ping -c1 -q `cat /etc/ccollect/sources/$name/source | cut -d"@" -f2 | cut -d":" -f1` +09:33 < Obri> telmich: so bricht das backup ab wenn der host nicht erreichbar ist +09:33 < Obri> ohne dass ein altes backup entsorgt wird + + +- remove basename + -> include config from cconf! -> standard! + +reject: +-------------------------------------------------------------------------------- +- fix german documentation! + => I'm the coder, somebody else can fix it. +- add wrapper, to do logging and analyse + * output filename in the wrapper, save into variable + => mktemp + * call analyser + => output stdout? + => no use for that currently diff --git a/software/ccollect/ccollect-0.8/doc/todo/0.7.2 b/software/ccollect/ccollect-0.8/doc/todo/0.7.2 new file mode 100644 index 00000000..79a50aa2 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/todo/0.7.2 @@ -0,0 +1,63 @@ +-------------------------------------------------------------------------------- +Stats version +-------------------------------------------------------------------------------- + +Add tools/ccollect_stats.sh, clearify license +-------------------------------------------------------------------------------- +Preamble: + Netstream (www.netstream.ch) may consider using ccollect for backing up many + unix servers and needs some clean reporting. The following things + need to be done, so ccollect will be useful for netstream: + +Logger: + - Needs to write small mails (sysadmins don't have time to read mails) + - Needs to be able to only write a mail on error + * needs configuration option to also mail on warnings + - Should be able to send one mail per backup source + * or one for the whole backup job + +Messages: (to be used for filtering) + Errors: + Read from remote host .*: Connection timed out + + Warnings: + rsync: mknod .* failed: Invalid argument (22) + file has vanished: ".*" + +-------------------------------------------------------------------------------- + +Analyzer: + - grosse Dateien + - grosse Veraenderungen +-------------------------------------------------------------------------------- +exclude-lists-doku: +freebsd: + /usr/ports/* + /proc/* + /dev/* + /tmp/* + /var/tmp/* +linux: + /sys/* + /proc/* + /dev/* + /tmp/* + +-------------------------------------------------------------------------------- +done: + +rsync_extra global! +- \n delimeted +-------------------------------------------------------------------------------- + -S, --sparse + Try to handle sparse files efficiently so they take up less space on the des‐ + tination. Conflicts with --inplace because it’s not possible to overwrite + data in a sparse fashion. + +-------------------------------------------------------------------------------- + Always report return code! + +[12:00] u0255:ddba034.netstream.ch# rsync -n -a --delete --stats --progress daily.20080324-0313.17841/ daily.20080325-0313.31148/ + +$tool + diff --git a/software/ccollect/ccollect-0.8/doc/todo/0.7.3 b/software/ccollect/ccollect-0.8/doc/todo/0.7.3 new file mode 100644 index 00000000..73e5ffc6 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/todo/0.7.3 @@ -0,0 +1,2 @@ +- add -a (archive) to ccollect_delete_source.text +- add listing of intervals to ccollect_list_intervals diff --git a/software/ccollect/ccollect-0.8/doc/todo/0.7.4 b/software/ccollect/ccollect-0.8/doc/todo/0.7.4 new file mode 100644 index 00000000..70515e76 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/todo/0.7.4 @@ -0,0 +1 @@ +add support for wipe diff --git a/software/ccollect/ccollect-0.8/doc/todo/0.8.0 b/software/ccollect/ccollect-0.8/doc/todo/0.8.0 new file mode 100644 index 00000000..2b5130dc --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/todo/0.8.0 @@ -0,0 +1,6 @@ +- restructure code to easily allow global versus source options: + f_name=.... + check_option $f_name + => source + => defaults +- support all senseful options as default and source specific diff --git a/software/ccollect/ccollect-0.8/doc/todo/extern b/software/ccollect/ccollect-0.8/doc/todo/extern new file mode 100644 index 00000000..cb42a075 --- /dev/null +++ b/software/ccollect/ccollect-0.8/doc/todo/extern @@ -0,0 +1,26 @@ +These todos are things that would be senseful todo, but are just nice +to have. This means I won't code them, so somebody can code them. +-------------------------------------------------------------------------------- + +- Add ccollect-restore.sh + (new project, perhaps coordinated with jchome's ccollect-config) + + Helper (in dialog?) to restore backups + + Perhaps create ccollect-gui.{whatever} instead, which does all the + end user staff? + + Including cronjob, creating configuration, displaying status of + backups, allowing manual remove, allowing manual trigger of ccollect + (with realtime message window?)? + +- write mkccollectconfig (for 0.5!) + -> jchome works on it! + o create source configuration + o another script for changing defaults + x intervalls + x pre-/post exec + o (X)dialog based + +- add option ("run_only_once") to prevent ccollect to run twice + (per source/general) diff --git a/software/ccollect/ccollect-0.8/release.sh b/software/ccollect/ccollect-0.8/release.sh new file mode 100644 index 00000000..ae095ecd --- /dev/null +++ b/software/ccollect/ccollect-0.8/release.sh @@ -0,0 +1,60 @@ +#!/bin/sh +# +# 200?-2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Standard release script for dummies like me +# + +if [ $# -ne 2 ]; then + echo "$0: version description" + exit 23 +fi + +echo "Did you change version and date information in the script?" +read bla + +VERSION="$1"; shift +DESCRIPTION="$1"; shift +NAME=ccollect-${VERSION} +TARNAME=${NAME}.tar.bz2 +TARLOCAL=../${TARNAME} +#DHOST=nico@home.schottelius.org +DHOST=nico@localhost +#DDIR=www/unix.schottelius.org/www/ccollect/ +DDIR=/home/users/nico/privat/computer/net/netzseiten/www.nico.schottelius.org/src/software/ccollect +DESTINATION="$DHOST:$DDIR" + +set -e +set -x +git tag -m "$DESCRIPTION" "$VERSION" +git push --mirror +git archive --prefix="${NAME}/" "$VERSION" | bzip2 > "$TARLOCAL" +scp "${TARLOCAL}" "$DESTINATION" + +# create & publish documentation for the end user +make publish-doc + + +ssh "$DHOST" "( cd "$DDIR" &&; tar xfj \"$TARNAME\" )" + +echo "setting paranoid permissions to public..." +ssh "$DHOST" "( cd "$DDIR" && find -type d -exec chmod 0755 {} \; )" +ssh "$DHOST" "( cd "$DDIR" &&; find -type f -exec chmod 0644 {} \; )" + +cat "doc/release-checklist" +exit 0 diff --git a/software/ccollect/ccollect-0.8/tools/README b/software/ccollect/ccollect-0.8/tools/README new file mode 100644 index 00000000..73120184 --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/README @@ -0,0 +1,19 @@ +Files and their tasks / destinations: + +ccollect_add_source.sh: bin +ccollect_analyse_logs.sh: bin +ccollect_create_source2.sh: ??? +ccollect_create_source.sh: ??? +ccollect_delete_source.sh: bin +ccollect_list_intervals.sh: bin +ccollect_logwrapper.sh: bin +ccollect-stats.sh: ??? +config-pre-0.4-to-0.4.BUGS: only to be used for converting +config-pre-0.4-to-0.4.sh: only to be used for converting +config-pre-0.4-to-0.4.sub.sh: only to be used for converting +config-pre-0.6-to-0.6.sh: only to be used for converting +config-pre-0.6-to-0.6.sub.sh: only to be used for converting +config-pre-0.7-to-0.7.sh: only to be used for converting +config-pre-0.7-to-0.7.sub.sh: only to be used for converting +gnu-du-backup-size-compare.sh +README diff --git a/software/ccollect/ccollect-0.8/tools/ccollect_add_source.sh b/software/ccollect/ccollect-0.8/tools/ccollect_add_source.sh new file mode 100644 index 00000000..4a60d383 --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/ccollect_add_source.sh @@ -0,0 +1,123 @@ +#!/bin/sh +# +# 2007-2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# 2007-08-16: Written for Netstream (www.netstream.ch) +# +# Creates a source from standard values specified in +# /etc/ccollect/defaults/sources +# + +# standard values +CCOLLECT_CONF="${CCOLLECT_CONF:-/etc/ccollect}" +CSOURCES="${CCOLLECT_CONF}/sources" +CDEFAULTS="${CCOLLECT_CONF}/defaults" +SCONFIG="${CDEFAULTS}/sources" + +# standard options: variable2filename +exclude="exclude" +summary="summary" +intervals="intervals" +pre_exec="pre_exec" +post_exec="post_exec" +rsync_options="rsync_options" +verbose="verbose" +very_verbose="very_verbose" + +# options that we simply copy over +standard_opts="exclude summary intervals pre_exec post_exec rsync_options verbose very_verbose" + +# options not in standard ccollect, used only for source generation +src_prefix="${SCONFIG}/source_prefix" +src_postfix="${SCONFIG}/source_postfix" +destination_base="${SCONFIG}/destination_base" + +self="$(basename $0)" + +# functions first +_echo() +{ + echo "${self}> $@" +} + +_exit_err() +{ + _echo "$@" + rm -f "$TMP" + exit 1 +} + +# argv +if [ $# -lt 1 ]; then + _exit_err "" +fi + +_echo "Reading defaults from ${SCONFIG} ..." + +while [ $# -gt 0 ]; do + source="$1"; shift + + # Create + _echo "Creating ${source} ..." + + fullname="${CSOURCES}/${source}" + + # create source + if [ -e "${fullname}" ]; then + _echo "${fullname} already exists. Skipping." + continue + fi + mkdir -p "${fullname}" || _exit_err Cannot create \"${fullname}\". + + # copy standard files + for file in $standard_opts; do + eval rfile=\"\$$file\" + eval filename="${SCONFIG}/${rfile}" + if [ -e "${filename}" ]; then + _echo Copying \"$rfile\" to ${fullname} ... + cp -r "${filename}" "${fullname}/${rfile}" + fi + done + + # create source entry + if [ -f "${src_prefix}" ]; then + source_source="$(cat "${src_prefix}")" || _exit_err "${src_prefix}: Reading failed." + fi + source_source="${source_source}${source}" + if [ -f "${src_postfix}" ]; then + source_source="${source_source}$(cat "${src_postfix}")" || _exit_err "${src_postfix}: Reading failed." + fi + _echo "Adding \"${source_source}\" as source for ${source}" + echo "${source_source}" > "${fullname}/source" || _exit_err "Creating ${fullname}/source: failed." + + # create destination directory + absbase=$(cd "${destination_base}" 2>/dev/null && pwd -P) || \ + _exit_err "${destination_base} must exist before creating sources." + + dest="${absbase}/${source}" + _echo "Creating ${dest} ..." + mkdir -p "${dest}" || _exit_err "${dest}: Cannot create." + + # link destination directory + dest_abs=$(cd "${dest}" && pwd -P) || _exit_err "${dest}: Changing to newly created directory failed." + echo "${dest_abs}" > "${fullname}/destination" || \ + _exit_err "${fullname}/destination: Failed to create." + +done + +exit 0 diff --git a/software/ccollect/ccollect-0.8/tools/ccollect_analyse_logs.sh b/software/ccollect/ccollect-0.8/tools/ccollect_analyse_logs.sh new file mode 100644 index 00000000..7ff9916b --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/ccollect_analyse_logs.sh @@ -0,0 +1,126 @@ +#!/bin/sh +# +# 2007-2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Written for Netstream (www.netstream.ch) on Di 21. Aug 17:10:15 CEST 2007 +# +# Analyse logs +# + +version=0.2 +date=2008-06-13 +fullversion="${version} (${date})" +args=iwe + +usage() { + echo "$(basename "$0") ${fullversion}: [iwe]" + echo "" + echo " i: print informational messages" + echo " w: print warning messages" + echo " e: print error messages" + echo "" + echo "Reading input from stdin, displaying to stdout." + exit 1 +} + +# +# read and verify argv +# +if [ "$#" -ne 1 ]; then + usage +fi +argv="$1"; shift + +wrong="$(echo ${argv} | grep -e "[^${args}]")" +if [ "${wrong}" ]; then + usage +fi + + +# set output levels +search_err="$(echo ${argv} | grep 'e')" +search_warn="$(echo ${argv} | grep 'w')" +search_info="$(echo ${argv} | grep 'i')" + +# +# Interesting strings in the logs: errors +# --------------------------------------- + +if [ "$search_err" ]; then + set -- "$@" "-e" 'Read from remote host .*: Connection timed out$' + set -- "$@" "-e" 'Read from remote host .*: Connection reset by peer$' + set -- "$@" "-e" 'rsync: .*: Invalid argument (22)$' + set -- "$@" "-e" 'rsync: .*: Input/output error (5)$' + set -- "$@" "-e" 'cannot send long-named file "' + set -- "$@" "-e" 'ERROR: .* failed verification -- update discarded.$' + set -- "$@" "-e" 'Host key verification failed.' + set -- "$@" "-e" 'ssh: connect to host .*: Connection timed out' + set -- "$@" "-e" 'rsync error: unexplained error (code 255)' + set -- "$@" "-e" 'rsync: connection unexpectedly closed' +fi + +# known error strings: +#[ddba049.netstream.ch] receiving file list ... rsync: readdir("/proc"): Invalid argument (22) +#[ddba033.netstream.ch] rsync: readlink "/usr/local/inetpub2/webmailroot/2wire.ch/royal@2wire.ch" failed: Input/output error (5) +#[ddba033.netstream.ch] rsync: read errors mapping "/usr/local/inetpub2/netstream/adsl.netstream.ch/mrtg/lbswiss.rrd": Input/output error (5) +#[zrha165.netstream.ch] Read from remote host zrha165.netstream.ch: Connection reset by peer +#[ddba017.netstream.ch] receiving file list ... cannot send long-named file "/usr/local/www/apache22/cgi-bin/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/ backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/bac kup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup /backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/ba ckup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backu p/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/backup/back" +#[ddba033.netstream.ch] ERROR: usr/local/inetpub2/netstream/adsl.netstream.ch/mrtg/lbswiss.rrd failed verification -- update discarded. +# [ddba034.netstream.ch] Host key verification failed. +# [ddba034.netstream.ch] rsync error: unexplained error (code 255) at io.c(454) [receiver=2.6.9] +# [ddba034.netstream.ch] rsync: connection unexpectedly closed (0 bytes received so far) [receiver] + + + + +# +# Interesting strings in the logs: warnings +# ----------------------------------------- +#[ddba033.netstream.ch] rsync: read errors mapping "/usr/local/inetpub2/netstream/adsl.netstream.ch/mrtg/lbswiss.rrd": Input/output error (5) + +# [ddba015.netstream.ch] send_files failed to open /usr/local/dnscache/log/main/@4000000046ca0f3616939c14.s: No such file or directory +# [ddba015.netstream.ch] 2007-08-21-02:17:28: Finished backup (rsync return code: 23). +# [ddba017.netstream.ch] file has vanished: "/var/spool/postfix/active/657575D686" +#[ddba026.netstream.ch] 2007-08-21-05:35:13: Finished backup (rsync return code: 24). +#[ddba045.netstream.ch] send_files failed to open /data/hsphere/local/var/named/logs/@4000000046c98fa9079f39ac.s: No such file or directory +# file has vanished: ".*" + +if [ "$search_warn" ]; then + # warn on non-zero exit code + set -- "$@" "-e" 'Finished backup (rsync return code: [^0]' + set -- "$@" "-e" 'WARNING: .* failed verification -- update discarded (will try again).' +fi +# known warnings: +#[ddba033.netstream.ch] WARNING: usr/local/inetpub2/netstream/adsl.netstream.ch/mrtg/lbswiss.rrd failed verification -- update discarded (will try again). + + +# +# Interesting strings in the logs: informational +# ---------------------------------------------- +if [ "$search_info" ]; then + set -- "$@" "-e" 'total size is [[:digit:]]* speedup is' + set -- "$@" "-e" 'Backup lasted: [[:digit:]]*:[[:digit:]]\{1,2\}:[[:digit:]]* (h:m:s)$' + set -- "$@" "-e" 'send [[:digit:]]* bytes received [0-9]* bytes [0-9]* bytes/sec$' +fi + +# info includes: +#[ddba012.netstream.ch] total size is 22384627486 speedup is 13.75 +# [u0160.nshq.ch.netstream.com] 2007-08-20-18:26:06: Backup lasted: 0:43:34 (h:m:s) +#[ddba012.netstream.ch] sent 3303866 bytes received 1624630525 bytes 122700.92 bytes/sec + +grep "$@" diff --git a/software/ccollect/ccollect-0.8/tools/ccollect_archive_config.sh b/software/ccollect/ccollect-0.8/tools/ccollect_archive_config.sh new file mode 100644 index 00000000..f42c88a8 --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/ccollect_archive_config.sh @@ -0,0 +1,53 @@ +#!/bin/sh +# +# 2009 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Write a tar containing the configuration +# + +# standard values +CCOLLECT_CONF="${CCOLLECT_CONF:-/etc/ccollect}" + +self="$(basename $0)" + +# functions first +_echo() +{ + echo "${self}> $@" +} + +_exit_err() +{ + _echo "$@" + rm -f "$TMP" + exit 1 +} + +# argv +if [ $# -ne 1 ]; then + _echo "${self} " + _exit_err "Exiting." +fi + +file="$1" + +_echo "Creating $file ..." + +[ -d "${CCOLLECT_CONF}" ] || _exit_err "No configuration in $CCOLLECT_CONF" + +tar cf "$file" "${CCOLLECT_CONF}" diff --git a/software/ccollect/ccollect-0.8/tools/ccollect_check_config.sh b/software/ccollect/ccollect-0.8/tools/ccollect_check_config.sh new file mode 100644 index 00000000..c24a4006 --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/ccollect_check_config.sh @@ -0,0 +1,40 @@ +#!/bin/sh +# +# 2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# + +################################################################################ +# standard vars stolen from cconf +__pwd="$(pwd -P)" +__mydir="${0%/*}"; __abs_mydir="$(cd "$__mydir" && pwd -P)" +__myname=${0##*/}; __abs_myname="$__abs_mydir/$__myname" + + +################################################################################ +# ccollect standard vars +CCOLLECT_CONF="${CCOLLECT_CONF:-/etc/ccollect}" +CDEFAULTS="${CCOLLECT_CONF}/defaults" +CLOGDIR="${CDEFAULTS}/logdir" +CRONTAB="${CRONTAB:-/etc/crontab}" + +# Parameters: +# -c, --crontab +# -f, --fix +# -l, --logs + + diff --git a/software/ccollect/ccollect-0.8/tools/ccollect_delete_source.sh b/software/ccollect/ccollect-0.8/tools/ccollect_delete_source.sh new file mode 100644 index 00000000..509e95fd --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/ccollect_delete_source.sh @@ -0,0 +1,108 @@ +#!/bin/sh +# +# 2007-2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# 2007-08-16 Written for Netstream (www.netstream.ch) +# Delete sources and their backups (optionally). +# + +# standard values +CCOLLECT_CONF="${CCOLLECT_CONF:-/etc/ccollect}" +CSOURCES="${CCOLLECT_CONF}/sources" + +self="$(basename $0)" + +# functions first +_echo() +{ + echo "${self}> $@" +} + +_exit_err() +{ + _echo "$@" + rm -f "$TMP" + exit 1 +} + +# argv +if [ $# -lt 1 ]; then + _echo "${self} [-f] [-d] " + _echo " -f: Do not ask, simply delete. Dangerous and good for sysadmins." + _echo " -d: Also delete the destination (removes all backups)" + _exit_err "Exiting." +fi + +params_possible=yes +force="" +backups="" + +while [ $# -gt 0 ]; do + if [ "$params_possible" ]; then + case "$1" in + "-f"|"--force") + force="-r" + shift; continue + ;; + "-d"|"--destination") + backups=yes + shift; continue + ;; + --) + params_possible="" + shift; continue + ;; + -*|--*) + _exit_err "Unknown option: $1" + ;; + esac + fi + + # Reached here? So there are no more parameters. + params_possible="" + + source="$1"; shift + + # Create + _echo "Deleting ${source} ..." + fullname="${CSOURCES}/${source}" + + # ask the user per source, if she's not forcing us + if [ -z "$force" ]; then + sure="" + echo -n "Do you really want to delete ${source} (y/n)? " + read sure + + if [ "$sure" != "y" ]; then + _echo "Skipping ${source}." + continue + fi + fi + + if [ "$backups" ]; then + ddir="$(cat "${fullname}/destination")" + absdir="$(cd "${ddir}" && pwd -P)" || _exit_err "Cannot change to ${ddir}" + _echo "Deleting ${absdir} ..." + echo rm -r $force "${absdir}" + fi + + _echo "Deleting ${fullname} ..." + echo rm -r $force "${fullname}" +done + +exit 0 diff --git a/software/ccollect/ccollect-0.8/tools/ccollect_list_intervals.sh b/software/ccollect/ccollect-0.8/tools/ccollect_list_intervals.sh new file mode 100644 index 00000000..d95bbde7 --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/ccollect_list_intervals.sh @@ -0,0 +1,44 @@ +#!/bin/sh +# +# 2006-2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Initially written on 24-Jun-2006 +# + +CCOLLECT_CONF=${CCOLLECT_CONF:-/etc/ccollect} +CCOLLECT_INTERVALS="$CCOLLECT_CONF/defaults/intervals" + +me="$(basename $0)" + +_echo() +{ + echo "$me> $@" +} + +if [ ! -d "${CCOLLECT_INTERVALS}" ]; then + _echo "No intervals defined in ${CCOLLECT_INTERVALS}" + exit 1 +fi + +set -e +cd "${CCOLLECT_INTERVALS}" + +for interval in *; do + eval int_${interval}=$(cat "${interval}"); + eval echo ${interval}: \$int_${interval}; +done diff --git a/software/ccollect/ccollect-0.8/tools/ccollect_logwrapper.sh b/software/ccollect/ccollect-0.8/tools/ccollect_logwrapper.sh new file mode 100644 index 00000000..ac7f4961 --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/ccollect_logwrapper.sh @@ -0,0 +1,64 @@ +#!/bin/sh +# +# 2007-2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Written for Netstream (www.netstream.ch) on Fr 8. Jun 10:30:24 CEST 2007 +# +# Call the log-wrapper instead of ccollect.sh and it will create nice logs +# + +# +# where to find our configuration and temporary file +# +CCOLLECT_CONF=${CCOLLECT_CONF:-/etc/ccollect} +LOGCONF=$CCOLLECT_CONF/logwrapper + +logdir="${LOGCONF}/destination" +CDATE="date +%Y%m%d-%H%M" +we="$(basename $0)" +pid=$$ + +export ccollect_logfile="${logdir}/$(${CDATE}).${pid}" + +# use syslog normally +# Also use echo, can be redirected with > /dev/null if someone cares +_echo() +{ + string="${we} (${pid}): $@" + logger "${string}" + echo "${string}" +} + + +# exit on error +_exit_err() +{ + _echo "$@" >&2 + rm -f "${TMP}" + exit 1 +} + +# put everything into that specified file +_echo "Starting with arguments: $@" +touch "${ccollect_logfile}" || _exit_err "Failed to create ${ccollect_logfile}" + +# First line in the logfile is always the commandline +echo ccollect.sh "$@" > "${ccollect_logfile}" 2>&1 +ccollect.sh "$@" >> "${ccollect_logfile}" 2>&1 + +_echo "Finished." diff --git a/software/ccollect/ccollect-0.8/tools/ccollect_stats.sh b/software/ccollect/ccollect-0.8/tools/ccollect_stats.sh new file mode 100644 index 00000000..bfaefab8 --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/ccollect_stats.sh @@ -0,0 +1,25 @@ +#!/bin/sh +# +# 2007 Daniel Aubry +# 2008 Nico Schottelius (added minimal header) +# +# Copying license: GPL2-only +# + +# TODO: +# add variables, add copying, add configuration + +if [ ! -e /tmp/ccollect-stats.lock ] +then + touch /tmp/ccollect-stats.lock + + # changes after license clearify + # for dest in /etc/ccollect/sources/ -type f -name destination | while read line + + find /etc/ccollect/sources/ -type l | while read line + d=$(basename $(readlink $line)) + echo "====[Backup: $backupname]====" | tee -a /var/log/backup.log + du -sh $line/* | tee -a /var/log/backup.log + done + rm /tmp/ccollect-stats.lock +fi diff --git a/software/ccollect/ccollect-0.8/tools/config-pre-0.4-to-0.4.BUGS b/software/ccollect/ccollect-0.8/tools/config-pre-0.4-to-0.4.BUGS new file mode 100644 index 00000000..2be9d213 --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/config-pre-0.4-to-0.4.BUGS @@ -0,0 +1,27 @@ +-------------------------------------------------------------------------------- +BUGS, 2006-04-27, Nico Schottelius +-------------------------------------------------------------------------------- +The scripts are only for 'standard' configurations. + +If you've multiple directories named 'intervalls', the +scripts will try to replace _all_ instances of intervalls with +"intervals". + +This will fail: + + intervalls/defaults/intervalls/ exists before. + + Then we'll replace the first time: + + intervals/defaults/intervalls/ exists now. + + Then find will fail, as the first directory disappeared. + +Also, if you have a source named 'intervalls' +(why so ever you would want to have something like that) and +a 'intervalls' directory below it, it will rename your source. + +Yes this is a bug. + +No, I do not care about it, this is just a dumb helper script which +should be used once. diff --git a/software/ccollect/ccollect-0.8/tools/config-pre-0.4-to-0.4.sh b/software/ccollect/ccollect-0.8/tools/config-pre-0.4-to-0.4.sh new file mode 100644 index 00000000..23b3307e --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/config-pre-0.4-to-0.4.sh @@ -0,0 +1,48 @@ +#!/bin/sh +# +# 2006-2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Written on Do Apr 27 09:13:26 CEST 2006 +# + +if [ $# -ne 1 ]; then + echo "$0: ccollect-configuration directory" + echo "" + echo " Fix pre 0.4 configuration directories to match 0.4 style" + echo "" + exit 23 +fi + +tmp=$(mktemp) +tmp2=$(mktemp) +script=$(echo $0 | sed 's/\.sh/.sub.sh/') + +find "$1" -type d -name intervalls > "$tmp" + +# +# reverse found data, so deepest directories are renamed first +# +tac "$tmp" > "$tmp2" + +while read intervals + do + "$script" "$intervals" +done < "$tmp2" + +rm -f "$tmp" "$tmp2" diff --git a/software/ccollect/ccollect-0.8/tools/config-pre-0.4-to-0.4.sub.sh b/software/ccollect/ccollect-0.8/tools/config-pre-0.4-to-0.4.sub.sh new file mode 100644 index 00000000..e28e7184 --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/config-pre-0.4-to-0.4.sub.sh @@ -0,0 +1,41 @@ +#!/bin/sh +# +# 2006-2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Written on Do Apr 27 09:13:26 CEST 2006 +# + +master=$(echo $0 | sed 's/\.sub//') + +if [ $# -ne 1 ]; then + echo "$0:" + echo "" + echo " DO NOT CALL ME DIRECTLY" + echo "" + echo "Use $master, please." + exit 23 +fi + +# strip trailing / +oldname=$(echo $1 | sed 's,/$,,') + +# replace the last component of the path "intervalls" +newname=$(echo $oldname | sed 's/intervalls$/intervals/') + +echo mv "$oldname" "$newname" +mv "$oldname" "$newname" diff --git a/software/ccollect/ccollect-0.8/tools/config-pre-0.6-to-0.6.sh b/software/ccollect/ccollect-0.8/tools/config-pre-0.6-to-0.6.sh new file mode 100644 index 00000000..58a538ae --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/config-pre-0.6-to-0.6.sh @@ -0,0 +1,37 @@ +#!/bin/sh +# +# 2007-2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Written on 20070816-2225 +# Transfer configuration to 0.6 layout +# + +if [ $# -ne 1 ]; then + echo "$0: ccollect-configuration directory" + echo "" + echo " Fix pre 0.6 configuration directories to match 0.6 style" + echo "" + exit 23 +fi + +dir="$1" +script=$(echo $0 | sed 's/\.sh$/.sub.sh/') + +find "${dir}/sources/" -type f -name rsync_options -exec "${script}" {} \; + +echo "Finished." diff --git a/software/ccollect/ccollect-0.8/tools/config-pre-0.6-to-0.6.sub.sh b/software/ccollect/ccollect-0.8/tools/config-pre-0.6-to-0.6.sub.sh new file mode 100644 index 00000000..7744f9eb --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/config-pre-0.6-to-0.6.sub.sh @@ -0,0 +1,39 @@ +#!/bin/sh +# +# 2007-2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Written on 20070816-2227 +# Transfer configuration to 0.6 layout (subscript) +# + +if [ $# -ne 1 ]; then + echo "$0: rsync_options file" + echo "" + echo " Fix pre 0.6 configuration directories to match 0.6 style (sub)" + echo "" + exit 23 +fi + +tmp=$(mktemp) + +echo "Working on $1 ..." + +for option in $(cat "$1"); do + echo "${option}" >> "${tmp}" +done +mv ${tmp} "$1" diff --git a/software/ccollect/ccollect-0.8/tools/config-pre-0.7-to-0.7.sh b/software/ccollect/ccollect-0.8/tools/config-pre-0.7-to-0.7.sh new file mode 100644 index 00000000..6d540abb --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/config-pre-0.7-to-0.7.sh @@ -0,0 +1,36 @@ +#!/bin/sh +# +# 2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Convert configuration to 0.7 layout +# + +if [ $# -ne 1 ]; then + echo "$0: ccollect-configuration directory" + echo "" + echo " Fix pre 0.7 configuration directories to match 0.7 style" + echo "" + exit 23 +fi + +dir="$1" +script=$(echo $0 | sed 's/\.sh$/.sub.sh/') + +find "${dir}/sources/" -name destination -exec "${script}" {} \; + +echo "Finished." diff --git a/software/ccollect/ccollect-0.8/tools/config-pre-0.7-to-0.7.sub.sh b/software/ccollect/ccollect-0.8/tools/config-pre-0.7-to-0.7.sub.sh new file mode 100644 index 00000000..5bc6f759 --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/config-pre-0.7-to-0.7.sub.sh @@ -0,0 +1,59 @@ +#!/bin/sh +# +# 2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Helper script +# + +if [ $# -ne 1 ]; then + echo "$0: destination-file" + echo "" + echo " Fix pre 0.7 configuration directories to match 0.6 style (sub)" + echo "" + exit 23 +fi + +tmp=$(mktemp) +file="$1" + +echo "Working on $file ..." + +if [ -L "${file}" ]; then + echo "Converting ${file} ..." + dir="$(cd "${file}" && pwd -P)"; ret=$? + + if [ $ret -ne 0 ]; then + echo "ERROR: $file is a broken link" + exit 1 + else + echo "${dir}" > "${tmp}" + rm -f "${file}"; ret=$? + if [ $ret -ne 0 ]; then + echo "ERROR: Removing $file failed" + exit 1 + fi + mv "${tmp}" "${file}"; ret=$? + if [ $ret -ne 0 ]; then + echo "ERROR: Moving ${tmp} to ${file} failed, your source is broken." + exit 1 + fi + fi +else + echo "$file is not a link, not converting" + exit 1 +fi diff --git a/software/ccollect/ccollect-0.8/tools/gnu-du-backup-size-compare.sh b/software/ccollect/ccollect-0.8/tools/gnu-du-backup-size-compare.sh new file mode 100644 index 00000000..cf3b1948 --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/gnu-du-backup-size-compare.sh @@ -0,0 +1,48 @@ +#!/bin/sh +# +# 2007-2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Written on 2007-08-16 +# + +exit 1 +# not yet finished. + +CCOLLECT_CONF=${CCOLLECT_CONF:-/etc/ccollect} +CCOLLECT_SOURCES="$CCOLLECT_CONF/defaults/sources" + +me="$(basename "$0")" + +if [ $# -lt 1 ]; then + echo "${me}: sources names + exit 1 +fi + +if [ ! -d "$CCOLLECT_SOURCES" ]; then + echo "No sources defined in $CCOLLECT_SOURCES" + exit 1 +fi + +cd "${CCOLLECT_SOURCES}" + +while [ "$#" -gt 0 ]; do + source="$1"; shift + fsource="${CCOLLECT_SOURCES}/${source}" + du -s "${fsource}/"* "${fsource}" + # du -l should follow +done diff --git a/software/ccollect/ccollect-0.8/tools/old/ccollect_create_source.sh b/software/ccollect/ccollect-0.8/tools/old/ccollect_create_source.sh new file mode 100644 index 00000000..7a34c6b0 --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/old/ccollect_create_source.sh @@ -0,0 +1,75 @@ +#!/bin/sh +# Nico Schottelius +# 2007-08-07 +# Written for Netstream (www.netstream.ch) +# Creates a source, including exclude + +# standard values +CCOLLECT_CONF=${CCOLLECT_CONF:-/etc/ccollect} +CSOURCES=$CCOLLECT_CONF/sources +CDEFAULTS=$CCOLLECT_CONF/defaults + +self=$(basename $0) + +# functions first +_echo() +{ + echo "${self}> $@" +} + +_exit_err() +{ + _echo "$@" + rm -f "$TMP" + exit 1 +} + +# argv +if [ $# -ne 3 ]; then + _echo " " + _echo "Example: \"my-notebook\" \"192.168.42.42\" \"/home/server/backup/my-notebook\"" + exit 1 +fi + +name="$1" +source="$2" +destination="$3" +fullname="${CSOURCES}/${name}" + +# Tests +if [ -e "${fullname}" ]; then + _echo "${fullname} already exists. Aborting." + exit 2 +fi + +_echo "Trying to reach ${source} ..." +ping -c1 "${source}" || _exit_err "Cannot reach ${source}. Aborting." + +# Create +_echo "Creating ${fullname} ..." +mkdir -p "${fullname}" || exit 3 + +echo "root@${source}:/" > "${fullname}/source" +cat << eof > "${fullname}/exclude" || exit 4 +/dev/* +/proc/* +/tmp/* +eof + +# Destination +if [ -e "${destination}" ]; then + if [ ! -d "${destination}" ]; then + echo "${destination} exists, but is not a directory. Aborting." + exit 5 + fi +else + _echo "Creating ${destination} ..." + mkdir -p "${destination}" || _exit_err "Failed to create ${destination}." +fi + +ln -s "${destination}" "${fullname}/destination" || \ + _exit_err "Failed to link \"${destination}\" to \"${fullname}/destination\"" + +# finish +_echo "Added some default values, please verify \"${fullname}\"." +_echo "Finished." diff --git a/software/ccollect/ccollect-0.8/tools/old/ccollect_create_source2.sh b/software/ccollect/ccollect-0.8/tools/old/ccollect_create_source2.sh new file mode 100644 index 00000000..842afb65 --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/old/ccollect_create_source2.sh @@ -0,0 +1,85 @@ +#!/bin/sh +# Nico Schottelius +# 2007-08-07 +# Written for Netstream (www.netstream.ch) +# Creates a source, including exclude + +# standard values +CCOLLECT_CONF=${CCOLLECT_CONF:-/etc/ccollect} +CSOURCES=$CCOLLECT_CONF/sources +CDEFAULTS=$CCOLLECT_CONF/defaults + +self=$(basename $0) + +# functions first +_echo() +{ + echo "${self}> $@" +} + +_exit_err() +{ + _echo "$@" + rm -f "$TMP" + exit 1 +} + +# argv +if [ $# -ne 2 ]; then + _echo " " + _echo "Example: \"192.168.42.42\" \"/home/server/backup/\"" + _echo " This will create ${CSOURCES}/192.168.42.42 and /home/server/backup/192.168.42.42." + exit 1 +fi + +# sourcename / servername +name="$1" + +# basedir +basedir="$2" + +fullname="${CSOURCES}/${name}" +destination="${basedir}/${name}" + +# Tests +if [ -e "${fullname}" ]; then + _echo "${fullname} already exists. Aborting." + exit 2 +fi + +_echo "Trying to reach ${name} ..." +ping -c1 "${name}" || _exit_err "Cannot reach ${name}. Aborting." + +if [ ! -d "${basedir}" ]; then + echo "${basedir} is not a directory. Aborting." + exit 7 +fi + +# Create +_echo "Creating ${fullname} ..." +mkdir -p "${fullname}" || exit 3 + +echo "root@${name}:/" > "${fullname}/source" +cat << eof > "${fullname}/exclude" || exit 4 +/dev/* +/proc/* +/tmp/* +eof + +# Destination +if [ -e "${destination}" ]; then + if [ ! -d "${destination}" ]; then + echo "${destination} exists, but is not a directory. Aborting." + exit 5 + fi +else + _echo "Creating ${destination} ..." + mkdir -p "${destination}" || _exit_err "Failed to create ${destination}." +fi + +ln -s "${destination}" "${fullname}/destination" || \ + _exit_err "Failed to link \"${destination}\" to \"${fullname}/destination\"" + +# finish +_echo "Added some default values, please verify \"${fullname}\"." +_echo "Finished." diff --git a/software/ccollect/ccollect-0.8/tools/report_success.sh b/software/ccollect/ccollect-0.8/tools/report_success.sh new file mode 100644 index 00000000..b0f9d1bf --- /dev/null +++ b/software/ccollect/ccollect-0.8/tools/report_success.sh @@ -0,0 +1,63 @@ +#!/bin/sh +# +# 2008 Nico Schottelius (nico-ccollect at schottelius.org) +# +# This file is part of ccollect. +# +# ccollect is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# ccollect is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with ccollect. If not, see . +# +# Sends feedback +# + +subject="==> success: ccollect" +to="nico-ccollect-success" +host="schottelius.org" +fullto="${to}@${host}" + +software="ccollect" +author="Nico Schottelius" +info="$(uname -s -v -r -m)" + +echo "Reporting success for $software to ${author}" +echo "-----------------" +echo -n "Your name (leave free for anonymous): " +read name +echo -n "Your email (leave free for anonymous): " +read email +echo -n "Comment (leave free for no comment): " +read comment + +echo "" +echo "The following information will be send to ${author}:" +echo "" + +cat << eof +Name: $name (will be used to contact you and kept secret) +E-Mail: $email (will be used to contact you and kept secret) +Comment: $comment +Info: $info + +eof + +echo -n "Is it ok to send out that mail (press enter to send or ctrl-c to abort)? " +read yes + +cat << eof | mail -s "$subject" "$fullto" +Name: $name (will be used to contact you and kept secret) +E-Mail: $email (will be used to contact you and kept secret) +Comment: $comment +Info: $info +eof + +echo "Send. Thank you for your feedback."