__postgres_*: fix forgotten edge cases in delimited identifier escape

PreOS code did not use configuration support. This fix adds support
for using cdist configuration, which takes into account cdist
configuration file, environment variables and command line options,
especially conf_dir.

It also adds support for -g, --config-file option, for specifying
custom configuration file.

.gitignore

@@ -24,6 +24,8 @@ docs/src/man1/*.1
 docs/src/man7/*.7
 docs/src/man7/cdist-type__*.rst
 docs/src/cdist-reference.rst
+docs/src/cdist-types.rst
+docs/src/cdist.cfg.skeleton
 
 # Ignore cdist cache for version control
 /cache/

.gitlab-ci.yml

@@ -0,0 +1,18 @@
+stages:
+    - test
+
+unit_tests:
+    stage: test
+    script:
+        - ./bin/build-helper version
+        - ./bin/build-helper test
+
+pycodestyle:
+    stage: test
+    script:
+        - ./bin/build-helper pycodestyle
+
+shellcheck:
+    stage: test
+    script:
+        - ./bin/build-helper shellcheck

LICENSE

@@ -0,0 +1,674 @@
+                    GNU GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU General Public License is a free, copyleft license for
+software and other kinds of works.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+the GNU General Public License is intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.  We, the Free Software Foundation, use the
+GNU General Public License for most of our software; it applies also to
+any other work released this way by its authors.  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  To protect your rights, we need to prevent others from denying you
+these rights or asking you to surrender the rights.  Therefore, you have
+certain responsibilities if you distribute copies of the software, or if
+you modify it: responsibilities to respect the freedom of others.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must pass on to the recipients the same
+freedoms that you received.  You must make sure that they, too, receive
+or can get the source code.  And you must show them these terms so they
+know their rights.
+
+  Developers that use the GNU GPL protect your rights with two steps:
+(1) assert copyright on the software, and (2) offer you this License
+giving you legal permission to copy, distribute and/or modify it.
+
+  For the developers' and authors' protection, the GPL clearly explains
+that there is no warranty for this free software.  For both users' and
+authors' sake, the GPL requires that modified versions be marked as
+changed, so that their problems will not be attributed erroneously to
+authors of previous versions.
+
+  Some devices are designed to deny users access to install or run
+modified versions of the software inside them, although the manufacturer
+can do so.  This is fundamentally incompatible with the aim of
+protecting users' freedom to change the software.  The systematic
+pattern of such abuse occurs in the area of products for individuals to
+use, which is precisely where it is most unacceptable.  Therefore, we
+have designed this version of the GPL to prohibit the practice for those
+products.  If such problems arise substantially in other domains, we
+stand ready to extend this provision to those domains in future versions
+of the GPL, as needed to protect the freedom of users.
+
+  Finally, every program is threatened constantly by software patents.
+States should not allow patents to restrict development and use of
+software on general-purpose computers, but in those that do, we wish to
+avoid the special danger that patents applied to a free program could
+make it effectively proprietary.  To prevent this, the GPL assures that
+patents cannot be used to render the program non-free.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Use with the GNU Affero General Public License.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU Affero General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the special requirements of the GNU Affero General Public License,
+section 13, concerning interaction through a network will apply to the
+combination as such.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    cdist
+    Copyright (C) 2019  ungleich-public
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If the program does terminal interaction, make it output a short
+notice like this when it starts in an interactive mode:
+
+    cdist  Copyright (C) 2019  ungleich-public
+    This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, your program's commands
+might be different; for a GUI interface, you would use an "about box".
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU GPL, see
+<http://www.gnu.org/licenses/>.
+
+  The GNU General Public License does not permit incorporating your program
+into proprietary programs.  If your program is a subroutine library, you
+may consider it more useful to permit linking proprietary applications with
+the library.  If this is what you want to do, use the GNU Lesser General
+Public License instead of this License.  But first, please read
+<http://www.gnu.org/philosophy/why-not-lgpl.html>.

Makefile

@@ -63,6 +63,18 @@ DOCSREFSH=$(DOCS_SRC_DIR)/cdist-reference.rst.sh
 $(DOCSREF): $(DOCSREFSH)
 	$(DOCSREFSH)
 
+# Html types list with references
+DOCSTYPESREF=$(MAN7DSTDIR)/cdist-types.rst
+DOCSTYPESREFSH=$(DOCS_SRC_DIR)/cdist-types.rst.sh
+
+$(DOCSTYPESREF): $(DOCSTYPESREFSH)
+	$(DOCSTYPESREFSH)
+
+DOCSCFGSKEL=./configuration/cdist.cfg.skeleton
+
+configskel: $(DOCSCFGSKEL)
+	cp -f "$(DOCSCFGSKEL)" "$(DOCS_SRC_DIR)/"
+
 version:
 	@[ -f "cdist/version.py" ] || { \
 		printf "Missing 'cdist/version.py', please generate it first.\n" && exit 1; \
@@ -72,7 +84,7 @@ version:
 man: version $(MANTYPES) $(DOCSREF)
 	$(SPHINXM)
 
-html: version $(MANTYPES) $(DOCSREF)
+html: version configskel $(MANTYPES) $(DOCSREF) $(DOCSTYPESREF)
 	$(SPHINXH)
 
 docs: man html
@@ -114,6 +126,8 @@ speeches: $(SPEECHES)
 #
 clean: docs-clean
 	rm -f $(DOCS_SRC_DIR)/cdist-reference.rst
+	rm -f $(DOCS_SRC_DIR)/cdist-types.rst
+	rm -f $(DOCS_SRC_DIR)/cdist.cfg.skeleton
 
 	find "$(DOCS_SRC_DIR)" -mindepth 2 -type l \
 	| xargs rm -f

bin/build-helper

@@ -74,6 +74,7 @@ SHELLCHECKCMD="shellcheck -s sh -f gcc -x"
 # Skip SC2154 for variables starting with __ since such variables are cdist
 # environment variables.
 SHELLCHECK_SKIP=': __.*is referenced but not assigned.*\[SC2154\]'
+SHELLCHECKTMP=".shellcheck.tmp"
 
 # Change to checkout directory
 basedir="${0%/*}/../"
@@ -369,7 +370,7 @@ eof
         cat << eof
 Manual steps post release:
     - cdist-web
-    - send mail body generated in mailinglist.tmp and inform Dmitry for deb
+    - send generated mailinglist.tmp mail
     - twitter
 eof
     ;;
@@ -405,7 +406,7 @@ eof
     ;;
 
     pycodestyle|pep8)
-        pycodestyle "${basedir}" "${basedir}/scripts/cdist" | less
+        pycodestyle "${basedir}" "${basedir}/scripts/cdist"
     ;;
 
     check-pycodestyle)
@@ -431,53 +432,67 @@ eof
     ;;
 
     shellcheck-global-explorers)
-        find cdist/conf/explorer -type f -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" || exit 0
+        # shellcheck disable=SC2086
+        find cdist/conf/explorer -type f -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" > "${SHELLCHECKTMP}"
+        test ! -s "${SHELLCHECKTMP}" || { cat "${SHELLCHECKTMP}"; exit 1; }
     ;;
 
     shellcheck-type-explorers)
-        find cdist/conf/type -type f -path "*/explorer/*" -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" || exit 0
+        # shellcheck disable=SC2086
+        find cdist/conf/type -type f -path "*/explorer/*" -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" > "${SHELLCHECKTMP}"
+        test ! -s "${SHELLCHECKTMP}" || { cat "${SHELLCHECKTMP}"; exit 1; }
     ;;
 
     shellcheck-manifests)
-        find cdist/conf/type -type f -name manifest -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" || exit 0
+        # shellcheck disable=SC2086
+        find cdist/conf/type -type f -name manifest -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" > "${SHELLCHECKTMP}"
+        test ! -s "${SHELLCHECKTMP}" || { cat "${SHELLCHECKTMP}"; exit 1; }
     ;;
 
     shellcheck-local-gencodes)
-        find cdist/conf/type -type f -name gencode-local -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" || exit 0
+        # shellcheck disable=SC2086
+        find cdist/conf/type -type f -name gencode-local -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" > "${SHELLCHECKTMP}"
+        test ! -s "${SHELLCHECKTMP}" || { cat "${SHELLCHECKTMP}"; exit 1; }
     ;;
 
     shellcheck-remote-gencodes)
-        find cdist/conf/type -type f -name gencode-remote -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" || exit 0
+        # shellcheck disable=SC2086
+        find cdist/conf/type -type f -name gencode-remote -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" > "${SHELLCHECKTMP}"
+        test ! -s "${SHELLCHECKTMP}" || { cat "${SHELLCHECKTMP}"; exit 1; }
     ;;
 
     shellcheck-scripts)
-        ${SHELLCHECKCMD} scripts/cdist-dump scripts/cdist-new-type || exit 0
+        # shellcheck disable=SC2086
+        ${SHELLCHECKCMD} scripts/cdist-dump scripts/cdist-new-type > "${SHELLCHECKTMP}"
+        test ! -s "${SHELLCHECKTMP}" || { cat "${SHELLCHECKTMP}"; exit 1; }
     ;;
 
     shellcheck-gencodes)
-        "$0" shellcheck-local-gencodes
+        "$0" shellcheck-local-gencodes || exit 1
-        "$0" shellcheck-remote-gencodes
+        "$0" shellcheck-remote-gencodes || exit 1
     ;;
 
     shellcheck-types)
-        "$0" shellcheck-type-explorers
+        "$0" shellcheck-type-explorers || exit 1
-        "$0" shellcheck-manifests
+        "$0" shellcheck-manifests || exit 1
-        "$0" shellcheck-gencodes
+        "$0" shellcheck-gencodes || exit 1
     ;;
 
     shellcheck)
-        "$0" shellcheck-global-explorers
+        "$0" shellcheck-global-explorers || exit 1
-        "$0" shellcheck-types
+        "$0" shellcheck-types || exit 1
-        "$0" shellcheck-scripts
+        "$0" shellcheck-scripts || exit 1
     ;;
 
     shellcheck-type-files)
-        find cdist/conf/type -type f -path "*/files/*" -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" || exit 0
+        # shellcheck disable=SC2086
+        find cdist/conf/type -type f -path "*/files/*" -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" > "${SHELLCHECKTMP}"
+        test ! -s "${SHELLCHECKTMP}" || { cat "${SHELLCHECKTMP}"; exit 1; }
     ;;
 
     shellcheck-with-files)
-        "$0" shellcheck
+        "$0" shellcheck || exit 1
-        "$0" shellcheck-type-files
+        "$0" shellcheck-type-files || exit 1
     ;;
 
     shellcheck-build-helper)
@@ -535,6 +550,7 @@ eof
 
         # Temp files
         rm -f ./*.tmp
+        rm -f ./.*.tmp
     ;;
 
     distclean)

cdist/argparse.py

@@ -6,6 +6,7 @@ import collections
 import functools
 import cdist.configuration
 import cdist.preos
+import cdist.info
 
 
 # set of beta sub-commands
@@ -103,7 +104,7 @@ def get_parsers():
                                    name="log level"),
             help=('Set the specified verbosity level. '
                   'The levels, in order from the lowest to the highest, are: '
-                  'ERROR (-1), WARNING (0), INFO (1), VERBOSE (2), DEBUG (3) '
+                  'ERROR (-1), WARNING (0), INFO (1), VERBOSE (2), DEBUG (3), '
                   'TRACE (4 or higher). If used along with -v then -v '
                   'increases last set value and -l overwrites last set '
                   'value.'),
@@ -424,7 +425,7 @@ def get_parsers():
     parser['inventory'].set_defaults(
             func=cdist.inventory.Inventory.commandline)
 
-    # PreOs
+    # PreOS
     parser['preos'] = parser['sub'].add_parser('preos', add_help=False)
 
     # Shell
@@ -436,6 +437,37 @@ def get_parsers():
                   ' should be POSIX compatible shell.'))
     parser['shell'].set_defaults(func=cdist.shell.Shell.commandline)
 
+    # Info
+    parser['info'] = parser['sub'].add_parser('info')
+    parser['info'].add_argument(
+            '-a', '--all', help='Display all info. This is the default.',
+            action='store_true', default=False)
+    parser['info'].add_argument(
+            '-c', '--conf-dir',
+            help='Add configuration directory (can be repeated).',
+            action='append')
+    parser['info'].add_argument(
+            '-e', '--global-explorers',
+            help='Display info for global explorers.', action='store_true',
+            default=False)
+    parser['info'].add_argument(
+            '-F', '--fixed-string',
+            help='Interpret pattern as a fixed string.', action='store_true',
+            default=False)
+    parser['info'].add_argument(
+            '-f', '--full', help='Display full details.',
+            action='store_true', default=False)
+    parser['info'].add_argument(
+           '-g', '--config-file',
+           help='Use specified custom configuration file.',
+           dest="config_file", required=False)
+    parser['info'].add_argument(
+            '-t', '--types', help='Display info for types.',
+            action='store_true', default=False)
+    parser['info'].add_argument(
+            'pattern', nargs='?', help='Glob pattern.')
+    parser['info'].set_defaults(func=cdist.info.Info.commandline)
+
     for p in parser:
         parser[p].epilog = EPILOG
 

cdist/conf/type/__acl/gencode-remote

@@ -28,7 +28,10 @@ acl_path="/$__object_id"
 
 acl_is="$( cat "$__object/explorer/acl_is" )"
 
-if [ -f "$__object/parameter/acl" ]
+if [ -f "$__object/parameter/entry" ]
+then
+    acl_should="$( cat "$__object/parameter/entry" )"
+elif [ -f "$__object/parameter/acl" ]
 then
     acl_should="$( cat "$__object/parameter/acl" )"
 elif

cdist/conf/type/__acl/man.rst

@@ -15,7 +15,7 @@ See ``setfacl`` and ``acl`` manpages for more details.
 
 REQUIRED MULTIPLE PARAMETERS
 ----------------------------
-acl
+entry
    Set ACL entry following ``getfacl`` output syntax.
 
 
@@ -36,8 +36,8 @@ remove
 
 DEPRECATED PARAMETERS
 ---------------------
-Parameters ``user``, ``group``, ``mask`` and ``other`` are deprecated and they
+Parameters ``acl``, ``user``, ``group``, ``mask`` and ``other`` are deprecated and they
-will be removed in future versions. Please use ``acl`` parameter instead.
+will be removed in future versions. Please use ``entry`` parameter instead.
 
 
 EXAMPLES
@@ -49,27 +49,27 @@ EXAMPLES
         --default \
         --recursive \
         --remove \
-        --acl user:alice:rwx \
+        --entry user:alice:rwx \
-        --acl user:bob:r-x \
+        --entry user:bob:r-x \
-        --acl group:project-group:rwx \
+        --entry group:project-group:rwx \
-        --acl group:some-other-group:r-x \
+        --entry group:some-other-group:r-x \
-        --acl mask::r-x \
+        --entry mask::r-x \
-        --acl other::r-x
+        --entry other::r-x
 
     # give Alice read-only access to subdir,
     # but don't allow her to see parent content.
 
     __acl /srv/project2 \
         --remove \
-        --acl default:group:secret-project:rwx \
+        --entry default:group:secret-project:rwx \
-        --acl group:secret-project:rwx \
+        --entry group:secret-project:rwx \
-        --acl user:alice:--x
+        --entry user:alice:--x
 
     __acl /srv/project2/subdir \
         --default \
         --remove \
-        --acl group:secret-project:rwx \
+        --entry group:secret-project:rwx \
-        --acl user:alice:r-x
+        --entry user:alice:r-x
 
 
 AUTHORS

cdist/conf/type/__acl/parameter/deprecated/acl

@@ -0,0 +1 @@
+see manual for details

cdist/conf/type/__acl/parameter/optional_multiple

@@ -1,3 +1,4 @@
+entry
 acl
 user
 group

cdist/conf/type/__consul_agent/manifest

@@ -181,7 +181,7 @@ init_upstart()
 
 # Install init script to start on boot
 case "$os" in
-    alpine|devuan)
+    devuan)
         init_sysvinit debian
         ;;
     centos|redhat)

cdist/conf/type/__install_chroot_umount/manifest

@@ -0,0 +1 @@
+../__chroot_umount/manifest

cdist/conf/type/__install_directory/man.rst

@@ -1 +0,0 @@
-../__directory/man.rst

cdist/conf/type/__install_directory/man.rst

@@ -0,0 +1,101 @@
+cdist-type__install_directory(7)
+================================
+
+NAME
+----
+cdist-type__install_directory - Manage a directory with install command
+
+
+DESCRIPTION
+-----------
+This cdist type allows you to create or remove directories on the target.
+
+
+REQUIRED PARAMETERS
+-------------------
+None.
+
+
+OPTIONAL PARAMETERS
+-------------------
+state
+   'present' or 'absent', defaults to 'present'
+
+group
+   Group to chgrp to.
+
+mode
+   Unix permissions, suitable for chmod.
+
+owner
+   User to chown to.
+
+
+BOOLEAN PARAMETERS
+------------------
+parents
+   Whether to create parents as well (mkdir -p behaviour).
+   Warning: all intermediate directory permissions default
+   to whatever mkdir -p does. 
+
+   Usually this means root:root, 0700.
+
+recursive
+   If supplied the chgrp and chown call will run recursively.
+   This does *not* influence the behaviour of chmod.
+
+MESSAGES
+--------
+chgrp <group>
+    Changed group membership
+chown <owner>
+    Changed owner
+chmod <mode>
+    Changed mode
+create
+    Empty directory was created
+remove
+    Directory exists, but state is absent, directory will be removed by generated code.
+remove non directory
+    Something other than a directory with the same name exists and was removed prior to create.
+
+
+EXAMPLES
+--------
+
+.. code-block:: sh
+
+    # A silly example
+    __install_directory /tmp/foobar
+
+    # Remove a directory
+    __install_directory /tmp/foobar --state absent
+
+    # Ensure /etc exists correctly
+    __install_directory /etc --owner root --group root --mode 0755
+
+    # Create nfs service directory, including parents
+    __install_directory /home/services/nfs --parents
+
+    # Change permissions recursively
+    __install_directory /home/services --recursive --owner root --group root
+
+    # Setup a temp directory
+    __install_directory /local --mode 1777
+
+    # Take it all
+    __install_directory /home/services/kvm --recursive --parents \
+        --owner root --group root --mode 0755 --state present
+
+
+AUTHORS
+-------
+Nico Schottelius <nico-cdist--@--schottelius.org>
+
+
+COPYING
+-------
+Copyright \(C) 2011 Nico Schottelius. You can redistribute it
+and/or modify it under the terms of the GNU General Public License as
+published by the Free Software Foundation, either version 3 of the
+License, or (at your option) any later version.

cdist/conf/type/__install_file/man.rst

@@ -23,6 +23,10 @@ symlink
 directory
   replace it with the source file
 
+One exception is that when state is pre-exists, an error is raised if
+the file would have been created otherwise (e.g. it is not present or
+not a regular file).
+
 In any case, make sure that the file attributes are as specified.
 
 
@@ -33,7 +37,7 @@ None.
 OPTIONAL PARAMETERS
 -------------------
 state
-   'present', 'absent' or 'exists', defaults to 'present' where:
+   'present', 'absent', 'exists' or 'pre-exists', defaults to 'present' where:
 
    present
       the file is exactly the one from source
@@ -41,6 +45,9 @@ state
       the file does not exist
    exists
       the file from source but only if it doesn't already exist
+   pre-exists
+      check that the file exists and is a regular file, but do not
+      create or modify it
 
 group
    Group to chgrp to.
@@ -56,6 +63,9 @@ source
    If not supplied, an empty file or directory will be created.
    If source is '-' (dash), take what was written to stdin as the file content.
 
+onchange
+   The code to run if file is modified.
+
 MESSAGES
 --------
 chgrp <group>
@@ -93,6 +103,8 @@ EXAMPLES
     __install_file /home/frodo/.bashrc --source "/etc/skel/.bashrc" \
        --state exists \
        --owner frodo --mode 0600
+    # Check that the file is present, show an error when it is not
+    __install_file /etc/somefile --state pre-exists
     # Take file content from stdin
     __install_file /tmp/whatever --owner root --group root --mode 644 --source - << DONE
         Here goes the content for /tmp/whatever

cdist/conf/type/__mysql_database/explorer/state

@@ -0,0 +1,33 @@
+#!/bin/sh -e
+#
+# 2020 Ander Punnar (ander-at-kvlt-dot-ee)
+#
+# This file is part of cdist.
+#
+# cdist is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# cdist is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with cdist. If not, see <http://www.gnu.org/licenses/>.
+#
+
+if [ -f "$__object/parameter/name" ]
+then
+    name="$( cat "$__object/parameter/name" )"
+else
+    name="$__object_id"
+fi
+
+if [ -n "$( mysql -B -N -e "show databases like '$name'" )" ]
+then
+    echo 'present'
+else
+    echo 'absent'
+fi

cdist/conf/type/__mysql_database/gencode-remote

@@ -1,6 +1,6 @@
 #!/bin/sh -e
 #
-# 2012 Benedikt Koeppel (code@benediktkoeppel.ch)
+# 2020 Ander Punnar (ander-at-kvlt-dot-ee)
 #
 # This file is part of cdist.
 #
@@ -17,38 +17,30 @@
 # You should have received a copy of the GNU General Public License
 # along with cdist. If not, see <http://www.gnu.org/licenses/>.
 #
-#
 
-# if --database was specified
+state_is="$( cat "$__object/explorer/state" )"
-if [ -f "$__object/parameter/name" ]; then
+
-   database="$(cat "$__object/parameter/name")"
+state_should="$( cat "$__object/parameter/state" )"
-else # otherwise use the object id as database name
+
-   database="$__object_id"
+if [ "$state_is" = "$state_should" ]
+then
+    exit 0
 fi
 
-cat <<-EOFF
+if [ -f "$__object/parameter/name" ]
-mysql -u root <<-EOF
+then
-	CREATE DATABASE IF NOT EXISTS $database
+    name="$( cat "$__object/parameter/name" )"
-EOF
+else
-EOFF
+    name="$__object_id"
-
-# if --user was specified
-if [ -f "$__object/parameter/user" ]; then
-   user="$(cat "$__object/parameter/user")"
-
-   # if --password was specified
-   if [ -f "$__object/parameter/password" ]; then
-      password="$(cat "$__object/parameter/password")"
-      cat <<-EOFF
-      mysql -u root <<-EOF
-      	GRANT ALL PRIVILEGES ON $database.* to '$user'@'localhost' IDENTIFIED BY '$password';
-EOF
-EOFF
-   else
-      cat <<-EOFF
-      mysql -u root <<-EOF
-      	GRANT ALL PRIVILEGES ON $database.* to '$user'@'localhost';
-EOF
-EOFF
-   fi
 fi
+
+case "$state_should" in
+    present)
+        echo "mysql -e 'create database \`$name\`'"
+        echo "create database $name" >> "$__messages_out"
+    ;;
+    absent)
+        echo "mysql -e 'drop database \`$name\`'"
+        echo "drop database $name" >> "$__messages_out"
+    ;;
+esac

cdist/conf/type/__mysql_database/man.rst

@@ -8,24 +8,24 @@ cdist-type__mysql_database - Manage a MySQL database
 
 DESCRIPTION
 -----------
-This cdist type allows you to install a MySQL database.
 
+Create MySQL database and optionally user with all privileges.
 
-REQUIRED PARAMETERS
--------------------
-None.
 
 OPTIONAL PARAMETERS
 -------------------
 name
-   The name of the database to install
+   Name of database. Defaults to object id.
-   defaults to the object id
 
 user
-   A user that should have access to the database
+   Create user and give all privileges to database.
 
 password
-   The password for the user who manages the database
+   Password for user.
+
+state
+   Defaults to present.
+   If absent and user is also set, both will be removed (with privileges).
 
 
 EXAMPLES
@@ -33,17 +33,23 @@ EXAMPLES
 
 .. code-block:: sh
 
-    __mysql_database "cdist" --name "cdist" --user "myuser" --password "mypwd"
+    # just create database
+    __mysql_database foo
+
+    # create database with respective user with all privileges to database
+    __mysql_database bar \
+        --user name \
+        --password secret
 
 
 AUTHORS
 -------
-Benedikt Koeppel <code@benediktkoeppel.ch>
+Ander Punnar <ander-at-kvlt-dot-ee>
 
 
 COPYING
 -------
-Copyright \(C) 2012 Benedikt Koeppel. You can redistribute it
+Copyright \(C) 2020 Ander Punnar. You can redistribute it and/or modify it
-and/or modify it under the terms of the GNU General Public License as
+under the terms of the GNU General Public License as published by the Free
-published by the Free Software Foundation, either version 3 of the
+Software Foundation, either version 3 of the License, or (at your option) any
-License, or (at your option) any later version.
+later version.

cdist/conf/type/__mysql_database/manifest

@@ -0,0 +1,52 @@
+#!/bin/sh -e
+#
+# 2020 Ander Punnar (ander-at-kvlt-dot-ee)
+#
+# This file is part of cdist.
+#
+# cdist is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# cdist is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with cdist. If not, see <http://www.gnu.org/licenses/>.
+#
+
+if [ -f "$__object/parameter/user" ]
+then
+    user="$( cat "$__object/parameter/user" )"
+fi
+
+if [ -f "$__object/parameter/password" ]
+then
+    password="$( cat "$__object/parameter/password" )"
+fi
+
+if [ -n "$user" ] && [ -n "$password" ]
+then
+    if [ -f "$__object/parameter/name" ]
+    then
+        database="$( cat "$__object/parameter/name" )"
+    else
+        database="$__object_id"
+    fi
+
+    state_should="$( cat "$__object/parameter/state" )"
+
+    __mysql_user "$user" \
+        --password "$password" \
+        --state "$state_should"
+
+    # removing user should remove all user's privileges
+    require="__mysql_user/$user" \
+        __mysql_privileges "$database/$user" \
+            --database "$database" \
+            --user "$user" \
+            --state "$state_should"
+fi

cdist/conf/type/__mysql_database/parameter/default/state

@@ -0,0 +1 @@
+present

cdist/conf/type/__mysql_database/parameter/optional

@@ -1,3 +1,4 @@
 name
 user
 password
+state

cdist/conf/type/__mysql_privileges/explorer/state

@@ -0,0 +1,40 @@
+#!/bin/sh -e
+#
+# 2020 Ander Punnar (ander-at-kvlt-dot-ee)
+#
+# This file is part of cdist.
+#
+# cdist is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# cdist is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with cdist. If not, see <http://www.gnu.org/licenses/>.
+#
+
+privileges="$( cat "$__object/parameter/privileges" )"
+
+database="$( cat "$__object/parameter/database" )"
+
+table="$( cat "$__object/parameter/table" )"
+
+user="$( cat "$__object/parameter/user" )"
+
+host="$( cat "$__object/parameter/host" )"
+
+check_privileges="$( 
+    mysql -B -N -e "show grants for '$user'@'$host'" \
+        | grep -Ei "^grant $privileges on .$database.\..$table. to " || true )"
+
+if [ -n "$check_privileges" ]
+then
+    echo 'present'
+else
+    echo 'absent'
+fi

cdist/conf/type/__mysql_privileges/gencode-remote

@@ -0,0 +1,49 @@
+#!/bin/sh -e
+#
+# 2020 Ander Punnar (ander-at-kvlt-dot-ee)
+#
+# This file is part of cdist.
+#
+# cdist is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# cdist is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with cdist. If not, see <http://www.gnu.org/licenses/>.
+#
+
+state_is="$( cat "$__object/explorer/state" )"
+
+state_should="$( cat "$__object/parameter/state" )"
+
+if [ "$state_is" = "$state_should" ]
+then
+    exit 0
+fi
+
+privileges="$( cat "$__object/parameter/privileges" )"
+
+database="$( cat "$__object/parameter/database" )"
+
+table="$( cat "$__object/parameter/table" )"
+
+user="$( cat "$__object/parameter/user" )"
+
+host="$( cat "$__object/parameter/host" )"
+
+case "$state_should" in
+    present)
+        echo "mysql -e 'grant $privileges on \`$database\`.\`$table\` to \`$user\`@\`$host\`'"
+        echo "grant $privileges on $database.$table to $user@$host" >> "$__messages_out"
+    ;;
+    absent)
+        echo "mysql -e 'revoke $privileges on \`$database\`.\`$table\` from \`$user\`@\`$host\`'"
+        echo "revoke $privileges on $database.$table from $user@$host" >> "$__messages_out"
+    ;;
+esac

cdist/conf/type/__mysql_privileges/man.rst

@@ -0,0 +1,57 @@
+cdist-type__mysql_privileges(7)
+===============================
+
+NAME
+----
+cdist-type__mysql_privileges - Manage MySQL privileges
+
+
+DESCRIPTION
+-----------
+
+Grant and revoke privileges of MySQL user.
+
+
+REQUIRED PARAMETERS
+-------------------
+database
+   Name of database.
+
+User
+   Name of user.
+
+
+OPTIONAL PARAMETERS
+-------------------
+privileges
+   Defaults to "all".
+
+table
+   Defaults to "*".
+
+host
+   Defaults to localhost.
+
+state
+   "present" grants and "absent" revokes. Defaults to present.
+
+
+EXAMPLES
+--------
+
+.. code-block:: sh
+
+    __mysql_privileges user-to-db --database db --user user
+
+
+AUTHORS
+-------
+Ander Punnar <ander-at-kvlt-dot-ee>
+
+
+COPYING
+-------
+Copyright \(C) 2020 Ander Punnar. You can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the Free
+Software Foundation, either version 3 of the License, or (at your option) any
+later version.

cdist/conf/type/__mysql_privileges/parameter/default/host

@@ -0,0 +1 @@
+localhost

cdist/conf/type/__mysql_privileges/parameter/default/privileges

@@ -0,0 +1 @@
+all privileges

cdist/conf/type/__mysql_privileges/parameter/default/state

@@ -0,0 +1 @@
+present

cdist/conf/type/__mysql_privileges/parameter/default/table

@@ -0,0 +1 @@
+*

cdist/conf/type/__mysql_privileges/parameter/optional

@@ -0,0 +1,4 @@
+privileges
+table
+host
+state

cdist/conf/type/__mysql_privileges/parameter/required

@@ -0,0 +1,2 @@
+database
+user

cdist/conf/type/__mysql_user/explorer/state

@@ -0,0 +1,54 @@
+#!/bin/sh -e
+#
+# 2020 Ander Punnar (ander-at-kvlt-dot-ee)
+#
+# This file is part of cdist.
+#
+# cdist is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# cdist is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with cdist. If not, see <http://www.gnu.org/licenses/>.
+#
+
+if [ -f "$__object/parameter/name" ]
+then
+    name="$( cat "$__object/parameter/name" )"
+else
+    name="$__object_id"
+fi
+
+if [ -f "$__object/parameter/password" ]
+then
+    password="$( cat "$__object/parameter/password" )"
+else
+    password=''
+fi
+
+host="$( cat "$__object/parameter/host" )"
+
+check_user="$( mysql -B -N -e "select user from mysql.user where user = '$name' and host = '$host'" )"
+
+if [ -n "$check_user" ]
+then
+    if [ -n "$password" ]
+    then
+        check_password="$( mysql -B -N -e "select user from mysql.user where user = '$name' and host = '$host' and password = password( '$password' )" )"
+    fi
+
+    if [ -n "$password" ] && [ -z "$check_password" ]
+    then
+        echo 'change-password'
+    else
+        echo 'present'
+    fi
+else
+    echo 'absent'
+fi

cdist/conf/type/__mysql_user/gencode-remote

@@ -0,0 +1,68 @@
+#!/bin/sh -e
+#
+# 2020 Ander Punnar (ander-at-kvlt-dot-ee)
+#
+# This file is part of cdist.
+#
+# cdist is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# cdist is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with cdist. If not, see <http://www.gnu.org/licenses/>.
+#
+
+state_is="$( cat "$__object/explorer/state" )"
+
+state_should="$( cat "$__object/parameter/state" )"
+
+if [ "$state_is" = "$state_should" ]
+then
+    exit 0
+fi
+
+if [ -f "$__object/parameter/name" ]
+then
+    name="$( cat "$__object/parameter/name" )"
+else
+    name="$__object_id"
+fi
+
+host="$( cat "$__object/parameter/host" )"
+
+if [ -f "$__object/parameter/password" ]
+then
+    password="$( cat "$__object/parameter/password" )"
+else
+    if [ "$state_should" = 'present' ]
+    then
+        echo '--password needed' >&2
+        exit 1
+    else
+        password=''
+    fi
+fi
+
+if [ "$state_is" = 'absent' ] && [ "$state_should" = 'present' ]
+then
+    echo "mysql -e 'create user \`$name\`@\`$host\` identified by \"$password\"'"
+    echo "create user $name@$host" >> "$__messages_out"
+
+elif [ "$state_is" != 'absent' ] && [ "$state_should" = 'absent' ]
+then
+    echo "mysql -e 'drop user \`$name\`@\`$host\`'"
+    echo "drop user $name@$host" >> "$__messages_out"
+
+elif [ "$state_is" = 'change-password' ]
+then
+    # this only works with MySQL 5.7.6 and later or MariaDB 10.1.20 and later
+    echo "mysql -e 'alter user \`$name\`@\`$host\` identified by \"$password\"'"
+    echo "mysql -e 'flush privileges'"
+    echo "change password $name@$host" >> "$__messages_out"
+fi

cdist/conf/type/__mysql_user/man.rst

@@ -0,0 +1,48 @@
+cdist-type__mysql_user(7)
+=========================
+
+NAME
+----
+cdist-type__mysql_user - Manage a MySQL user
+
+
+DESCRIPTION
+-----------
+
+Create MySQL user or change password for the user.
+
+
+OPTIONAL PARAMETERS
+-------------------
+name
+   Name of user. Defaults to object id.
+
+host
+   Host of user. Defaults to localhost.
+
+password
+   Password of user.
+
+state
+   Defaults to present.
+
+
+EXAMPLES
+--------
+
+.. code-block:: sh
+
+    __mysql_user user --password secret
+
+
+AUTHORS
+-------
+Ander Punnar <ander-at-kvlt-dot-ee>
+
+
+COPYING
+-------
+Copyright \(C) 2020 Ander Punnar. You can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the Free
+Software Foundation, either version 3 of the License, or (at your option) any
+later version.

cdist/conf/type/__mysql_user/parameter/default/host

@@ -0,0 +1 @@
+localhost

cdist/conf/type/__mysql_user/parameter/default/state

@@ -0,0 +1 @@
+present

cdist/conf/type/__mysql_user/parameter/optional

@@ -0,0 +1,4 @@
+name
+host
+password
+state

cdist/conf/type/__openldap_server/gencode-remote

@@ -0,0 +1,44 @@
+#!/bin/sh
+
+manager_dn=$(cat "${__object}/parameter/manager-dn")
+manager_password=$(cat "${__object}/parameter/manager-password")
+description=$(cat "${__object}/parameter/description")
+suffix=$(cat "${__object}/parameter/suffix")
+suffix_dc=$(printf "%s" "${suffix}" | awk -F',' '{print $1}' | awk -F'=' '{print $2}')
+
+SLAPD_IPC=$(tr '\n' ' ' < "${__object}/parameter/slapd-url" | awk '{ print $1}')
+
+cat <<DONE # | tee /dev/stderr
+# Restart service
+service slapd restart
+# It can sometimes take a tiny bit to bind
+sleep 1
+# Create or update base object
+if ldapsearch -xZ -D "${manager_dn}" -w "${manager_password}" -H '${SLAPD_IPC}' -b '${suffix}' -s base 2>&1 > /dev/null; then
+    # Already exists, use ldapmodify
+    ldapmodify -xZ -D "${manager_dn}" -w "${manager_password}" -H '${SLAPD_IPC}' <<EOF
+dn: ${suffix}
+changetype: modify
+replace: objectClass
+objectClass: top
+objectClass: dcObject
+objectClass: organization
+-
+replace: o
+o: ${description}
+-
+replace: dc
+dc: ${suffix_dc}
+EOF
+else
+    # Does not exist, use ldapadd
+    ldapadd -xZ -D "${manager_dn}" -w "${manager_password}" -H '${SLAPD_IPC}' <<EOF
+dn: ${suffix}
+objectClass: top
+objectClass: dcObject
+objectClass: organization
+o: ${description}
+dc: ${suffix_dc}
+EOF
+fi
+DONE

cdist/conf/type/__openldap_server/man.rst

@@ -0,0 +1,212 @@
+cdist-type__openldap_server(7)
+==============================
+
+NAME
+----
+cdist-type__openldap_server - Setup an openldap(4) server instance
+
+
+DESCRIPTION
+-----------
+This type can be used to bootstrap an LDAP environment using openldap as slapd.
+
+It bootstraps the LDAP server with sane defaults and creates and manages the
+base DN defined by `suffix`.
+
+
+REQUIRED PARAMETERS
+-------------------
+manager-dn
+    The rootdn to set up in the directory.
+    E.g. `cn=manager,dc=ungleich,dc=ch`. See `slapd.conf(5)`.
+
+manager-password
+    The password for `manager-dn` in the directory.
+    This will be used to connect to the LDAP server on the first `slapd-url`
+    with the given `manager-dn`.
+
+manager-password-hash
+    The password for `manager-dn` in the directory.
+    This should be valid for `slapd.conf` like `{SSHA}qV+mCs3u8Q2sCmUXT4Ybw7MebHTASMyr`.
+    Generate e.g. with: `slappasswd -s weneedgoodsecurity`.
+    See `slappasswd(8C)`, `slapd.conf(5)`.
+    TODO: implement this: http://blog.adamsbros.org/2015/06/09/openldap-ssha-salted-hashes-by-hand/
+      to derive from the manager-password parameter and ensure idempotency (care with salts).
+      At that point, manager-password-hash should be deprecated and ignored.
+
+serverid
+    The server for the directory.
+    E.g. `dc=ungleich,dc=ch`. See `slapd.conf(5)`.
+
+suffix
+    The suffix for the directory.
+    E.g. `dc=ungleich,dc=ch`. See `slapd.conf(5)`.
+
+
+REQUIRED MULTIPLE PARAMETERS
+----------------------------
+slapd-url
+    A URL for slapd to listen on.
+    Pass once for each URL you want to support,
+    e.g.: `--slapd-url ldaps://my.fqdn/ --slapd-url ldap://my.fqdn/`.
+    The first instance that is passed will be used as the main URL to
+    connect to this LDAP server
+    See the `-h` flag in `slapd(8C)`.
+
+
+OPTIONAL PARAMETERS
+-------------------
+syncrepl-credentials
+    Only has an effect if `replicate` is set; required in that case.
+    This secret is shared amongst the hosts that will replicate the directory.
+    Note that each replication server needs this secret and it is saved in
+    plain text in the directory.
+
+syncrepl-searchbase
+    Only has an effect if `replicate` is set; required in that case.
+    The searchbase to use for replication.
+    E.g. `dc=ungleich,dc=ch`. See `slapd.conf(5)`.
+
+admin-email
+    Passed to `cdist-type__letsencrypt_cert`; has otherwise no use.
+    Required if using `__letsencrypt_cert`.
+    Where to send Let's Encrypt emails like "certificate needs renewal".
+
+tls-cipher-suite
+    Setting for TLSCipherSuite.
+    Defaults to `NORMAL` in a Debian-like OS and `HIGH:MEDIUM:+SSLv2` on FreeBSD.
+    See `slapd.conf(5)`.
+
+tls-cert
+    If defined, `__letsencrypt_cert` is not used and this must be the path in
+    the remote hosts to the PEM-encoded TLS certificate.
+    Requires: `tls-privkey` and `tls-ca`.
+    Permissions, existence and renewal of these files are left up to the
+    type's user.
+
+tls-privkey
+    Required if `tls-cert` is defined.
+    Path in the remote hosts to the PEM-encoded private key file.
+
+tls-ca
+    Required if `tls-cert` is defined.
+    Path in the remote hosts to the PEM-encoded CA certificate file.
+
+
+OPTIONAL MULTIPLE PARAMETERS
+----------------------------
+syncrepl-host
+    Only has an effect if `replicate` is set; required in that case.
+    Set once per host that will replicate the directory.
+
+module
+    LDAP module to load. See `slapd.conf(5)`.
+    Default value is OS-dependent, see manifest.
+
+schema
+    Name of LDAP schema to load. Must be the name without extension of a
+    `.schema` file in slapd's schema directory (usually `/etc/slapd/schema` or
+    `/usr/local/etc/openldap/schema`).
+    Example value: `inetorgperson`
+    The type user must ensure that the schema file is deployed.
+    This defaults to a sensible subset, for details see the type definition.
+
+description
+    The description of the base DN passed in the `suffix` parameter.
+    Defaults to `Managed by cdist, do not edit manually.`
+
+
+BOOLEAN PARAMETERS
+------------------
+staging
+    Passed to `cdist-type__letsencrypt_cert`; has otherwise no use.
+    Obtain a test certificate from a staging server.
+
+replicate
+    Whether to setup replication or not.
+    If present `syncrepl-credentials` and `syncrepl-host` are also required.
+
+
+EXAMPLES
+--------
+
+.. code-block:: sh
+
+    # Example of a simple server with manual certificate management.
+    pki_prefix="/usr/local/etc/pki/realms/ldap.camilion.cloud"
+    __openldap_server \
+        --manager-dn 'cn=manager,dc=camilion,dc=cloud' \
+        --manager-password "foo" \
+        --manager-password-hash '{SSHA}foo' \
+        --serverid 0 \
+        --suffix 'dc=camilion,dc=cloud' \
+        --slapd-url 'ldaps://ldap.camilion.cloud' \
+        --tls-cert "${pki_prefix}/default.crt" \
+        --tls-privkey "${pki_prefix}/default.key" \
+        --tls-ca "${pki_prefix}/CA.crt"
+
+    # The created basedn looks as follows:
+    #
+    # dn: dc=camilion,dc=cloud
+    # objectClass: top
+    # objectClass: dcObject
+    # objectClass: organization
+    # o: Managed by cdist, do not edit manually.
+    # dc: camilion
+    #
+    # Do not change it manually, the type will overwrite your changes.
+
+
+    #
+    # Changing to a replicated setup is a simple change to something like:
+    #
+    # Example for multiple servers with replication and automatic
+    # Let's Encrypt certificate management through certbot.
+    id=1
+    for host in ldap-test1.ungleich.ch ldap-test2.ungleich.ch; do
+        echo "__ungleich_ldap \
+            --manager-dn 'cn=manager,dc=ungleich,dc=ch' \
+            --manager-psasword 'foo' \
+            --manager-password-hash '{SSHA}fooo' \
+            --serverid '${id}' \
+            --suffix 'dc=ungleich,dc=ch' \
+            --slapd-url ldap://${host} \
+            --searchbase 'dc=ungleich,dc=ch' \
+            --syncrepl-credentials 'fooo' \
+            --syncrepl-host 'ldap-test1.ungleich.ch' \
+            --syncrepl-host 'ldap-test2.ungleich.ch' \
+            --description 'Ungleich LDAP server'" \
+            --staging \
+            | cdist config -i - -v ${host}
+        id=$((id + 1))
+    done
+
+    # The created basedn looks as follows:
+    #
+    # dn: dc=ungleich,dc=ch
+    # objectClass: top
+    # objectClass: dcObject
+    # objectClass: organization
+    # o: Ungleich LDAP server
+    # dc: ungleich
+    #
+    # Do not change it manually, the type will overwrite your changes.
+
+
+SEE ALSO
+--------
+:strong:`cdist-type__letsencrypt_cert`\ (7)
+
+
+AUTHORS
+-------
+ungleich <foss--@--ungleich.ch>
+Evilham <contact--@--evilham.com>
+
+
+COPYING
+-------
+Copyright \(C) 2020 ungleich glarus ag. You can redistribute it
+and/or modify it under the terms of the GNU General Public License as
+published by the Free Software Foundation, either version 3 of the
+License, or (at your option) any later version.

cdist/conf/type/__openldap_server/manifest

@@ -0,0 +1,263 @@
+#!/bin/sh
+
+name="${__target_host}"
+manager_dn=$(cat "${__object}/parameter/manager-dn")
+manager_password_hash=$(cat "${__object}/parameter/manager-password-hash")
+serverid=$(cat "${__object}/parameter/serverid")
+suffix=$(cat "${__object}/parameter/suffix")
+slapd_modules=$(cat "${__object}/parameter/module" 2>/dev/null || true)
+schemas=$(cat "${__object}/parameter/schema")
+slapd_urls=$(tr '\n' ' ' < "${__object}/parameter/slapd-url")
+tls_cipher_suite=$(cat "${__object}/parameter/tls-cipher-suite" 2>/dev/null || true)
+
+
+os="$(cat "${__global}/explorer/os")"
+
+# Setup OS-dependent vars
+CONF_OWNER="root"
+CONF_GROUP="root"
+case "${os}" in
+    freebsd)
+        PKGS="openldap-server"
+        ETC="/usr/local/etc"
+        SLAPD_DIR="/usr/local/etc/openldap"
+        SLAPD_DATA_DIR="/var/db/openldap-data"
+        SLAPD_RUN_DIR="/var/run/openldap"
+        SLAPD_MODULE_PATH="/usr/local/libexec/openldap"
+        if [ -z "${slapd_modules}" ]; then
+            # It looks like ppolicy and syncprov must be compiled
+            slapd_modules="back_mdb back_monitor"
+        fi
+        CONF_OWNER="ldap"
+        CONF_GROUP="ldap"
+        if [ -z "${tls_cipher_suite}" ]; then
+            # TODO: research default for FreeBSD. 'NORMAL' appears to not work
+            tls_cipher_suite="HIGH:MEDIUM:+SSLv2"
+        fi
+        ;;
+    debian|ubuntu|devuan)
+        PKGS="slapd ldap-utils"
+        ETC="/etc"
+        SLAPD_DIR="/etc/ldap"
+        SLAPD_DATA_DIR="/var/lib/ldap"
+        SLAPD_RUN_DIR="/var/run/slapd"
+        SLAPD_MODULE_PATH="/usr/lib/ldap"
+        if [ -z "${slapd_modules}" ]; then
+            slapd_modules="back_mdb ppolicy syncprov back_monitor"
+        fi
+        if [ -z "${tls_cipher_suite}" ]; then
+            tls_cipher_suite="NORMAL"
+        fi
+        ;;
+    *)
+        echo "Don't know the openldap defaults for: $os" >&2
+        exit 1
+        ;;
+esac
+
+PKG_MAIN=$(echo "${PKGS}" | awk '{print $1;}')
+
+
+# Determine if __letsencrypt_cert is to be used and setup vars accordingly
+if [ -f "${__object}/parameter/tls-cert" ]; then
+    tls_cert=$(cat "${__object}/parameter/tls-cert")
+
+    if [ ! -f "${__object}/parameter/tls-privkey" ]; then
+        echo "When tls-cert is defined, tls-privkey is also required." >&2
+        exit 1
+    fi
+    tls_privkey=$(cat "${__object}/parameter/tls-privkey")
+
+    if [ ! -f "${__object}/parameter/tls-ca" ]; then
+        echo "When tls-cert is defined, tls-ca is also required." >&2
+        exit 1
+    fi
+    tls_ca=$(cat "${__object}/parameter/tls-ca")
+
+    _skip_letsencrypt_cert="YES"
+else
+    if [ ! -f "${__object}/parameter/admin-email" ]; then
+        echo "When using __letsencrypt_cert, admin-email is also required." >&2
+        exit 1
+    fi
+    admin_email=$(cat "${__object}/parameter/admin-email")
+
+    tls_cert="${SLAPD_DIR}/sasl2/cert.pem"
+    tls_privkey="${SLAPD_DIR}/sasl2/privkey.pem"
+    tls_ca="${SLAPD_DIR}/sasl2/chain.pem"
+fi
+
+mkdir "${__object}/files"
+ldapconf="${__object}/files/ldapconf"
+
+replication=""
+if [ -f "${__object}/parameter/replicate" ]; then
+    replication=yes
+
+    if [ ! -f "${__object}/parameter/syncrepl-searchbase" ]; then
+        echo "Requiring the searchbase for replication" >&2
+        exit 1
+    fi
+    syncrepl_searchbase=$(cat "${__object}/parameter/syncrepl-searchbase")
+
+    if [ ! -f "${__object}/parameter/syncrepl-credentials" ]; then
+        echo "Requiring credentials for replication" >&2
+        exit 1
+    fi
+
+    syncrepl_credentials=$(cat "${__object}/parameter/syncrepl-credentials")
+
+    if [ ! -f "${__object}/parameter/syncrepl-host" ]; then
+        echo "Requiring host(s) for replication" >&2
+        exit 1
+    fi
+    syncrepl_hosts=$(cat "${__object}/parameter/syncrepl-host")
+
+fi
+
+# Install required packages
+for pkg in ${PKGS}; do
+    __package "${pkg}"
+done
+
+
+require="__package/${PKG_MAIN}" __start_on_boot slapd
+
+# Setup -h flag for the listeners. See man slapd (-h flag).
+case "${os}" in
+    freebsd)
+        require="__start_on_boot/slapd" __key_value \
+               --file "/etc/rc.conf" \
+               --key "slapd_flags" \
+               --value "\"-h '${slapd_urls}'\"" \
+               --delimiter "=" \
+               --comment "# LDAP Listener URLs" \
+               "${__target_host}__slapd_flags"
+        ;;
+    debian|ubuntu|devuan)
+        require="__package/${PKG_MAIN}" __line rm_slapd_conf \
+               --file ${ETC}/default/slapd \
+               --regex 'SLAPD_CONF=.*' \
+               --state absent
+
+        require="__package/${PKG_MAIN}" __line rm_slapd_services \
+               --file ${ETC}/default/slapd \
+               --regex 'SLAPD_SERVICES=.*' \
+               --state absent
+
+        require="__line/rm_slapd_conf" __line add_slapd_conf \
+               --file ${ETC}/default/slapd \
+               --line "SLAPD_CONF=${SLAPD_DIR}/slapd.conf" \
+               --state present
+
+        require="__line/rm_slapd_services" __line add_slapd_services \
+               --file ${ETC}/default/slapd \
+               --line "SLAPD_SERVICES=\"${slapd_urls}\"" \
+               --state present
+        ;;
+    *)
+        # Nothing to do here, move on.
+        ;;
+esac
+
+
+if [ -z "${_skip_letsencrypt_cert}" ]; then
+    if [ -f "${__object}/parameter/staging" ]; then
+        staging="--staging"
+    else
+        staging=""
+    fi
+
+    __letsencrypt_cert "${name}" --admin-email "${admin_email}" \
+        --renew-hook "cp ${ETC}/letsencrypt/live/${name}/*.pem ${SLAPD_DIR}/sasl2 && chown -R openldap:openldap ${SLAPD_DIR}/sasl2 && service slapd restart" \
+        --automatic-renewal ${staging}
+fi
+
+require="__package/${PKG_MAIN}" __directory ${SLAPD_DIR}/slapd.d --state absent
+
+if [ -z "${_skip_letsencrypt_cert}" ]; then
+    require="__package/${PKG_MAIN} __letsencrypt_cert/${name}" \
+           __file ${SLAPD_DIR}/slapd.conf --owner ${CONF_OWNER} --group ${CONF_GROUP} --mode 644 \
+           --source "${ldapconf}"
+else
+    require="__package/${PKG_MAIN}" \
+           __file ${SLAPD_DIR}/slapd.conf --owner ${CONF_OWNER} --group ${CONF_GROUP} --mode 644 \
+           --source "${ldapconf}"
+fi
+
+# Start slapd.conf
+cat << EOF > "${ldapconf}"
+pidfile ${SLAPD_RUN_DIR}/slapd.pid
+argsfile ${SLAPD_RUN_DIR}/slapd.args
+
+TLSCipherSuite ${tls_cipher_suite}
+TLSCertificateFile ${tls_cert}
+TLSCertificateKeyFile ${tls_privkey}
+TLSCACertificateFile ${tls_ca}
+
+disallow bind_anon
+require bind
+security tls=1
+EOF
+
+# Add specified schemas
+for schema in ${schemas}; do
+    echo "include ${SLAPD_DIR}/schema/${schema}.schema" >> "${ldapconf}"
+done
+
+# Add specified modules
+echo "modulepath ${SLAPD_MODULE_PATH}" >> "${ldapconf}"
+for module in ${slapd_modules}; do
+    echo "moduleload ${module}.la" >> "${ldapconf}"
+done
+
+# Rest of the config
+cat << EOF >> "${ldapconf}"
+loglevel 1024
+
+database mdb
+maxsize 1073741824
+
+suffix "${suffix}"
+directory ${SLAPD_DATA_DIR}
+rootdn "${manager_dn}"
+rootpw "${manager_password_hash}"
+
+index objectClass eq,pres
+index ou,cn,mail,surname,givenname eq,pres,sub
+index uidNumber,gidNumber,loginShell eq,pres
+index uid,memberUid eq,pres,sub
+index nisMapName,nisMapEntry eq,pres,sub
+index entryCSN,entryUUID eq
+
+serverid ${serverid}
+EOF
+
+# Setup replication
+if [ "${replication}" ]; then
+    rid=1;
+    for syncrepl in ${syncrepl_hosts}; do
+    cat <<EOF >> "${ldapconf}"
+syncrepl rid=${rid}
+ provider=ldap://${syncrepl}
+ bindmethod=simple
+ starttls=yes
+ binddn="${manager_dn}"
+ credentials=${syncrepl_credentials}
+ searchbase="${syncrepl_searchbase}"
+ type=refreshAndPersist
+ retry="5 + 5 +"
+ interval=00:00:00:05
+EOF
+    rid=$((rid + 1))
+    done
+    cat <<EOF >> "${ldapconf}"
+mirrormode true
+overlay syncprov
+syncprov-checkpoint 100 5
+syncprov-sessionlog 100
+
+database monitor
+limits dn.exact="${manager_dn}" time=unlimited size=unlimited
+EOF
+fi

cdist/conf/type/__openldap_server/parameter/boolean

@@ -0,0 +1,2 @@
+staging
+replicate

cdist/conf/type/__openldap_server/parameter/default/description

@@ -0,0 +1 @@
+Managed by cdist, do not edit manually.

cdist/conf/type/__openldap_server/parameter/default/schema

@@ -0,0 +1,12 @@
+corba
+core
+cosine
+duaconf
+dyngroup
+inetorgperson
+java
+misc
+nis
+openldap
+ppolicy
+collective

cdist/conf/type/__openldap_server/parameter/optional

@@ -0,0 +1,8 @@
+description
+syncrepl-credentials
+syncrepl-searchbase
+admin-email
+tls-cipher-suite
+tls-cert
+tls-privkey
+tls-ca

cdist/conf/type/__openldap_server/parameter/optional_multiple

@@ -0,0 +1,3 @@
+syncrepl-host
+module
+schema

cdist/conf/type/__openldap_server/parameter/required

@@ -0,0 +1,5 @@
+manager-dn
+manager-password
+manager-password-hash
+serverid
+suffix

cdist/conf/type/__openldap_server/parameter/required_multiple

@@ -0,0 +1 @@
+slapd-url

cdist/conf/type/__package_update_index/gencode-remote

@@ -47,9 +47,9 @@ case "$type" in
         echo "pacman --noprogressbar --sync --refresh"
         echo "pacman package database synced (age was: $currage)" >> "$__messages_out"
         ;;
-    alpine)
+    apk)
         echo "apk update"
-        echo "apk package database updated."
+        echo "apk package database updated." >>"$__messages_out"
         ;;
     *)
         echo "Don't know how to manage packages for type: $type" >&2

cdist/conf/type/__postgres_database/gencode-remote

@@ -41,12 +41,16 @@ if [ "$state_should" != "$state_is" ]; then
       present)
          owner=""
          if [ -f "$__object/parameter/owner" ]; then
-            owner="-O '$(cat "$__object/parameter/owner")'"
+            owner="-O \"$(cat "$__object/parameter/owner")\""
          fi
-         echo "su - '$postgres_user' -c \"createdb $owner '$name'\""
+         cat << EOF
+su - '$postgres_user' -c "createdb $owner \"$name\""
+EOF
       ;;
       absent)
-         echo "su - '$postgres_user' -c \"dropdb '$name'\""
+         cat << EOF
+su - '$postgres_user' -c "dropdb \"$name\""
+EOF
       ;;
    esac
 fi

cdist/conf/type/__postgres_role/gencode-remote

@@ -53,11 +53,13 @@ case "$state_should" in
         done
 
         [ -n "$password" ] && password="PASSWORD '$password'"
-
+        cat << EOF
-        cmd="CREATE ROLE $name WITH $password $booleans"
+su - '$postgres_user' -c "psql postgres -wc 'CREATE ROLE \"$name\" WITH $password $booleans;'"
-        echo "su - '$postgres_user' -c \"psql postgres -wc \\\"$cmd\\\"\""
+EOF
     ;;
     absent)
-        echo "su - '$postgres_user' -c \"dropuser \\\"$name\\\"\""
+        cat << EOF
+su - '$postgres_user' -c "dropuser \"$name\""
+EOF
     ;;
 esac

cdist/conf/type/__start_on_boot/man.rst

@@ -12,7 +12,7 @@ This cdist type allows you to enable or disable stuff to be started
 at boot of your operating system.
 
 Warning: This type has not been tested intensively and is not fully
-supported (i.e. \*BSD are not implemented).
+supported.
 
 
 REQUIRED PARAMETERS

cdist/config.py

@@ -124,6 +124,7 @@ class Config(object):
         """Remove files and directories for the run"""
         if self.remove_remote_files_dirs:
             self._remove_remote_files_dirs()
+        self.manifest.cleanup()
 
     @staticmethod
     def hosts(source):
@@ -368,7 +369,7 @@ class Config(object):
     def resolve_target_addresses(host, family):
         try:
             return ipaddr.resolve_target_addresses(host, family)
-        except:
+        except:  # noqa
             e = sys.exc_info()[1]
             raise cdist.Error(("Error resolving target addresses for host '{}'"
                                ": {}").format(host, e))
@@ -787,6 +788,9 @@ class Config(object):
         self.explorer.run_type_explorers(cdist_object, transfer_type_explorers)
         try:
             self.manifest.run_type_manifest(cdist_object)
+            self.log.trace("[ORDER_DEP] Removing order dep files for %s",
+                           cdist_object)
+            cdist_object.cleanup()
             cdist_object.state = core.CdistObject.STATE_PREPARED
         except cdist.Error as e:
             raise cdist.CdistObjectError(cdist_object, e)

cdist/core/cdist_object.py

@@ -243,6 +243,16 @@ class CdistObject(object):
             lambda obj: os.path.join(obj.base_path, obj.code_local_path))
     code_remote = fsproperty.FileStringProperty(
             lambda obj: os.path.join(obj.base_path, obj.code_remote_path))
+    typeorder = fsproperty.FileListProperty(
+            lambda obj: os.path.join(obj.absolute_path, 'typeorder'))
+    typeorder_dep = fsproperty.FileListProperty(
+            lambda obj: os.path.join(obj.absolute_path, 'typeorder_dep'))
+
+    def cleanup(self):
+        try:
+            os.remove(os.path.join(self.absolute_path, 'typeorder_dep'))
+        except FileNotFoundError:
+            pass
 
     @property
     def exists(self):

cdist/core/manifest.py

@@ -96,6 +96,10 @@ class Manifest(object):
     """Executes cdist manifests.
 
     """
+
+    ORDER_DEP_STATE_NAME = 'order_dep_state'
+    TYPEORDER_DEP_NAME = 'typeorder_dep'
+
     def __init__(self, target_host, local, dry_run=False):
         self.target_host = target_host
         self.local = local
@@ -212,3 +216,13 @@ class Manifest(object):
                     type_manifest,
                     env=self.env_type_manifest(cdist_object),
                     message_prefix=message_prefix)
+
+    def cleanup(self):
+        def _rm_file(fname):
+            try:
+                self.log.trace("[ORDER_DEP] Removing %s", fname)
+                os.remove(os.path.join(self.local.base_path, fname))
+            except FileNotFoundError:
+                pass
+        _rm_file(Manifest.ORDER_DEP_STATE_NAME)
+        _rm_file(Manifest.TYPEORDER_DEP_NAME)

cdist/emulator.py

@@ -29,6 +29,7 @@ import sys
 import cdist
 from cdist import core
 from cdist import flock
+from cdist.core.manifest import Manifest
 
 
 class MissingRequiredEnvironmentVariableError(cdist.Error):
@@ -44,7 +45,7 @@ class MissingRequiredEnvironmentVariableError(cdist.Error):
 class DefaultList(list):
     """Helper class to allow default values for optional_multiple parameters.
 
-       @see https://groups.google.com/forum/#!msg/comp.lang.python/sAUvkJEDpRc/RnRymrzJVDYJ
+       @see https://groups.google.com/forum/#!msg/comp.lang.python/sAUvkJEDpRc/RnRymrzJVDYJ  # noqa
     """
     def __copy__(self):
         return []
@@ -82,6 +83,11 @@ class Emulator(object):
         self.object_base_path = os.path.join(self.global_path, "object")
         self.typeorder_path = os.path.join(self.global_path, "typeorder")
 
+        self.typeorder_dep_path = os.path.join(self.global_path,
+                                               Manifest.TYPEORDER_DEP_NAME)
+        self.order_dep_state_path = os.path.join(self.global_path,
+                                                 Manifest.ORDER_DEP_STATE_NAME)
+
         self.type_name = os.path.basename(argv[0])
         self.cdist_type = core.CdistType(self.type_base_path, self.type_name)
 
@@ -206,6 +212,14 @@ class Emulator(object):
         return params
 
     def setup_object(self):
+        # CDIST_ORDER_DEPENDENCY state
+        order_dep_on = self._order_dep_on()
+        order_dep_defined = "CDIST_ORDER_DEPENDENCY" in self.env
+        if not order_dep_defined and order_dep_on:
+            self._set_order_dep_state_off()
+        if order_dep_defined and not order_dep_on:
+            self._set_order_dep_state_on()
+
         # Create object with given parameters
         self.parameters = {}
         for key, value in vars(self.args).items():
@@ -237,6 +251,20 @@ class Emulator(object):
             # record the created object in typeorder file
             with open(self.typeorder_path, 'a') as typeorderfile:
                 print(self.cdist_object.name, file=typeorderfile)
+            # record the created object in parent object typeorder file
+            __object_name = self.env.get('__object_name', None)
+            depname = self.cdist_object.name
+            if __object_name:
+                parent = self.cdist_object.object_from_name(__object_name)
+                parent.typeorder.append(self.cdist_object.name)
+                if self._order_dep_on():
+                    self.log.trace(('[ORDER_DEP] Adding %s to typeorder dep'
+                                    ' for %s'), depname, parent.name)
+                    parent.typeorder_dep.append(depname)
+            elif self._order_dep_on():
+                self.log.trace('[ORDER_DEP] Adding %s to global typeorder dep',
+                               depname)
+                self._add_typeorder_dep(depname)
 
         # Record / Append source
         self.cdist_object.source.append(self.object_source)
@@ -293,45 +321,73 @@ class Emulator(object):
 
         return cdist_object.name
 
+    def _order_dep_on(self):
+        return os.path.exists(self.order_dep_state_path)
+
+    def _set_order_dep_state_on(self):
+        self.log.trace('[ORDER_DEP] Setting order dep state on')
+        with open(self.order_dep_state_path, 'w'):
+            pass
+
+    def _set_order_dep_state_off(self):
+        self.log.trace('[ORDER_DEP] Setting order dep state off')
+        # remove order dep state file
+        try:
+            os.remove(self.order_dep_state_path)
+        except FileNotFoundError:
+            pass
+        # remove typeorder dep file
+        try:
+            os.remove(self.typeorder_dep_path)
+        except FileNotFoundError:
+            pass
+
+    def _add_typeorder_dep(self, name):
+        with open(self.typeorder_dep_path, 'a') as f:
+            print(name, file=f)
+
+    def _read_typeorder_dep(self):
+        try:
+            with open(self.typeorder_dep_path, 'r') as f:
+                return f.readlines()
+        except FileNotFoundError:
+            return []
+
     def record_requirements(self):
         """Record requirements."""
 
+        order_dep_on = self._order_dep_on()
+
         # Inject the predecessor, but not if its an override
         # (this would leed to an circular dependency)
-        if ("CDIST_ORDER_DEPENDENCY" in self.env and
+        if (order_dep_on and 'CDIST_OVERRIDE' not in self.env):
-                'CDIST_OVERRIDE' not in self.env):
+            try:
-            # load object name created befor this one from typeorder file ...
+                # __object_name is the name of the object whose type
-            with open(self.typeorder_path, 'r') as typecreationfile:
+                # manifest is currently executed
-                typecreationorder = typecreationfile.readlines()
+                __object_name = self.env.get('__object_name', None)
-                # get the type created before this one ...
+                # load object name created befor this one from typeorder
-                try:
+                # dep file
-                    lastcreatedtype = typecreationorder[-2].strip()
+                if __object_name:
-                    # __object_name is the name of the object whose type
+                    parent = self.cdist_object.object_from_name(
-                    # manifest is currently executed
+                        __object_name)
-                    __object_name = self.env.get('__object_name', None)
+                    typeorder = parent.typeorder_dep
-                    if lastcreatedtype == __object_name:
+                else:
-                        self.log.debug(("Not injecting require for "
+                    typeorder = self._read_typeorder_dep()
-                                        "CDIST_ORDER_DEPENDENCY: %s for %s,"
+                # get the type created before this one
-                                        " %s's type manifest is currently"
+                lastcreatedtype = typeorder[-2].strip()
-                                        " being executed"),
+                if 'require' in self.env:
-                                       lastcreatedtype,
+                    if lastcreatedtype not in self.env['require']:
-                                       self.cdist_object.name,
+                        self.env['require'] += " " + lastcreatedtype
-                                       lastcreatedtype)
+                else:
-                    else:
+                    self.env['require'] = lastcreatedtype
-                        if 'require' in self.env:
+                self.log.debug(("Injecting require for "
-                            appendix = " " + lastcreatedtype
+                                "CDIST_ORDER_DEPENDENCY: %s for %s"),
-                            if appendix not in self.env['require']:
+                               lastcreatedtype,
-                                self.env['require'] += appendix
+                               self.cdist_object.name)
-                        else:
+            except IndexError:
-                            self.env['require'] = lastcreatedtype
+                # if no second last line, we are on the first type,
-                        self.log.debug(("Injecting require for "
+                # so do not set a requirement
-                                        "CDIST_ORDER_DEPENDENCY: %s for %s"),
+                pass
-                                       lastcreatedtype,
-                                       self.cdist_object.name)
-                except IndexError:
-                    # if no second last line, we are on the first type,
-                    # so do not set a requirement
-                    pass
 
         reqs = set()
         if "require" in self.env:

cdist/exec/local.py

@@ -69,7 +69,6 @@ class Local(object):
 
         self.exec_path = exec_path
         self.custom_initial_manifest = initial_manifest
-        self._add_conf_dirs = add_conf_dirs
         self.cache_path_pattern = cache_path_pattern
         self.quiet_mode = quiet_mode
         if configuration:
@@ -84,16 +83,7 @@ class Local(object):
         self._init_cache_dir(None)
         self._init_paths()
         self._init_object_marker()
-        self._init_conf_dirs()
+        self._init_conf_dirs(add_conf_dirs)
-
-    @property
-    def dist_conf_dir(self):
-        return os.path.abspath(os.path.join(os.path.dirname(cdist.__file__),
-                                            "conf"))
-
-    @property
-    def home_dir(self):
-        return cdist.home_dir()
 
     def _init_log(self):
         self.log = logging.getLogger(self.target_host[0])
@@ -140,28 +130,9 @@ class Local(object):
         # Does not need to be secure - just randomly different from .cdist
         self.object_marker_name = tempfile.mktemp(prefix='.cdist-', dir='')
 
-    def _init_conf_dirs(self):
+    def _init_conf_dirs(self, add_conf_dirs):
-        self.conf_dirs = []
+        self.conf_dirs = util.resolve_conf_dirs(
-
+            self.configuration, add_conf_dirs=add_conf_dirs)
-        self.conf_dirs.append(self.dist_conf_dir)
-
-        # Is the default place for user created explorer, type and manifest
-        if self.home_dir:
-            self.conf_dirs.append(self.home_dir)
-
-        # Add directories defined in the CDIST_PATH environment variable
-        # if 'CDIST_PATH' in os.environ:
-        #     cdist_path_dirs = re.split(r'(?<!\\):', os.environ['CDIST_PATH'])
-        #     cdist_path_dirs.reverse()
-        #     self.conf_dirs.extend(cdist_path_dirs)
-        if 'conf_dir' in self.configuration:
-            conf_dirs = self.configuration['conf_dir']
-            if conf_dirs:
-                self.conf_dirs.extend(conf_dirs)
-
-        # Add command line supplied directories
-        if self._add_conf_dirs:
-            self.conf_dirs.extend(self._add_conf_dirs)
 
     def _init_directories(self):
         self.mkdir(self.conf_path)
@@ -187,10 +158,11 @@ class Local(object):
             self.object_marker_name, self.object_marker_file))
 
     def _init_cache_dir(self, cache_dir):
+        home_dir = cdist.home_dir()
         if cache_dir:
             self.cache_path = cache_dir
-        elif self.home_dir:
+        elif home_dir:
-            self.cache_path = os.path.join(self.home_dir, "cache")
+            self.cache_path = os.path.join(home_dir, "cache")
         else:
             raise cdist.Error(
                 "No homedir setup and no cache dir location given")

cdist/exec/util.py

@@ -24,6 +24,7 @@ import os
 from tempfile import TemporaryFile
 
 import cdist
+import cdist.configuration
 
 
 # IMPORTANT:
@@ -175,3 +176,34 @@ def log_std_fd(log, command, stdfd, prefix):
         stdfd.seek(0, 0)
         log.trace("Command: {}; {}: {}".format(
             command, prefix, stdfd.read().decode()))
+
+
+def dist_conf_dir():
+    return os.path.abspath(os.path.join(os.path.dirname(cdist.__file__),
+                                        "conf"))
+
+
+def resolve_conf_dirs(configuration, add_conf_dirs):
+    conf_dirs = []
+
+    conf_dirs.append(dist_conf_dir())
+
+    home_dir = cdist.home_dir()
+    if home_dir:
+        conf_dirs.append(home_dir)
+
+    if 'conf_dir' in configuration:
+        x = configuration['conf_dir']
+        if x:
+            conf_dirs.extend(x)
+
+    if add_conf_dirs:
+        conf_dirs.extend(add_conf_dirs)
+    conf_dirs = set(conf_dirs)
+    return conf_dirs
+
+
+def resolve_conf_dirs_from_config_and_args(args):
+    cfg = cdist.configuration.Configuration(args)
+    configuration = cfg.get_config(section='GLOBAL')
+    return resolve_conf_dirs(configuration, args.conf_dir)

cdist/info.py

@@ -0,0 +1,181 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+#
+# 2019-2020 Darko Poljak (darko.poljak at gmail.com)
+#
+# This file is part of cdist.
+#
+# cdist is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# cdist is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with cdist. If not, see <http://www.gnu.org/licenses/>.
+#
+#
+
+import cdist
+import cdist.configuration
+import cdist.core
+import cdist.exec.util as util
+import os
+import glob
+import fnmatch
+
+
+class Info(object):
+
+    def __init__(self, conf_dirs, args):
+        self.conf_dirs = conf_dirs
+        self.all = args.all
+        self.display_global_explorers = args.global_explorers
+        self.display_types = args.types
+        if not self.display_global_explorers and not self.display_types:
+            self.all = True
+        self.fixed_string = args.fixed_string
+        self._setup_glob_pattern(args.pattern)
+        self.full = args.full
+
+    def _setup_glob_pattern(self, pattern):
+        if pattern is None:
+            self.glob_pattern = '*'
+        elif ('?' in pattern or '*' in pattern or '[' in pattern or
+              self.fixed_string):
+            self.glob_pattern = pattern
+        else:
+            self.glob_pattern = '*' + pattern + '*'
+
+    @classmethod
+    def commandline(cls, args):
+        conf_dirs = util.resolve_conf_dirs_from_config_and_args(args)
+        c = cls(conf_dirs, args)
+        c.run()
+
+    def _get_global_explorers(self, conf_path):
+        rv = []
+        global_explorer_path = os.path.join(conf_path, "explorer",
+                                            self.glob_pattern)
+        if self.fixed_string:
+            if os.path.exists(global_explorer_path):
+                rv.append(global_explorer_path)
+        else:
+            for explorer in glob.glob(global_explorer_path):
+                rv.append(explorer)
+        return rv
+
+    def _should_display_type(self, dir_entry):
+        if not dir_entry.is_dir():
+            return False
+        if self.glob_pattern is None:
+            return True
+        if self.fixed_string:
+            return dir_entry.name == self.glob_pattern
+        else:
+            return fnmatch.fnmatch(dir_entry.name, self.glob_pattern)
+
+    def _get_types(self, conf_path):
+        rv = []
+        types_path = os.path.join(conf_path, "type")
+        if not os.path.exists(types_path):
+            return rv
+        with os.scandir(types_path) as it:
+            for entry in it:
+                if self._should_display_type(entry):
+                    rv.append(entry.path)
+        return rv
+
+    def _display_details(self, title, details, default_values=None,
+                         deprecated=None):
+        if not details:
+            return
+        if isinstance(details, bool):
+            print("\t{}: {}".format(title, 'yes' if details else 'no'))
+        elif isinstance(details, str):
+            print("\t{}: {}".format(title, details))
+        elif isinstance(details, list):
+            dv = dict(default_values) if default_values else {}
+            dp = dict(deprecated) if deprecated else {}
+
+            print("\t{}:".format(title))
+            for x in sorted(details):
+                print("\t\t{}".format(x), end='')
+                has_default = x in dv
+                is_deprecated = x in dp
+                need_comma = False
+                if has_default or is_deprecated:
+                    print(" (", end='')
+                if has_default:
+                    print("default: {}".format(dv[x]), end='')
+                    need_comma = True
+                if is_deprecated:
+                    print("{}deprecated".format(', ' if need_comma else ''),
+                          end='')
+                if has_default or is_deprecated:
+                    print(")", end='')
+                print()
+
+    def _display_type_parameters(self, cdist_type):
+        self._display_details("required parameters",
+                              cdist_type.required_parameters,
+                              default_values=cdist_type.parameter_defaults,
+                              deprecated=cdist_type.deprecated_parameters)
+        self._display_details("required multiple parameters",
+                              cdist_type.required_multiple_parameters,
+                              default_values=cdist_type.parameter_defaults,
+                              deprecated=cdist_type.deprecated_parameters)
+        self._display_details("optional parameters",
+                              cdist_type.optional_parameters,
+                              default_values=cdist_type.parameter_defaults,
+                              deprecated=cdist_type.deprecated_parameters)
+        self._display_details("optional multiple parameters",
+                              cdist_type.optional_multiple_parameters,
+                              default_values=cdist_type.parameter_defaults,
+                              deprecated=cdist_type.deprecated_parameters)
+        self._display_details("boolean parameters",
+                              cdist_type.boolean_parameters,
+                              default_values=cdist_type.parameter_defaults,
+                              deprecated=cdist_type.deprecated_parameters)
+
+    def _display_type_characteristics(self, cdist_type):
+        characteristics = []
+        if cdist_type.is_install:
+            characteristics.append('install')
+        else:
+            characteristics.append('config')
+        if cdist_type.is_singleton:
+            characteristics.append('singleton')
+        if cdist_type.is_nonparallel:
+            characteristics.append('nonparallel')
+        else:
+            characteristics.append('parallel')
+        if cdist_type.deprecated is not None:
+            characteristics.append('deprecated')
+        print("\t{}".format(', '.join(characteristics)))
+
+    def _display_type_details(self, type_path):
+        dirname, basename = os.path.split(type_path)
+        cdist_type = cdist.core.CdistType(dirname, basename)
+
+        self._display_type_characteristics(cdist_type)
+        self._display_type_parameters(cdist_type)
+
+    def run(self):
+        rv = []
+        for cp in self.conf_dirs:
+            conf_path = os.path.expanduser(cp)
+            if self.all or self.display_global_explorers:
+                rv.extend((x, 'E', ) for x in self._get_global_explorers(
+                    conf_path))
+            if self.all or self.display_types:
+                rv.extend((x, 'T', ) for x in self._get_types(conf_path))
+        rv = sorted(rv, key=lambda x: x[0])
+        for x, t in rv:
+            print(x)
+            if self.full and t == 'T':
+                self._display_type_details(x)

cdist/preos.py

@@ -5,6 +5,9 @@ import inspect
 import argparse
 import cdist
 import logging
+import cdist.argparse
+import cdist.configuration
+import cdist.exec.util as util
 
 
 _PREOS_CALL = "commandline"
@@ -12,17 +15,16 @@ _PREOS_NAME = "_preos_name"
 _PREOS_MARKER = "_cdist_preos"
 _PLUGINS_DIR = "preos"
 _PLUGINS_PATH = [os.path.join(os.path.dirname(__file__), _PLUGINS_DIR), ]
-cdist_home = cdist.home_dir()
-if cdist_home:
-    cdist_home_preos = os.path.join(cdist_home, "preos")
-    if os.path.isdir(cdist_home_preos):
-        _PLUGINS_PATH.append(cdist_home_preos)
-sys.path.extend(_PLUGINS_PATH)
-
-
 log = logging.getLogger("PreOS")
 
 
+def extend_plugins_path(dirs):
+    for dir in dirs:
+        preos_dir = os.path.expanduser(os.path.join(dir, "preos"))
+        if os.path.isdir(preos_dir):
+            _PLUGINS_PATH.append(preos_dir)
+
+
 def preos_plugin(obj):
     """It is preos if _PREOS_MARKER is True and has _PREOS_CALL."""
     if hasattr(obj, _PREOS_MARKER):
@@ -71,31 +73,58 @@ def check_root():
         raise cdist.Error("Must be run with root privileges")
 
 
+def get_available_preoses_string(cls):
+    preoses = ['    - {}'.format(x) for x in sorted(set(cls.preoses))]
+    return "Available PreOS-es:\n{}".format("\n".join(preoses))
+
+
 class PreOS(object):
     preoses = None
 
     @classmethod
     def commandline(cls, argv):
-
+        cdist_parser = cdist.argparse.get_parsers()
-        if not cls.preoses:
-            cls.preoses = find_preoses()
-
         parser = argparse.ArgumentParser(
-            description="Create PreOS", prog="cdist preos")
+            description="Create PreOS", prog="cdist preos",
-        parser.add_argument('preos', help='PreOS to create, one of: {}'.format(
+            parents=[cdist_parser['loglevel'], ])
-            set(cls.preoses)))
+        parser.add_argument('preos', help='PreOS to create',
-        args = parser.parse_args(argv[1:2])
+                            nargs='?', default=None)
+        parser.add_argument('-c', '--conf-dir',
+                            help=('Add configuration directory (one that '
+                                  'contains "preos" subdirectory)'),
+                            action='append')
+        parser.add_argument('-g', '--config-file',
+                            help='Use specified custom configuration file.',
+                            dest="config_file", required=False)
+        parser.add_argument('-L', '--list-preoses',
+                            help='List available PreOS-es',
+                            action='store_true', default=False)
+        parser.add_argument('remainder_args', nargs=argparse.REMAINDER)
+        args = parser.parse_args(argv[1:])
+        cdist.argparse.handle_loglevel(args)
+        log.debug("preos args : {}".format(args))
+
+        conf_dirs = util.resolve_conf_dirs_from_config_and_args(args)
+
+        extend_plugins_path(conf_dirs)
+        sys.path.extend(_PLUGINS_PATH)
+        cls.preoses = find_preoses()
+
+        if args.list_preoses or not args.preos:
+            print(get_available_preoses_string(cls))
+            sys.exit(0)
 
         preos_name = args.preos
         if preos_name in cls.preoses:
             preos = cls.preoses[preos_name]
             func = getattr(preos, _PREOS_CALL)
             if inspect.ismodule(preos):
-                func_args = [preos, argv[2:], ]
+                func_args = [preos, args.remainder_args, ]
             else:
-                func_args = [argv[2:], ]
+                func_args = [args.remainder_args, ]
             log.info("Running preos : {}".format(preos_name))
             func(*func_args)
         else:
-            log.error("Unknown preos: {}, available preoses: {}".format(
+            raise cdist.Error(
-                preos_name, set(cls.preoses.keys())))
+                "Invalid PreOS {}. {}".format(
+                    preos_name, get_available_preoses_string(cls)))

cdist/test/emulator/__init__.py

@@ -24,8 +24,6 @@
 import io
 import os
 import shutil
-import string
-import filecmp
 import random
 import logging
 
@@ -34,7 +32,6 @@ from cdist import test
 from cdist.exec import local
 from cdist import emulator
 from cdist import core
-from cdist import config
 
 import os.path as op
 my_dir = op.abspath(op.dirname(__file__))
@@ -115,7 +112,7 @@ class EmulatorTestCase(test.CdistTestCase):
     def test_requirement_pattern(self):
         argv = ['__file', '/tmp/foobar']
         self.env['require'] = '__file/etc/*'
-        emu = emulator.Emulator(argv, env=self.env)
+        emulator.Emulator(argv, env=self.env)
         # if we get here all is fine
 
     def test_loglevel(self):
@@ -172,6 +169,44 @@ class EmulatorTestCase(test.CdistTestCase):
         self.assertEqual(list(file_object.requirements), ['__planet/mars'])
         # if we get here all is fine
 
+    def test_order_dependency_context(self):
+        test_seq = ('A', True, 'B', 'C', 'D', False, 'E', 'F', True, 'G',
+                    'H', False, 'I', )
+        expected_requirements = {
+            'C': set(('__planet/B', )),
+            'D': set(('__planet/C', )),
+            'H': set(('__planet/G', )),
+        }
+        # Ensure env var is not in env
+        if 'CDIST_ORDER_DEPENDENCY' in self.env:
+            del self.env['CDIST_ORDER_DEPENDENCY']
+
+        for x in test_seq:
+            if isinstance(x, str):
+                # Clear because of order dep injection
+                # In real world, this is not shared over instances
+                if 'require' in self.env:
+                    del self.env['require']
+                argv = ['__planet', x]
+                emu = emulator.Emulator(argv, env=self.env)
+                emu.run()
+            elif isinstance(x, bool):
+                if x:
+                    self.env['CDIST_ORDER_DEPENDENCY'] = 'on'
+                elif 'CDIST_ORDER_DEPENDENCY' in self.env:
+                    del self.env['CDIST_ORDER_DEPENDENCY']
+        cdist_type = core.CdistType(self.local.type_path, '__planet')
+        for x in test_seq:
+            if isinstance(x, str):
+                obj = core.CdistObject(cdist_type, self.local.object_path,
+                                       self.local.object_marker_name, x)
+                reqs = set(obj.requirements)
+                if x in expected_requirements:
+                    self.assertEqual(reqs, expected_requirements[x])
+                else:
+                    self.assertTrue(len(reqs) == 0)
+        # if we get here all is fine
+
 
 class EmulatorConflictingRequirementsTestCase(test.CdistTestCase):
 

docs/changelog

@@ -1,6 +1,33 @@
 Changelog
 ---------
 
+next:
+	* Type __acl: Add --entry parameter to replace --acl, deprecate --acl (Ander Punnar)
+	* Core: preos: Fix missing configuration file usage, support -g, --config-file option (Darko Poljak)
+	* Core info command: Support tilde expansion of conf directories (Darko Poljak)
+
+6.4.0: 2020-01-04
+	* Type __consul_agent: Don't deploy init script on Alpine anymore, it ships with one itself (Nico Schottelius)
+	* Type __install_chroot_umount: Bugfix: type was not using __chroot_umount/manifest (Steven Armstrong)
+	* Types __postgres_*: Use double quoted identifiers in generated SQL (Timothée Floure)
+	* Core: Add cdist info command (Darko Poljak)
+	* New types: __mysql_user, __mysql_privileges (Ander Punnar)
+	* Type __mysql_database: Rewrite (Ander Punnar)
+
+6.3.0: 2019-12-12
+	* Type __package_update_index: Fix Alpine part (Dominique Roux)
+	* Documentation: Fix man pages for install types (Darko Poljak)
+	* Documentation: Embed config skeleton instead of rewriting it (Darko Poljak)
+	* Documentation: Remove cdist-type prefix and man page reference from type list in html (Darko Poljak)
+	* Documentation: PreOS english nitpicking (Evil Ham)
+	* Documentation: Add installing from source with signature verification (Darko Poljak)
+	* Core: preos: Support top command logging options, custom conf-dir option and CDIST_PATH env var (Darko Poljak)
+	* Type __start_on_boot: Docs: remove unsupported *BSD claim (Evil Ham)
+	* New type: __openldap_server (Evil Ham)
+
+6.2.0: 2019-11-30
+	* Core: Redefine/reimplement/fix CDIST_ORDER_DEPENDENCY (Darko Poljak)
+
 6.1.1: 2019-11-26
 	* Type __xymon_config: Add parameters for ownership and generic rsync options (Thomas Eckert)
 	* Type __xymon_client: Add msgcache parameter to support passive clients (Thomas Eckert)

docs/src/cdist-best-practice.rst

@@ -226,8 +226,8 @@ and also to store all important files in one
 repository.
 
 
-Perils of CDIST_ORDER_DEPENDENCY
+Notes on CDIST_ORDER_DEPENDENCY
---------------------------------
+-------------------------------
 With CDIST_ORDER_DEPENDENCY all types are executed in the order in which they
 are created in the manifest. The current created object automatically depends
 on the previously created object.
@@ -235,96 +235,11 @@ on the previously created object.
 It essentially helps you to build up blocks of code that build upon each other
 (like first creating the directory xyz than the file below the directory).
 
-This can be helpful, but it can also be the source of *evil*.
+This can be helpful, but one must be aware of its side effects.
-
-
-CDIST_ORDER_DEPENDENCY easily causes unobvious dependency cycles
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Let's see an example. Suppose you have special init manifest where among other
-things you are assuring that remote host has packages `sudo` and `curl`
-installed.
-
-**init1**
-
-.. code-block:: sh
-
-   CDIST_ORDER_DEPENDENCY=1
-   export CDIST_ORDER_DEPENDENCY
-
-   for p in sudo curl
-   do
-      __package "${p}"
-   done
-
-Then you have some other special init manifest where among other things you are
-assuring `sudo` package is installed.
-
-**init2**
-
-.. code-block:: sh
-
-   CDIST_ORDER_DEPENDENCY=1
-   export CDIST_ORDER_DEPENDENCY
-
-   __package sudo
-
-Then you have third init manifest where you combine those two init manifests,
-by including them:
-
-**init**
-
-.. code-block:: sh
-
-   sh -e "$__manifest/init1"
-   sh -e "$__manifest/init2"
-
-The resulting init manifest is then equal to:
-
-.. code-block:: sh
-
-   CDIST_ORDER_DEPENDENCY=1
-   export CDIST_ORDER_DEPENDENCY
-
-   for p in sudo curl
-   do
-      __package "${p}"
-   done
-
-   CDIST_ORDER_DEPENDENCY=1
-   export CDIST_ORDER_DEPENDENCY
-
-   __package sudo
-
-In the end you get the following dependencies:
-
-* `__package/curl` depends on `__package/sudo`
-* `__package/sudo` depends on `__package/curl`
-
-And here you have a circular dependency!
-
-In the real world manifest can be quite complex, dependencies can become
-complicated and circual dependencies are not so obvious. Resolving it can
-become cumbersome.
-
-**Practical solution?**
-
-Instead of managing complex init manifests you can write custom types.
-Each custom type can do one thing, it has well defined dependencies that will
-not leak into init manifest. In custom type you can also add special explorers
-and gencode.
-
-Then, in init manifest you combine your complex types. It is:  
-
-* cleaner
-* easier to follow
-* easier to maintain
-* easier to debug.
 
 
 CDIST_ORDER_DEPENDENCY kills parallelization
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
 Suppose you have defined CDIST_ORDER_DEPENDENCY and then, among other things,
 you specify creation of three, by nature independent, files.
 

docs/src/cdist-configuration.rst

@@ -31,73 +31,11 @@ can be used.
 
 Config file format
 ------------------
+
 cdist configuration file is in the INI file format. Currently it supports
 only [GLOBAL] section.
-The possible keywords and their meanings are as follows:
 
-:strong:`archiving`
+Here you can find configuration file skeleton:
-    Use specified archiving. Valid values include:
-    'none', 'tar', 'tgz', 'tbz2' and 'txz'.
 
-:strong:`beta`
+.. literalinclude:: cdist.cfg.skeleton
-    Enable beta functionality. It recognizes boolean values from
+    :language: ini
-    'yes'/'no', 'on'/'off', 'true'/'false' and '1'/'0'.
-
-:strong:`cache_path_pattern`
-    Specify cache path pattern.
-
-:strong:`conf_dir`
-    List of configuration directories separated with the character conventionally
-    used by the operating system to separate search path components (as in PATH),
-    such as ':' for POSIX or ';' for Windows.
-    If also specified at command line then values from command line are
-    appended to this value.
-
-:strong:`init_manifest`
-    Specify default initial manifest.
-
-:strong:`inventory_dir`
-    Specify inventory directory.
-
-:strong:`jobs`
-    Specify number of jobs for parallel processing. If -1 then the default,
-    number of CPU's in the system is used. If 0 then parallel processing in
-    jobs is disabled. If set to positive number then specified maximum
-    number of processes will be used.
-
-:strong:`local_shell`
-    Shell command used for local execution.
-
-:strong:`out_path`
-    Directory to save cdist output in.
-
-:strong:`parallel`
-    Process hosts in parallel. If -1 then the default, number of CPU's in
-    the system is used. If 0 then parallel processing of hosts is disabled.
-    If set to positive number then specified maximum number of processes
-    will be used.
-
-:strong:`remote_copy`
-    Command to use for remote copy (should behave like scp).
-
-:strong:`remote_exec`
-    Command to use for remote execution (should behave like ssh).
-
-:strong:`remote_out_path`
-    Directory to save cdist output in on the target host.
-
-:strong:`remote_shell`
-    Shell command at remote host used for remote execution.
-
-:strong:`save_output_streams`
-    Enable/disable saving output streams (enabled by default).
-    It recognizes boolean values from 'yes'/'no', 'on'/'off', 'true'/'false'
-    and '1'/'0'.
-
-:strong:`timestamp`
-    Timestamp log messages with the current local date and time
-    in the format: YYYYMMDDHHMMSS.us.
-
-:strong:`verbosity`
-    Set verbosity level. Valid values are: 
-    'ERROR', 'WARNING', 'INFO', 'VERBOSE', 'DEBUG', 'TRACE' and 'OFF'.

docs/src/cdist-install.rst

@@ -23,8 +23,6 @@ Target Hosts
 Install cdist
 -------------
 
-You can install cdist either from git or as a python package.
-
 From git
 ~~~~~~~~
 
@@ -42,6 +40,7 @@ To install cdist, execute the following commands:
 
 From version 4.2.0 cdist tags and releases are signed.
 You can get GPG public key used for signing `here <_static/pgp-key-EFD2AE4EC36B6901.asc>`_.
+It is assumed that you are familiar with *git* ways of signing and verification.
 
 You can also get cdist from `github mirror <https://github.com/ungleich/cdist>`_.
 
@@ -157,3 +156,27 @@ Cdist is available as a python package at
 .. code-block:: sh
 
     pip install cdist
+
+Installing from source with signature verification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to install cdist from signed source and verify it, first you need to
+download cdist archive and its detached signature.
+
+Get both, *cdist-x.y.z.tar.gz* and *cdist-x.y.z.tar.gz.asc* from release
+notes of the desired tag *x.y.z* at
+`cdist git repository <https://code.ungleich.ch/ungleich-public/cdist/-/tags>`_.
+
+Get GPG public key used for signing `here <_static/pgp-key-EFD2AE4EC36B6901.asc>`_
+and import it into GPG.
+
+Now cdist source archive can be verified using `gpg`, e.g. to verify `cdist-6.2.0`:
+
+.. code-block:: sh
+
+    $ gpg --verify cdist-6.2.0.tar.gz.asc cdist-6.2.0.targ.gz
+    gpg: Signature made Sat Nov 30 23:14:19 2019 CET
+    gpg:                using RSA key 69767822F3ECC3C349C1EFFFEFD2AE4EC36B6901
+    gpg: Good signature from "ungleich GmbH (ungleich FOSS) <foss@ungleich.ch>" [ultimate]
+
+Further steps are the same as for `installing from git <cdist-install.html#from-git>`_.

docs/src/cdist-manifest.rst

@@ -163,7 +163,126 @@ automatically depends on the previously created object.
 It essentially helps you to build up blocks of code that build upon each other
 (like first creating the directory xyz than the file below the directory).
 
-Read also about `perils of CDIST_ORDER_DEPENDENCY <cdist-best-practice.html#perils-of-cdist-order-dependency>`_.
+Read also about `notes on CDIST_ORDER_DEPENDENCY <cdist-best-practice.html#notes-on-cdist-order-dependency>`_.
+
+In version 6.2.0 semantic CDIST_ORDER_DEPENDENCY is finally fixed and well defined.
+
+CDIST_ORDER_DEPENDENCY defines type order dependency context. Order dependency context
+starts when CDIST_ORDER_DEPENDENCY is set, and ends when it is unset. After each
+manifest execution finishes, any existing order dependency context is automatically
+unset. This ensures that CDIST_ORDER_DEPENDENCY is valid within the manifest where it
+is used. When order dependency context is defined then cdist executes types in the
+order in which they are created in the manifest inside order dependency context.
+
+Sometimes the best way to see how something works is to see examples.
+
+Suppose you have defined **initial manifest**:
+
+.. code-block:: sh
+
+    __cycle1 cycle1
+    export CDIST_ORDER_DEPENDENCY=1
+    __cycle2 cycle2
+    __cycle3 cycle3
+
+with types **__cycle1**:
+
+.. code-block:: sh
+
+    export CDIST_ORDER_DEPENDENCY=1
+    __file /tmp/cycle11
+    __file /tmp/cycle12
+    __file /tmp/cycle13
+
+**__cycle2**:
+
+.. code-block:: sh
+
+    __file /tmp/cycle21
+    export CDIST_ORDER_DEPENDENCY=1
+    __file /tmp/cycle22
+    __file /tmp/cycle23
+    unset CDIST_ORDER_DEPENDENCY
+    __file /tmp/cycle24
+
+**__cycle3**:
+
+.. code-block:: sh
+
+    __file /tmp/cycle31
+    __file /tmp/cycle32
+    export CDIST_ORDER_DEPENDENCY=1
+    __file /tmp/cycle33
+    __file /tmp/cycle34
+
+For the above config, cdist results in the following expected *dependency graph*
+(type *__cycleX* is shown as *cX*, *__file/tmp/cycleXY* is shown as *fcXY*):
+
+::
+
+    c1---->fc11
+    |      /\
+    |       |
+    +----->fc12
+    |      /\
+    |       |
+    +----->fc13
+
+    c2--+--->fc21
+    /\  |
+    |   |
+    |   +----->fc22
+    |   |      /\
+    |   |       |
+    |   +----->fc23
+    |   |
+    |   |
+    |   +----->fc24
+    |
+    |
+    c3---->fc31
+    |
+    |
+    +----->fc32
+    |
+    |
+    +----->fc33
+    |      /\
+    |       |
+    +----->fc34
+
+Before version 6.2.0 the above configuration would result in cycle:
+
+::
+
+    ERROR: 185.203.112.26: Cycle detected in object dependencies:
+    __file/tmp/cycle11 -> __cycle3/cycle3 -> __cycle2/cycle2 -> __cycle1/cycle1 -> __file/tmp/cycle11!
+
+The following manifest shows an example for order dependency contexts:
+
+.. code-block:: sh
+
+    __file /tmp/fileA
+    export CDIST_ORDER_DEPENDENCY=1
+    __file /tmp/fileB
+    __file /tmp/fileC
+    __file /tmp/fileD
+    unset CDIST_ORDER_DEPENDENCY
+    __file /tmp/fileE
+    __file /tmp/fileF
+    export CDIST_ORDER_DEPENDENCY=1
+    __file /tmp/fileG
+    __file /tmp/fileH
+    unset CDIST_ORDER_DEPENDENCY
+    __file /tmp/fileI
+
+This means:
+
+* C depends on B
+* D depends on C
+* H depends on G
+
+and there are no other dependencies from this manifest.
 
 
 Overrides

docs/src/cdist-preos.rst

@@ -4,9 +4,9 @@ PreOS
 Description
 -----------
 With cdist you can install and configure new machines. You can use cdist to
-create PreOS, minimal OS which purpose is to boot new machine.
+create PreOS, minimal OS whose purpose is to boot a new machine.
-After PreOS is booted machine is ready for installing desired OS and
+After PreOS is booted, the machine is ready for installing the desired OS and
-then it is ready for configuration.
+afterwards it is ready for configuration.
 
 PreOS creation
 --------------
@@ -15,62 +15,62 @@ Currently supported PreOS-es include:
 
 * debian
 * ubuntu
-* devuan.
+* devuan
 
-PreOS is created using cdist preos command. preos command has subcommands that
+PreOS is created using the ``cdist preos`` command.
-create the desired PreOS.
+This command has subcommands that determine the desired PreOS.
 
-For example, to create ubuntu PreOS:
+For example, to create an ubuntu PreOS:
 
 .. code-block:: sh
 
     $ cdist preos ubuntu /preos/preos-ubuntu -B -C \
         -k ~/.ssh/id_rsa.pub -p /preos/pxe-ubuntu
 
-For more info about available options see cdist manual page.
+For more info about the available options see the cdist manual page.
 
-This will bootstrap (``-B``) ubuntu PreOS in ``/preos/preos-ubuntu`` directory, it
+This will bootstrap (``-B``) ubuntu PreOS in the ``/preos/preos-ubuntu``
-will be configured (``-C``) using default built-in initial manifest and with
+directory, it will be configured (``-C``) using default the built-in initial
-specified ssh authorized key (``-k``).
+manifest and with specified ssh authorized key (``-k``).
-After bootstrapping and configuration PXE
+After bootstrapping and configuration, the PXE boot directory will be
-boot directory will be created (``-p``) in ``/preos/pxe-ubuntu``.
+created (``-p``) in ``/preos/pxe-ubuntu``.
 
-After PreOS is created new machines can be booted using created PXE (after
+After PreOS is created, new machines can be booted using the created PXE
-proper dhcp, tftp setting).
+(after proper dhcp and tftp settings).
 
-Since PreOS is configured with ssh authorized key it can be accessed throguh
+Since PreOS is configured with ssh authorized key it can be accessed through
 ssh, i.e. it can be further installed and configured with cdist.
 
-Implementing new PreOS sub-command
+Implementing a new PreOS sub-command
-----------------------------------
+------------------------------------
 preos command is implemented as a plugin system. This plugin system scans for
-preos subcommands in ``cdist/preos/`` distribution directory and also in
+preos subcommands in the ``cdist/preos/`` distribution directory and also in
 ``~/.cdist/preos/`` directory if it exists.
 
 preos subcommand is a module or a class that satisfies the following:
 
-* it has attribute ``_cdist_preos`` set to ``True``
+* it has the attribute ``_cdist_preos`` set to ``True``
-* it has function/method ``commandline``.
+* it defines a function/method ``commandline``.
 
-For a module based preos subcommand ``commandline`` function accepts a module
+For a module-based preos subcommand, the ``commandline`` function accepts a
-object as its first argument and the list of command line
+module object as its first argument and the list of command line
 arguments (``sys.argv[2:]``).
 
-For a class preos subcommand ``commandline`` method should be staticmethod and
+For a class-based preos subcommand ``commandline`` method should be
-it accepts a class object as its first argument and the list of command line
+static-method and must accept a class as its first argument and the
-arguments(``sys.argv[2:]``).
+list of command line arguments (``sys.argv[2:]``).
 
 If preos scanning finds a module/class that has ``_cdist_preos`` set
-to ``True`` and it has function/method ``commandline`` then this module/class is
+to ``True`` and a function/method ``commandline`` then this module/class is
 registered to preos subcommands. The name of the command is set to ``_preos_name``
-attribute if it exists, otherwise it is set to the module/class name, lowercase.
+attribute if defined in the module/class, defaulting to the module/class name in lowercase.
-When registered preos subcommand is specified as preos command then ``commandline``
+When a registered preos subcommand is specified, ``commandline``
-will be called with first argument set to module/class object and second argument
+will be called with the first argument set to module/class and the second
-set to ``sys.argv[2:]``.
+argument set to ``sys.argv[2:]``.
 
-Example writing new dummy preos sub-command
+Example of writing new dummy preos sub-command
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Module based preos:
+Module-based preos:
 ^^^^^^^^^^^^^^^^^^^
 
 #. Create directory ``~/.cdist/preos/`` if it does not exist
@@ -88,16 +88,12 @@ When you try to run this new preos you will get:
 
 .. code-block:: sh
 
-    $ cdist preos -h
+    $ cdist preos -L
-    usage: cdist preos [-h] preos
+    Available PreOS-es:
-
+        - debian
-    Create PreOS
+        - devuan
-
+        - netbsd
-    positional arguments:
+        - ubuntu
-      preos       PreOS to create, one of: {'netbsd', 'debian', 'ubuntu'}
-
-    optional arguments:
-      -h, --help  show this help message and exit
     $ cdist preos netbsd
     NetBSD PreOS: []
 
@@ -121,17 +117,13 @@ When you try to run this new preos you will get:
 .. code-block:: sh
 
     $ cdist preos -h
-    usage: cdist preos [-h] preos
+    Available PreOS-es:
-
+        - debian
-    Create PreOS
+        - devuan
-
+        - freebsd
-    positional arguments:
+        - ubuntu
-      preos       PreOS to create, one of: {'freebsd', 'debian', 'ubuntu'}
-
-    optional arguments:
-      -h, --help  show this help message and exit
     $ cdist preos freebsd
     FreeBSD dummy preos: []
 
 In the ``commandline`` function/method you have all the freedom to actually create
-PreOS.
+a PreOS.

docs/src/cdist-reference.rst.sh

@@ -330,7 +330,7 @@ CDIST_OVERRIDE
 
 CDIST_ORDER_DEPENDENCY
     Create dependencies based on the execution order (see  \`cdist manifest <cdist-manifest.html>\`_).
-    Read also about \`perils of CDIST_ORDER_DEPENDENCY <cdist-best-practice.html#perils-of-cdist-order-dependency>\`_.
+    Note that in version 6.2.0 semantic of this processing mode is finally fixed and well defined.
 
 CDIST_REMOTE_EXEC
     Use this command for remote execution (should behave like ssh).

docs/src/cdist-types.rst

@@ -1,8 +0,0 @@
-cdist types
-===========
-
-.. toctree::
-   :titlesonly:
-   :glob:
-
-   man7/*

docs/src/cdist-types.rst.sh

@@ -0,0 +1,55 @@
+#!/bin/sh
+#
+# 2019 Darko Poljak (darko.poljak at gmail.com) 
+#
+# This file is part of cdist.
+#
+# cdist is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# cdist is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with cdist. If not, see <http://www.gnu.org/licenses/>.
+#
+#
+# Generate cdist-types.rst that lists available types.
+#
+
+__cdist_pwd="$(pwd -P)"
+__cdist_mydir="${0%/*}";
+__cdist_abs_mydir="$(cd "$__cdist_mydir" && pwd -P)"
+__cdist_myname=${0##*/};
+__cdist_abs_myname="$__cdist_abs_mydir/$__cdist_myname"
+
+filename="${__cdist_myname%.sh}"
+dest="$__cdist_abs_mydir/$filename"
+
+cd "$__cdist_abs_mydir"
+
+exec > "$dest"
+cat << eof 
+cdist types
+===========
+
+.. toctree::
+   :titlesonly:
+
+eof
+
+# If there is no such file then ls prints error to stderr,
+# so redirect stderr to /dev/null.
+for type in $(ls man7/cdist-type__*.rst 2>/dev/null | LC_ALL=C sort); do
+    no_dir="${type#man7/}";
+    no_type="${no_dir#cdist-type}";
+    name="${no_type%.rst}";
+    manref="${no_dir%.rst}"
+    man="${manref}(7)"
+
+    echo "   $name" "<man7/${manref}>"
+done

docs/src/man1/cdist.rst

@@ -11,7 +11,7 @@ SYNOPSIS
 
 ::
 
-    cdist [-h] [-V] {banner,config,install,inventory,preos,shell} ...
+    cdist [-h] [-V] {banner,config,install,inventory,preos,shell,info} ...
 
     cdist banner [-h] [-l LOGLEVEL] [-q] [-v]
 
@@ -59,31 +59,35 @@ SYNOPSIS
                          [-I INVENTORY_DIR] [-a] [-f HOSTFILE] [-H] [-t]
                          [host [host ...]]
 
-    cdist preos [-h] preos
+    cdist preos [-h] [-l LOGLEVEL] [-q] [-v] [-c CONF_DIR] [-g CONFIG_FILE]
+                [-L]
+                [preos] ...
 
-    cdist preos debian [-h] [-l LOGLEVEL] [-q] [-v] [-b] [-a ARCH] [-B]
+    cdist preos [preos-options] debian [-h] [-l LOGLEVEL] [-q] [-v] [-b] [-a ARCH] [-B]
-                       [-C] [-c CDIST_PARAMS] [-D DRIVE] [-e REMOTE_EXEC]
+                                       [-C] [-c CDIST_PARAMS] [-D DRIVE] [-e REMOTE_EXEC]
-                       [-i MANIFEST] [-k KEYFILE ] [-m MIRROR]
+                                       [-i MANIFEST] [-k KEYFILE ] [-m MIRROR]
-                       [-P ROOT_PASSWORD] [-p PXE_BOOT_DIR] [-r]
+                                       [-P ROOT_PASSWORD] [-p PXE_BOOT_DIR] [-r]
-                       [-S SCRIPT] [-s SUITE] [-y REMOTE_COPY]
+                                       [-S SCRIPT] [-s SUITE] [-y REMOTE_COPY]
-                       target_dir
+                                       target_dir
 
-    cdist preos devuan [-h] [-l LOGLEVEL] [-q] [-v] [-b] [-a ARCH] [-B]
+    cdist preos [preos-options] devuan [-h] [-l LOGLEVEL] [-q] [-v] [-b] [-a ARCH] [-B]
-                       [-C] [-c CDIST_PARAMS] [-D DRIVE] [-e REMOTE_EXEC]
+                                       [-C] [-c CDIST_PARAMS] [-D DRIVE] [-e REMOTE_EXEC]
-                       [-i MANIFEST] [-k KEYFILE ] [-m MIRROR]
+                                       [-i MANIFEST] [-k KEYFILE ] [-m MIRROR]
-                       [-P ROOT_PASSWORD] [-p PXE_BOOT_DIR] [-r]
+                                       [-P ROOT_PASSWORD] [-p PXE_BOOT_DIR] [-r]
-                       [-S SCRIPT] [-s SUITE] [-y REMOTE_COPY]
+                                       [-S SCRIPT] [-s SUITE] [-y REMOTE_COPY]
-                       target_dir
+                                       target_dir
 
-    cdist preos ubuntu [-h] [-l LOGLEVEL] [-q] [-v] [-b] [-a ARCH] [-B]
+    cdist preos [preos-options] ubuntu [-h] [-l LOGLEVEL] [-q] [-v] [-b] [-a ARCH] [-B]
-                       [-C] [-c CDIST_PARAMS] [-D DRIVE] [-e REMOTE_EXEC]
+                                       [-C] [-c CDIST_PARAMS] [-D DRIVE] [-e REMOTE_EXEC]
-                       [-i MANIFEST] [-k KEYFILE ] [-m MIRROR]
+                                       [-i MANIFEST] [-k KEYFILE ] [-m MIRROR]
-                       [-P ROOT_PASSWORD] [-p PXE_BOOT_DIR] [-r]
+                                       [-P ROOT_PASSWORD] [-p PXE_BOOT_DIR] [-r]
-                       [-S SCRIPT] [-s SUITE] [-y REMOTE_COPY]
+                                       [-S SCRIPT] [-s SUITE] [-y REMOTE_COPY]
-                       target_dir
+                                       target_dir
 
     cdist shell [-h] [-l LOGLEVEL] [-q] [-v] [-s SHELL]
 
+    cdist info [-h] [-a] [-c CONF_DIR] [-e] [-F] [-f] [-g CONFIG_FILE] [-t] [pattern]
+
 
 DESCRIPTION
 -----------
@@ -103,7 +107,7 @@ All commands accept the following options:
 **-l LOGLEVEL, --log-level LOGLEVEL**
     Set the specified verbosity level. The levels, in
     order from the lowest to the highest, are: ERROR (-1),
-    WARNING (0), INFO (1), VERBOSE (2), DEBUG (3) TRACE (4
+    WARNING (0), INFO (1), VERBOSE (2), DEBUG (3), TRACE (4
     or higher). If used along with -v then -v increases
     last set value and -l overwrites last set value.
 
@@ -116,7 +120,7 @@ All commands accept the following options:
     value is 0 which includes ERROR and WARNING levels.
     The levels, in order from the lowest to the highest,
     are: ERROR (-1), WARNING (0), INFO (1), VERBOSE (2),
-    DEBUG (3) TRACE (4 or higher). If used along with -l
+    DEBUG (3), TRACE (4 or higher). If used along with -l
     then -l overwrites last set value and -v increases
     last set value.
 
@@ -457,7 +461,18 @@ List inventory database.
 
 PREOS
 -----
-Create PreOS. Currently, the following PreOS-es are supported:
+Create PreOS.
+
+**-c CONF_DIR, --conf-dir CONF_DIR**
+    Add configuration directory (one that contains "preos" subdirectory).
+
+**-g CONFIG_FILE, --config-file CONFIG_FILE**
+    Use specified custom configuration file.
+
+**-L, --list-preoses**
+    List available PreOS-es.
+
+Currently, the following PreOS-es are supported:
 
 * debian
 * ubuntu
@@ -596,6 +611,39 @@ usage. Its primary use is for debugging type parameters.
     be POSIX compatible shell.
 
 
+INFO
+----
+Display information for cdist (global explorers, types).
+
+**pattern**
+    Glob pattern. If it contains special characters('?', '*', '[') then it is
+    used as specified, otherwise it is translated to `*pattern*`.
+
+**-h, --help**
+    Show help message and exit.
+
+**-a, --all**
+    Display all info. This is the default.
+
+**-c CONF_DIR, --conf-dir CONF_DIR**
+    Add configuration directory (can be repeated).
+
+**-e, --global-explorers**
+    Display info for global explorers.
+
+**-F, --fixed-string**
+    Interpret pattern as a fixed string.
+
+**-f, --full**
+    Display full details.
+
+**-g CONFIG_FILE, --config-file CONFIG_FILE**
+    Use specified custom configuration file.
+
+**-t, --types**
+    Display info for types.
+
+
 CONFIGURATION
 -------------
 cdist obtains configuration data from the following sources in the following
@@ -827,6 +875,8 @@ CDIST_OVERRIDE
 
 CDIST_ORDER_DEPENDENCY
     Create dependencies based on the execution order.
+    Note that in version 6.2.0 semantic of this processing mode is
+    finally fixed and well defined.
 
 CDIST_REMOTE_EXEC
     Use this command for remote execution (should behave like ssh).
@@ -905,5 +955,5 @@ such case and display a warning message. An example of such a case:
 
 COPYING
 -------
-Copyright \(C) 2011-2017 Nico Schottelius. Free use of this software is
+Copyright \(C) 2011-2019 Nico Schottelius. Free use of this software is
 granted under the terms of the GNU General Public License v3 or later (GPLv3+).

setup.py

@@ -21,7 +21,7 @@ else:
         raise DistutilsError("Missing version file {}".format(version_file))
 
 
-import cdist
+import cdist  # noqa
 
 
 def data_finder(data_dir):
@@ -66,7 +66,7 @@ setup(
         "Development Status :: 6 - Mature",
         "Environment :: Console",
         "Intended Audience :: System Administrators",
-        "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",
+        "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",  # noqa
         "Operating System :: MacOS :: MacOS X",
         "Operating System :: POSIX",
         "Operating System :: POSIX :: BSD",