From d5ac9ea348aa0ed640941232fbc2c783d95b8750 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Sun, 8 Dec 2019 16:34:38 +0100 Subject: [PATCH 01/38] Docs: install from source, verify signature Add chapter for installing from source with signature verification. Resolves #795. --- docs/changelog | 1 + docs/src/cdist-install.rst | 27 +++++++++++++++++++++++++-- 2 files changed, 26 insertions(+), 2 deletions(-) diff --git a/docs/changelog b/docs/changelog index f4e38578..33db10da 100644 --- a/docs/changelog +++ b/docs/changelog @@ -7,6 +7,7 @@ next: * Documentation: Embed config skeleton instead of rewriting it (Darko Poljak) * Documentation: Remove cdist-type prefix and man page reference from type list in html (Darko Poljak) * Documentation: PreOS english nitpicking (Evil Ham) + * Documentation: Add installing from source with signature verification (Darko Poljak) 6.2.0: 2019-11-30 * Core: Redefine/reimplement/fix CDIST_ORDER_DEPENDENCY (Darko Poljak) diff --git a/docs/src/cdist-install.rst b/docs/src/cdist-install.rst index a9b7d6b5..880b9f8e 100644 --- a/docs/src/cdist-install.rst +++ b/docs/src/cdist-install.rst @@ -23,8 +23,6 @@ Target Hosts Install cdist ------------- -You can install cdist either from git or as a python package. - From git ~~~~~~~~ @@ -42,6 +40,7 @@ To install cdist, execute the following commands: From version 4.2.0 cdist tags and releases are signed. You can get GPG public key used for signing `here <_static/pgp-key-EFD2AE4EC36B6901.asc>`_. +It is assumed that you are familiar with *git* ways of signing and verification. You can also get cdist from `github mirror `_. @@ -157,3 +156,27 @@ Cdist is available as a python package at .. code-block:: sh pip install cdist + +Installing from source with signature verification +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you want to install cdist from signed source and verify it, first you need to +download cdist archive and its detached signature. + +Get both, *cdist-x.y.z.tar.gz* and *cdist-x.y.z.tar.gz.asc* from release +notes of the desired tag *x.y.z* at +`cdist git repository `_. + +Get GPG public key used for signing `here <_static/pgp-key-EFD2AE4EC36B6901.asc>`_ +and import it into GPG. + +Now cdist source archive can be verified using `gpg`, e.g. to verify `cdist-6.2.0`: + +.. code-block:: sh + + $ gpg --verify cdist-6.2.0.tar.gz.asc cdist-6.2.0.targ.gz + gpg: Signature made Sat Nov 30 23:14:19 2019 CET + gpg: using RSA key 69767822F3ECC3C349C1EFFFEFD2AE4EC36B6901 + gpg: Good signature from "ungleich GmbH (ungleich FOSS) " [ultimate] + +Further steps are the same as for `installing from git `_. From 553cfc28e7193ade2678cd805a7f236e51e92972 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Mon, 9 Dec 2019 09:57:54 +0100 Subject: [PATCH 02/38] ++changelog --- docs/changelog | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/changelog b/docs/changelog index 33db10da..04ffac62 100644 --- a/docs/changelog +++ b/docs/changelog @@ -8,6 +8,7 @@ next: * Documentation: Remove cdist-type prefix and man page reference from type list in html (Darko Poljak) * Documentation: PreOS english nitpicking (Evil Ham) * Documentation: Add installing from source with signature verification (Darko Poljak) + * Core: preos: Support top command logging options, custom conf-dir option and CDIST_PATH env var (Darko Poljak) 6.2.0: 2019-11-30 * Core: Redefine/reimplement/fix CDIST_ORDER_DEPENDENCY (Darko Poljak) From bee95efa4317eacbde98237f7952d020cdfb5e61 Mon Sep 17 00:00:00 2001 From: Evilham Date: Mon, 9 Dec 2019 19:13:10 +0100 Subject: [PATCH 03/38] [new-type] __openldap_server Originally developed at ungleich and improved for Open Sourcing by Evilham. --- cdist/conf/type/__openldap_server/man.rst | 148 +++++++++++ cdist/conf/type/__openldap_server/manifest | 235 ++++++++++++++++++ .../type/__openldap_server/parameter/boolean | 2 + .../type/__openldap_server/parameter/optional | 4 + .../parameter/optional_multiple | 2 + .../type/__openldap_server/parameter/required | 4 + 6 files changed, 395 insertions(+) create mode 100644 cdist/conf/type/__openldap_server/man.rst create mode 100644 cdist/conf/type/__openldap_server/manifest create mode 100644 cdist/conf/type/__openldap_server/parameter/boolean create mode 100644 cdist/conf/type/__openldap_server/parameter/optional create mode 100644 cdist/conf/type/__openldap_server/parameter/optional_multiple create mode 100644 cdist/conf/type/__openldap_server/parameter/required diff --git a/cdist/conf/type/__openldap_server/man.rst b/cdist/conf/type/__openldap_server/man.rst new file mode 100644 index 00000000..29bbc231 --- /dev/null +++ b/cdist/conf/type/__openldap_server/man.rst @@ -0,0 +1,148 @@ +cdist-type__openldap_server(7) +============================== + +NAME +---- +cdist-type__openldap_server - Setup an openldap(4) server instance + + +DESCRIPTION +----------- +This type can be used to bootstrap an LDAP environment using openldap as slapd. + + +REQUIRED PARAMETERS +------------------- +manager-dn + The rootdn to set up in the directory. + E.g. `cn=manager,dc=ungleich,dc=ch`. See `slapd.conf(5)`. + +manager-password-hash + The password for `manager-dn` in the directory. + This should be valid for `slapd.conf` like `{SSHA}qV+mCs3u8Q2sCmUXT4Ybw7MebHTASMyr`. + Generate e.g. with: `slappasswd -s weneedgoodsecurity`. + See `slappasswd(8C)`, `slapd.conf(5)`. + TODO: implement this: http://blog.adamsbros.org/2015/06/09/openldap-ssha-salted-hashes-by-hand/ + to allow for a manager-password parameter and ensure idempotency (care with salts). + Such manager-password parameter should be mutually exclusive with this one. + +serverid + The server for the directory. + E.g. `dc=ungleich,dc=ch`. See `slapd.conf(5)`. + +suffix + The suffix for the directory. + E.g. `dc=ungleich,dc=ch`. See `slapd.conf(5)`. + + +OPTIONAL PARAMETERS +------------------- +syncrepl-credentials + Only has an effect if `replicate` is set; required in that case. + This secret is shared amongst the hosts that will replicate the directory. + Note that each replication server needs this secret and it is saved in + plain text in the directory. + +syncrepl-searchbase + Only has an effect if `replicate` is set; required in that case. + The searchbase to use for replication. + E.g. `dc=ungleich,dc=ch`. See `slapd.conf(5)`. + +tls-cert + If defined, `__letsencrypt_cert` is not used and this must be the path in + the remote hosts to the PEM-encoded TLS certificate. + Requires: `tls-privkey` and `tls-ca`. + Permissions, existence and renewal of these files are left up to the + type's user. + +tls-privkey + Required if `tls-cert` is defined. + Path in the remote hosts to the PEM-encoded private key file. + +tls-ca + Required if `tls-cert` is defined. + Path in the remote hosts to the PEM-encoded CA certificate file. + + +OPTIONAL MULTIPLE PARAMETERS +---------------------------- +syncrepl-host + Only has an effect if `replicate` is set; required in that case. + Set once per host that will replicate the directory. + +module + LDAP module to load. See `slapd.conf(5)`. + Default value is OS-dependent, see manifest. + + +BOOLEAN PARAMETERS +------------------ +staging + Passed to `cdist-type__letsencrypt_cert`; has otherwise no use. + Obtain a test certificate from a staging server. + +replicate + Whether to setup replication or not. + If present `syncrepl-credentials` and `syncrepl-host` are also required. + +EXAMPLES +-------- + +.. code-block:: sh + + # Modify the ruleset on $__target_host: + __pf_ruleset --state present --source /my/pf/ruleset.conf + require="__pf_ruleset" \ + __pf_apply + + # Remove the ruleset on $__target_host (implies disabling pf(4): + __pf_ruleset --state absent + require="__pf_ruleset" \ + __pf_apply + + root@ldap-for-guacamole:~# cat ldapbase.ldif + dn: dc=guaca-test,dc=ungleich,dc=ch + objectClass: top + objectClass: dcObject + objectClass: organization + o: Some description + dc: guaca-test + + + # Sample usage: + # + # id=1 + # for host in ldap-test1.ungleich.ch ldap-test2.ungleich.ch; do + # echo "__ungleich_ldap ${host} \ + # --manager-dn 'cn=manager,dc=ungleich,dc=ch' \ + # --manager-password '{SSHA}fooo' \ + # --serverid '${id}' \ + # --staging \ + # --suffix 'dc=ungleich,dc=ch' \ + # --searchbase 'dc=ungleich,dc=ch' \ + # --syncrepl-credentials 'fooo' \ + # --syncrepl-host 'ldap-test1.ungleich.ch' \ + # --syncrepl-host 'ldap-test2.ungleich.ch' \ + # --descriptiont 'Ungleich LDAP server'" \ + # | cdist config -i - -v ${host} + # id=$((id + 1)) + # done + + +SEE ALSO +-------- +:strong:`cdist-type__letsencrypt_cert`\ (7) + + +AUTHORS +------- +ungleich +Evilham + + +COPYING +------- +Copyright \(C) 2020 ungleich glarus ag. You can redistribute it +and/or modify it under the terms of the GNU General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. diff --git a/cdist/conf/type/__openldap_server/manifest b/cdist/conf/type/__openldap_server/manifest new file mode 100644 index 00000000..2acaaed5 --- /dev/null +++ b/cdist/conf/type/__openldap_server/manifest @@ -0,0 +1,235 @@ +#!/bin/sh + +name="${__target_host}" +manager_dn=$(cat "${__object}/parameter/manager-dn") +manager_password_hash=$(cat "${__object}/parameter/manager-password-hash") +serverid=$(cat "${__object}/parameter/serverid") +suffix=$(cat "${__object}/parameter/suffix") +slapd_modules=$(cat "${__object}/parameter/module" || true) + + +OS="$(cat "${__global}/explorer/os")" + +# Setup OS-dependent vars +# TODO: treat other OS better, defaulting to Debian-like +case "${OS}" in + freebsd) + PKGS="openldap-server" + ETC="/usr/local/etc" + SLAPD_DIR="/usr/local/etc/openldap" + SLAPD_DATA_DIR="/var/db/openldap-data" + SLAPD_RUN_DIR="/var/run/openldap" + SLAPD_MODULE_PATH="/usr/local/libexec/openldap" + if [ -z "${slapd_modules}" ]; then + # It looks like ppolicy and syncprov must be compiled + slapd_modules="back_mdb back_monitor" + fi + ;; + *) + PKGS="slapd ldap-utils" + ETC="/etc" + SLAPD_DIR="/etc/ldap" + SLAPD_DATA_DIR="/var/lib/ldap" + SLAPD_RUN_DIR="/var/run/slapd" + SLAPD_MODULE_PATH="/usr/lib/ldap" + if [ -z "${slapd_modules}" ]; then + slapd_modules="back_mdb ppolicy syncprov back_monitor" + fi + ;; +esac + + + +# Determine if __letsencrypt_cert is to be used and setup vars accordingly +if [ -f "${__object}/parameter/tls-cert" ]; then + tls_cert=$(cat "${__object}/parameter/tls-cert") + + if [ ! -f "${__object}/parameter/tls-privkey" ]; then + echo "When tls-cert is defined, tls-privkey is also required." >&2 + exit 1 + fi + tls_privkey=$(cat "${__object}/parameter/tls-privkey") + + if [ ! -f "${__object}/parameter/tls-ca" ]; then + echo "When tls-cert is defined, tls-ca is also required." >&2 + exit 1 + fi + tls_ca=$(cat "${__object}/parameter/tls-ca") + + _skip_letsencrypt_cert="YES" +else + tls_cert="${SLAPD_DIR}/sasl2/cert.pem" + tls_privkey="${SLAPD_DIR}/sasl2/privkey.pem" + tls_ca="${SLAPD_DIR}/sasl2/chain.pem" +fi + +mkdir "${__object}/files" +ldapconf="${__object}/files/ldapconf" + +replication="" +if [ -f "${__object}/parameter/replicate" ]; then + replication=yes + + if [ ! -f "${__object}/parameter/syncrepl-searchbase" ]; then + echo "Requiring the searchbase for replication" >&2 + exit 1 + fi + syncrepl_searchbase=$(cat "${__object}/parameter/syncrepl-searchbase") + + if [ ! -f "${__object}/parameter/syncrepl-credentials" ]; then + echo "Requiring credentials for replication" >&2 + exit 1 + fi + + syncrepl_credentials=$(cat "${__object}/parameter/syncrepl-credentials") + + if [ ! -f "${__object}/parameter/syncrepl-host" ]; then + echo "Requiring host(s) for replication" >&2 + exit 1 + fi + syncrepl_hosts=$(cat "${__object}/parameter/syncrepl-host") + +fi + +# Install required packages +for pkg in ${PKGS}; do + __package ${pkg} +done + + +# TODO: Implement __start_on_boot for BSD +require="__package/slapd" __start_on_boot slapd + +# TODO: treat other OS better. Defaulting to Debian-like. +if [ "${OS}" != "freebsd" ]; then + require="__package/slapd" __line rm_slapd_conf \ + --file ${ETC}/default/slapd \ + --regex 'SLAPD_CONF=.*' \ + --state absent + + require="__package/slapd" __line rm_slapd_services \ + --file ${ETC}/default/slapd \ + --regex 'SLAPD_SERVICES=.*' \ + --state absent + + require="__line/rm_slapd_conf" __line add_slapd_conf \ + --file ${ETC}/default/slapd \ + --line 'SLAPD_CONF=${SLAPD_DIR}/slapd.conf' \ + --state present + + require="__line/rm_slapd_services" __line add_slapd_services \ + --file ${ETC}/default/slapd \ + --line "SLAPD_SERVICES=\"ldap://localhost/ ldap://${name}/\"" \ + --state present +fi + + +if [ -z "${_skip_letsencrypt_cert}" ]; then + if [ -f "${__object}/parameter/staging" ]; then + staging="--staging" + else + staging="" + fi + + __letsencrypt_cert "${name}" --admin-email technik@ungleich.ch \ + --renew-hook "cp ${ETC}/letsencrypt/live/${name}/*.pem ${SLAPD_DIR}/sasl2 && chown -R openldap:openldap ${SLAPD_DIR}/sasl2 && service slapd restart" \ + --automatic-renewal ${staging} +fi + +require="__package/slapd" __directory ${SLAPD_DIR}/slapd.d --state absent + +if [ -z "${_skip_letsencrypt_cert}" ]; then + require="__package/slapd __letsencrypt_cert/${name}" \ + __file ${SLAPD_DIR}/slapd.conf --owner root --group root --mode 644 \ + --source "${ldapconf}" +else + require="__package/slapd" \ + __file ${SLAPD_DIR}/slapd.conf --owner root --group root --mode 644 \ + --source "${ldapconf}" +fi + +# Start slapd.conf +cat << EOF > "${ldapconf}" +pidfile ${SLAPD_RUN_DIR}/slapd.pid +argsfile ${SLAPD_RUN_DIR}/slapd.args + +TLSCipherSuite NORMAL +TLSCertificateFile ${tls_cert} +TLSCertificateKeyFile ${tls_privkey} +TLSCACertificateFile ${tls_ca} + +disallow bind_anon +require bind +security tls=1 + +include ${SLAPD_DIR}/schema/corba.schema +include ${SLAPD_DIR}/schema/core.schema +include ${SLAPD_DIR}/schema/cosine.schema +include ${SLAPD_DIR}/schema/duaconf.schema +include ${SLAPD_DIR}/schema/dyngroup.schema +include ${SLAPD_DIR}/schema/inetorgperson.schema +include ${SLAPD_DIR}/schema/java.schema +include ${SLAPD_DIR}/schema/misc.schema +include ${SLAPD_DIR}/schema/nis.schema +include ${SLAPD_DIR}/schema/openldap.schema +include ${SLAPD_DIR}/schema/ppolicy.schema +include ${SLAPD_DIR}/schema/collective.schema + +modulepath ${SLAPD_MODULE_PATH} +EOF + +# Add specified modules +for module in ${slapd_modules}; do + echo "moduleload ${module}.la" >> "${ldapconf}" +done + +# Rest of the config +cat << EOF >> "${ldapconf}" +loglevel 1024 + +database mdb +maxsize 1073741824 + +suffix "${suffix}" +directory ${SLAPD_DATA_DIR} +rootdn "${manager_dn}" +rootpw "${manager_password_hash}" + +index objectClass eq,pres +index ou,cn,mail,surname,givenname eq,pres,sub +index uidNumber,gidNumber,loginShell eq,pres +index uid,memberUid eq,pres,sub +index nisMapName,nisMapEntry eq,pres,sub +index entryCSN,entryUUID eq + +serverid ${serverid} +EOF + +# Setup replication +if [ "${replication}" ]; then + rid=1; + for syncrepl in ${syncrepl_hosts}; do + cat <> "${ldapconf}" +syncrepl rid=${rid} + provider=ldap://${syncrepl} + bindmethod=simple + starttls=yes + binddn="${manager_dn}" + credentials=${syncrepl_credentials} + searchbase="${syncrepl_searchbase}" + type=refreshAndPersist + retry="5 + 5 +" + interval=00:00:00:05 +EOF + rid=$((rid + 1)) + done + cat <> "${ldapconf}" +mirrormode true +overlay syncprov +syncprov-checkpoint 100 5 +syncprov-sessionlog 100 + +database monitor +limits dn.exact="${manager_dn}" time=unlimited size=unlimited +EOF +fi diff --git a/cdist/conf/type/__openldap_server/parameter/boolean b/cdist/conf/type/__openldap_server/parameter/boolean new file mode 100644 index 00000000..45056fe9 --- /dev/null +++ b/cdist/conf/type/__openldap_server/parameter/boolean @@ -0,0 +1,2 @@ +staging +replicate diff --git a/cdist/conf/type/__openldap_server/parameter/optional b/cdist/conf/type/__openldap_server/parameter/optional new file mode 100644 index 00000000..a9a8ab2c --- /dev/null +++ b/cdist/conf/type/__openldap_server/parameter/optional @@ -0,0 +1,4 @@ +description +syncrepl-credentials +syncrepl-searchbase +tls-cert diff --git a/cdist/conf/type/__openldap_server/parameter/optional_multiple b/cdist/conf/type/__openldap_server/parameter/optional_multiple new file mode 100644 index 00000000..107c03d9 --- /dev/null +++ b/cdist/conf/type/__openldap_server/parameter/optional_multiple @@ -0,0 +1,2 @@ +syncrepl-host +module diff --git a/cdist/conf/type/__openldap_server/parameter/required b/cdist/conf/type/__openldap_server/parameter/required new file mode 100644 index 00000000..1ee6f219 --- /dev/null +++ b/cdist/conf/type/__openldap_server/parameter/required @@ -0,0 +1,4 @@ +manager-dn +manager-password-hash +serverid +suffix From fd430eab622206b9a04780ca5fa9d7c807f16c93 Mon Sep 17 00:00:00 2001 From: Evilham Date: Mon, 9 Dec 2019 19:39:43 +0100 Subject: [PATCH 04/38] [new-type] __openldap_server: Add a "schema" optional parameter. --- cdist/conf/type/__openldap_server/man.rst | 7 +++++ cdist/conf/type/__openldap_server/manifest | 29 +++++++------------ .../parameter/default/schema | 12 ++++++++ .../type/__openldap_server/parameter/optional | 3 +- 4 files changed, 31 insertions(+), 20 deletions(-) create mode 100644 cdist/conf/type/__openldap_server/parameter/default/schema diff --git a/cdist/conf/type/__openldap_server/man.rst b/cdist/conf/type/__openldap_server/man.rst index 29bbc231..1fc24eaa 100644 --- a/cdist/conf/type/__openldap_server/man.rst +++ b/cdist/conf/type/__openldap_server/man.rst @@ -74,6 +74,13 @@ module LDAP module to load. See `slapd.conf(5)`. Default value is OS-dependent, see manifest. +schema + Name of LDAP schema to load. Must be the name without extension of a + `.schema` file in slapd's schema directory (usually `/etc/slapd/schema` or + `/usr/local/etc/openldap/schema`). + Example value: `inetorgperson` + The type user must ensure that the schema file is deployed. + This defaults to a sensible subset, for details see the type definition. BOOLEAN PARAMETERS ------------------ diff --git a/cdist/conf/type/__openldap_server/manifest b/cdist/conf/type/__openldap_server/manifest index 2acaaed5..518edd86 100644 --- a/cdist/conf/type/__openldap_server/manifest +++ b/cdist/conf/type/__openldap_server/manifest @@ -6,6 +6,7 @@ manager_password_hash=$(cat "${__object}/parameter/manager-password-hash") serverid=$(cat "${__object}/parameter/serverid") suffix=$(cat "${__object}/parameter/suffix") slapd_modules=$(cat "${__object}/parameter/module" || true) +schemas=$(cat "${__object}/parameter/schema") OS="$(cat "${__global}/explorer/os")" @@ -14,8 +15,8 @@ OS="$(cat "${__global}/explorer/os")" # TODO: treat other OS better, defaulting to Debian-like case "${OS}" in freebsd) - PKGS="openldap-server" - ETC="/usr/local/etc" + PKGS="openldap-server" + ETC="/usr/local/etc" SLAPD_DIR="/usr/local/etc/openldap" SLAPD_DATA_DIR="/var/db/openldap-data" SLAPD_RUN_DIR="/var/run/openldap" @@ -27,7 +28,7 @@ case "${OS}" in ;; *) PKGS="slapd ldap-utils" - ETC="/etc" + ETC="/etc" SLAPD_DIR="/etc/ldap" SLAPD_DATA_DIR="/var/lib/ldap" SLAPD_RUN_DIR="/var/run/slapd" @@ -39,7 +40,6 @@ case "${OS}" in esac - # Determine if __letsencrypt_cert is to be used and setup vars accordingly if [ -f "${__object}/parameter/tls-cert" ]; then tls_cert=$(cat "${__object}/parameter/tls-cert") @@ -161,24 +161,15 @@ TLSCACertificateFile ${tls_ca} disallow bind_anon require bind security tls=1 - -include ${SLAPD_DIR}/schema/corba.schema -include ${SLAPD_DIR}/schema/core.schema -include ${SLAPD_DIR}/schema/cosine.schema -include ${SLAPD_DIR}/schema/duaconf.schema -include ${SLAPD_DIR}/schema/dyngroup.schema -include ${SLAPD_DIR}/schema/inetorgperson.schema -include ${SLAPD_DIR}/schema/java.schema -include ${SLAPD_DIR}/schema/misc.schema -include ${SLAPD_DIR}/schema/nis.schema -include ${SLAPD_DIR}/schema/openldap.schema -include ${SLAPD_DIR}/schema/ppolicy.schema -include ${SLAPD_DIR}/schema/collective.schema - -modulepath ${SLAPD_MODULE_PATH} EOF +# Add specified schemas +for schema in ${schemas}; do + echo "include ${SLAPD_DIR}/schema/${schema}.schema" >> "${ldapconf}" +done + # Add specified modules +echo "modulepath ${SLAPD_MODULE_PATH}" >> "${ldapconf}" for module in ${slapd_modules}; do echo "moduleload ${module}.la" >> "${ldapconf}" done diff --git a/cdist/conf/type/__openldap_server/parameter/default/schema b/cdist/conf/type/__openldap_server/parameter/default/schema new file mode 100644 index 00000000..825bdb15 --- /dev/null +++ b/cdist/conf/type/__openldap_server/parameter/default/schema @@ -0,0 +1,12 @@ +corba +core +cosine +duaconf +dyngroup +inetorgperson +java +misc +nis +openldap +ppolicy +collective diff --git a/cdist/conf/type/__openldap_server/parameter/optional b/cdist/conf/type/__openldap_server/parameter/optional index a9a8ab2c..53587c4e 100644 --- a/cdist/conf/type/__openldap_server/parameter/optional +++ b/cdist/conf/type/__openldap_server/parameter/optional @@ -1,4 +1,5 @@ -description syncrepl-credentials syncrepl-searchbase tls-cert +tls-privkey +tls-ca From 1ad605a509cfd534334111c1d6c1ea8682446b10 Mon Sep 17 00:00:00 2001 From: Evilham Date: Mon, 9 Dec 2019 19:49:05 +0100 Subject: [PATCH 05/38] [new-type] __openldap_server: Add admin-email parameter. --- cdist/conf/type/__openldap_server/man.rst | 5 +++++ cdist/conf/type/__openldap_server/manifest | 8 +++++++- cdist/conf/type/__openldap_server/parameter/optional | 1 + 3 files changed, 13 insertions(+), 1 deletion(-) diff --git a/cdist/conf/type/__openldap_server/man.rst b/cdist/conf/type/__openldap_server/man.rst index 1fc24eaa..4447971a 100644 --- a/cdist/conf/type/__openldap_server/man.rst +++ b/cdist/conf/type/__openldap_server/man.rst @@ -48,6 +48,11 @@ syncrepl-searchbase The searchbase to use for replication. E.g. `dc=ungleich,dc=ch`. See `slapd.conf(5)`. +admin-email + Passed to `cdist-type__letsencrypt_cert`; has otherwise no use. + Required if using `__letsencrypt_cert`. + Where to send Let's Encrypt emails like "certificate needs renewal". + tls-cert If defined, `__letsencrypt_cert` is not used and this must be the path in the remote hosts to the PEM-encoded TLS certificate. diff --git a/cdist/conf/type/__openldap_server/manifest b/cdist/conf/type/__openldap_server/manifest index 518edd86..e10432d6 100644 --- a/cdist/conf/type/__openldap_server/manifest +++ b/cdist/conf/type/__openldap_server/manifest @@ -58,6 +58,12 @@ if [ -f "${__object}/parameter/tls-cert" ]; then _skip_letsencrypt_cert="YES" else + if [ ! -f "${__object}/parameter/admin-email" ]; then + echo "When using __letsencrypt_cert, admin-email is also required." >&2 + exit 1 + fi + admin_email=$(cat "${__object}/parameter/admin-email") + tls_cert="${SLAPD_DIR}/sasl2/cert.pem" tls_privkey="${SLAPD_DIR}/sasl2/privkey.pem" tls_ca="${SLAPD_DIR}/sasl2/chain.pem" @@ -131,7 +137,7 @@ if [ -z "${_skip_letsencrypt_cert}" ]; then staging="" fi - __letsencrypt_cert "${name}" --admin-email technik@ungleich.ch \ + __letsencrypt_cert "${name}" --admin-email "${admin_email}" \ --renew-hook "cp ${ETC}/letsencrypt/live/${name}/*.pem ${SLAPD_DIR}/sasl2 && chown -R openldap:openldap ${SLAPD_DIR}/sasl2 && service slapd restart" \ --automatic-renewal ${staging} fi diff --git a/cdist/conf/type/__openldap_server/parameter/optional b/cdist/conf/type/__openldap_server/parameter/optional index 53587c4e..f4254cb6 100644 --- a/cdist/conf/type/__openldap_server/parameter/optional +++ b/cdist/conf/type/__openldap_server/parameter/optional @@ -1,5 +1,6 @@ syncrepl-credentials syncrepl-searchbase +admin-email tls-cert tls-privkey tls-ca From 42914d26c5addae027cc382ec035a687cd3aaf6d Mon Sep 17 00:00:00 2001 From: Evilham Date: Mon, 9 Dec 2019 19:59:15 +0100 Subject: [PATCH 06/38] [new-type] __openldap_server: sudo make nico.happy. --- cdist/conf/type/__openldap_server/manifest | 53 ++++++++++++---------- 1 file changed, 30 insertions(+), 23 deletions(-) diff --git a/cdist/conf/type/__openldap_server/manifest b/cdist/conf/type/__openldap_server/manifest index e10432d6..ffd92626 100644 --- a/cdist/conf/type/__openldap_server/manifest +++ b/cdist/conf/type/__openldap_server/manifest @@ -9,11 +9,10 @@ slapd_modules=$(cat "${__object}/parameter/module" || true) schemas=$(cat "${__object}/parameter/schema") -OS="$(cat "${__global}/explorer/os")" +os="$(cat "${__global}/explorer/os")" # Setup OS-dependent vars -# TODO: treat other OS better, defaulting to Debian-like -case "${OS}" in +case "${os}" in freebsd) PKGS="openldap-server" ETC="/usr/local/etc" @@ -26,7 +25,7 @@ case "${OS}" in slapd_modules="back_mdb back_monitor" fi ;; - *) + debian|ubuntu|devuan) PKGS="slapd ldap-utils" ETC="/etc" SLAPD_DIR="/etc/ldap" @@ -37,6 +36,10 @@ case "${OS}" in slapd_modules="back_mdb ppolicy syncprov back_monitor" fi ;; + *) + echo "Don't know the openldap defaults for: $os" >&2 + exit 1 + ;; esac @@ -106,28 +109,32 @@ done # TODO: Implement __start_on_boot for BSD require="__package/slapd" __start_on_boot slapd -# TODO: treat other OS better. Defaulting to Debian-like. -if [ "${OS}" != "freebsd" ]; then - require="__package/slapd" __line rm_slapd_conf \ - --file ${ETC}/default/slapd \ - --regex 'SLAPD_CONF=.*' \ - --state absent +case "${os}" in + debian|ubuntu|devuan) + require="__package/slapd" __line rm_slapd_conf \ + --file ${ETC}/default/slapd \ + --regex 'SLAPD_CONF=.*' \ + --state absent - require="__package/slapd" __line rm_slapd_services \ - --file ${ETC}/default/slapd \ - --regex 'SLAPD_SERVICES=.*' \ - --state absent + require="__package/slapd" __line rm_slapd_services \ + --file ${ETC}/default/slapd \ + --regex 'SLAPD_SERVICES=.*' \ + --state absent - require="__line/rm_slapd_conf" __line add_slapd_conf \ - --file ${ETC}/default/slapd \ - --line 'SLAPD_CONF=${SLAPD_DIR}/slapd.conf' \ - --state present + require="__line/rm_slapd_conf" __line add_slapd_conf \ + --file ${ETC}/default/slapd \ + --line 'SLAPD_CONF=${SLAPD_DIR}/slapd.conf' \ + --state present - require="__line/rm_slapd_services" __line add_slapd_services \ - --file ${ETC}/default/slapd \ - --line "SLAPD_SERVICES=\"ldap://localhost/ ldap://${name}/\"" \ - --state present -fi + require="__line/rm_slapd_services" __line add_slapd_services \ + --file ${ETC}/default/slapd \ + --line "SLAPD_SERVICES=\"ldap://localhost/ ldap://${name}/\"" \ + --state present + ;; + *) + # Nothing to do here, move on. + ;; +esac if [ -z "${_skip_letsencrypt_cert}" ]; then From 22c5cd550bb1069c2bb8106bbe907f016b9b821a Mon Sep 17 00:00:00 2001 From: Evilham Date: Tue, 10 Dec 2019 12:49:07 +0100 Subject: [PATCH 07/38] [new-type] __openldap_server: first public version. This already takes care of setting up the base DN and managing it as well as allowing for settings for the listener URLS. The type was also made a singleton as it doesn't make much sense to setup multiple LDAP servers on the same machine. --- .../type/__openldap_server/gencode-remote | 44 ++++++ cdist/conf/type/__openldap_server/man.rst | 126 +++++++++++++----- cdist/conf/type/__openldap_server/manifest | 48 +++++-- .../parameter/default/description | 1 + .../type/__openldap_server/parameter/optional | 4 +- .../parameter/optional_multiple | 1 + .../type/__openldap_server/parameter/required | 1 + .../parameter/required_multiple | 1 + cdist/conf/type/__openldap_server/singleton | 0 9 files changed, 176 insertions(+), 50 deletions(-) create mode 100644 cdist/conf/type/__openldap_server/gencode-remote create mode 100644 cdist/conf/type/__openldap_server/parameter/default/description create mode 100644 cdist/conf/type/__openldap_server/parameter/required_multiple create mode 100644 cdist/conf/type/__openldap_server/singleton diff --git a/cdist/conf/type/__openldap_server/gencode-remote b/cdist/conf/type/__openldap_server/gencode-remote new file mode 100644 index 00000000..0ac434af --- /dev/null +++ b/cdist/conf/type/__openldap_server/gencode-remote @@ -0,0 +1,44 @@ +#!/bin/sh + +manager_dn=$(cat "${__object}/parameter/manager-dn") +manager_password=$(cat "${__object}/parameter/manager-password") +description=$(cat "${__object}/parameter/description") +suffix=$(cat "${__object}/parameter/suffix") +suffix_dc=$(echo -n ${suffix} | awk -F',' '{print $1}' | awk -F'=' '{print $2}') + +SLAPD_IPC=$(cat "${__object}/parameter/slapd-url" | tr '\n' ' ' | awk '{ print $1}') + +cat <&1 > /dev/null; then + # Already exists, use ldapmodify + ldapmodify -xZ -D "${manager_dn}" -w "${manager_password}" -H '${SLAPD_IPC}' </dev/null || true) schemas=$(cat "${__object}/parameter/schema") +slapd_urls=$(cat "${__object}/parameter/slapd-url" | tr '\n' ' ') +tls_cipher_suite=$(cat "${__object}/parameter/tls-cipher-suite" 2>/dev/null || true) os="$(cat "${__global}/explorer/os")" # Setup OS-dependent vars +CONF_OWNER="root" +CONF_GROUP="root" case "${os}" in freebsd) PKGS="openldap-server" @@ -24,6 +28,12 @@ case "${os}" in # It looks like ppolicy and syncprov must be compiled slapd_modules="back_mdb back_monitor" fi + CONF_OWNER="ldap" + CONF_GROUP="ldap" + if [ -z "${tls_cipher_suite}" ]; then + # TODO: research default for FreeBSD. 'NORMAL' appears to not work + tls_cipher_suite="HIGH:MEDIUM:+SSLv2" + fi ;; debian|ubuntu|devuan) PKGS="slapd ldap-utils" @@ -35,6 +45,9 @@ case "${os}" in if [ -z "${slapd_modules}" ]; then slapd_modules="back_mdb ppolicy syncprov back_monitor" fi + if [ -z "${tls_cipher_suite}" ]; then + tls_cipher_suite="NORMAL" + fi ;; *) echo "Don't know the openldap defaults for: $os" >&2 @@ -42,6 +55,8 @@ case "${os}" in ;; esac +PKG_MAIN=$(echo ${PKGS} | awk '{print $1;}') + # Determine if __letsencrypt_cert is to be used and setup vars accordingly if [ -f "${__object}/parameter/tls-cert" ]; then @@ -106,17 +121,26 @@ for pkg in ${PKGS}; do done -# TODO: Implement __start_on_boot for BSD -require="__package/slapd" __start_on_boot slapd +require="__package/${PKG_MAIN}" __start_on_boot slapd +# Setup -h flag for the listeners. See man slapd (-h flag). case "${os}" in + freebsd) + require="__package/${PKG_MAIN}" __key_value \ + --file "/etc/rc.conf" \ + --key "slapd_flags" \ + --value "\"-h '${slapd_urls}'\"" \ + --delimiter "=" \ + --comment "# LDAP Listener URLs" \ + "${__target_host}__slapd_flags" + ;; debian|ubuntu|devuan) - require="__package/slapd" __line rm_slapd_conf \ + require="__package/${PKG_MAIN}" __line rm_slapd_conf \ --file ${ETC}/default/slapd \ --regex 'SLAPD_CONF=.*' \ --state absent - require="__package/slapd" __line rm_slapd_services \ + require="__package/${PKG_MAIN}" __line rm_slapd_services \ --file ${ETC}/default/slapd \ --regex 'SLAPD_SERVICES=.*' \ --state absent @@ -128,7 +152,7 @@ case "${os}" in require="__line/rm_slapd_services" __line add_slapd_services \ --file ${ETC}/default/slapd \ - --line "SLAPD_SERVICES=\"ldap://localhost/ ldap://${name}/\"" \ + --line "SLAPD_SERVICES=\"${slapd_urls}\"" \ --state present ;; *) @@ -149,15 +173,15 @@ if [ -z "${_skip_letsencrypt_cert}" ]; then --automatic-renewal ${staging} fi -require="__package/slapd" __directory ${SLAPD_DIR}/slapd.d --state absent +require="__package/${PKG_MAIN}" __directory ${SLAPD_DIR}/slapd.d --state absent if [ -z "${_skip_letsencrypt_cert}" ]; then - require="__package/slapd __letsencrypt_cert/${name}" \ - __file ${SLAPD_DIR}/slapd.conf --owner root --group root --mode 644 \ + require="__package/${PKG_MAIN} __letsencrypt_cert/${name}" \ + __file ${SLAPD_DIR}/slapd.conf --owner ${CONF_OWNER} --group ${CONF_GROUP} --mode 644 \ --source "${ldapconf}" else - require="__package/slapd" \ - __file ${SLAPD_DIR}/slapd.conf --owner root --group root --mode 644 \ + require="__package/${PKG_MAIN}" \ + __file ${SLAPD_DIR}/slapd.conf --owner ${CONF_OWNER} --group ${CONF_GROUP} --mode 644 \ --source "${ldapconf}" fi @@ -166,7 +190,7 @@ cat << EOF > "${ldapconf}" pidfile ${SLAPD_RUN_DIR}/slapd.pid argsfile ${SLAPD_RUN_DIR}/slapd.args -TLSCipherSuite NORMAL +TLSCipherSuite ${tls_cipher_suite} TLSCertificateFile ${tls_cert} TLSCertificateKeyFile ${tls_privkey} TLSCACertificateFile ${tls_ca} diff --git a/cdist/conf/type/__openldap_server/parameter/default/description b/cdist/conf/type/__openldap_server/parameter/default/description new file mode 100644 index 00000000..6d8e37e1 --- /dev/null +++ b/cdist/conf/type/__openldap_server/parameter/default/description @@ -0,0 +1 @@ +Managed by cdist, do not edit manually. diff --git a/cdist/conf/type/__openldap_server/parameter/optional b/cdist/conf/type/__openldap_server/parameter/optional index f4254cb6..a92b9c6e 100644 --- a/cdist/conf/type/__openldap_server/parameter/optional +++ b/cdist/conf/type/__openldap_server/parameter/optional @@ -1,6 +1,8 @@ +description syncrepl-credentials syncrepl-searchbase admin-email +tls-cipher-suite tls-cert tls-privkey -tls-ca +tls-ca \ No newline at end of file diff --git a/cdist/conf/type/__openldap_server/parameter/optional_multiple b/cdist/conf/type/__openldap_server/parameter/optional_multiple index 107c03d9..52a83d5c 100644 --- a/cdist/conf/type/__openldap_server/parameter/optional_multiple +++ b/cdist/conf/type/__openldap_server/parameter/optional_multiple @@ -1,2 +1,3 @@ syncrepl-host module +schema diff --git a/cdist/conf/type/__openldap_server/parameter/required b/cdist/conf/type/__openldap_server/parameter/required index 1ee6f219..ff58158d 100644 --- a/cdist/conf/type/__openldap_server/parameter/required +++ b/cdist/conf/type/__openldap_server/parameter/required @@ -1,4 +1,5 @@ manager-dn +manager-password manager-password-hash serverid suffix diff --git a/cdist/conf/type/__openldap_server/parameter/required_multiple b/cdist/conf/type/__openldap_server/parameter/required_multiple new file mode 100644 index 00000000..848b8dc2 --- /dev/null +++ b/cdist/conf/type/__openldap_server/parameter/required_multiple @@ -0,0 +1 @@ +slapd-url \ No newline at end of file diff --git a/cdist/conf/type/__openldap_server/singleton b/cdist/conf/type/__openldap_server/singleton new file mode 100644 index 00000000..e69de29b From 79d58f0813fd08c77d826ac552dfdb9e0f40dbbd Mon Sep 17 00:00:00 2001 From: Evilham Date: Tue, 10 Dec 2019 13:04:24 +0100 Subject: [PATCH 08/38] [new-type] __openldap_server: fix docs / dependency. It's nicer to have slapd_flags after slapd_enable in rc.conf on freebsd. --- cdist/conf/type/__openldap_server/man.rst | 6 +++--- cdist/conf/type/__openldap_server/manifest | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/cdist/conf/type/__openldap_server/man.rst b/cdist/conf/type/__openldap_server/man.rst index 84b534bd..1c2c9b70 100644 --- a/cdist/conf/type/__openldap_server/man.rst +++ b/cdist/conf/type/__openldap_server/man.rst @@ -141,9 +141,9 @@ EXAMPLES --serverid 0 \ --suffix 'dc=camilion,dc=cloud' \ --slapd-url 'ldaps://ldap.camilion.cloud' \ - --tls-cert '${pki_prefix}/default.crt' \ - --tls-privkey '${pki_prefix}/default.key' \ - --tls-ca '${pki_prefix}/CA.crt' + --tls-cert "${pki_prefix}/default.crt" \ + --tls-privkey "${pki_prefix}/default.key" \ + --tls-ca "${pki_prefix}/CA.crt" # The created basedn looks as follows: # diff --git a/cdist/conf/type/__openldap_server/manifest b/cdist/conf/type/__openldap_server/manifest index 2d1df32f..070a31d5 100644 --- a/cdist/conf/type/__openldap_server/manifest +++ b/cdist/conf/type/__openldap_server/manifest @@ -126,7 +126,7 @@ require="__package/${PKG_MAIN}" __start_on_boot slapd # Setup -h flag for the listeners. See man slapd (-h flag). case "${os}" in freebsd) - require="__package/${PKG_MAIN}" __key_value \ + require="__start_on_boot/slapd" __key_value \ --file "/etc/rc.conf" \ --key "slapd_flags" \ --value "\"-h '${slapd_urls}'\"" \ From 35f0d4dbfa9307923fa013ff806d30087a8cf291 Mon Sep 17 00:00:00 2001 From: Evilham Date: Tue, 10 Dec 2019 13:10:19 +0100 Subject: [PATCH 09/38] [new-type] __openldap_server: another documentation typo. --- cdist/conf/type/__openldap_server/man.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cdist/conf/type/__openldap_server/man.rst b/cdist/conf/type/__openldap_server/man.rst index 1c2c9b70..d20101d1 100644 --- a/cdist/conf/type/__openldap_server/man.rst +++ b/cdist/conf/type/__openldap_server/man.rst @@ -133,8 +133,8 @@ EXAMPLES .. code-block:: sh # Example of a simple server with manual certificate management. - pki_prefix="/usr/local/etc/pki/realms/ldap.camilion.cloud" \ - __openldap_server \ + pki_prefix="/usr/local/etc/pki/realms/ldap.camilion.cloud" + __openldap_server \ --manager-dn 'cn=manager,dc=camilion,dc=cloud' \ --manager-password "foo" \ --manager-password-hash '{SSHA}foo' \ From 3ba230c10db98cf2cddb0aeeb256392223fb2558 Mon Sep 17 00:00:00 2001 From: Evilham Date: Tue, 10 Dec 2019 13:12:54 +0100 Subject: [PATCH 10/38] [type-docs] __start_on_boot: remove unspported *BSD claim. The type appears to support {Open,Free}BSD properly. --- cdist/conf/type/__start_on_boot/man.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cdist/conf/type/__start_on_boot/man.rst b/cdist/conf/type/__start_on_boot/man.rst index b7c73ab1..f8afe94b 100644 --- a/cdist/conf/type/__start_on_boot/man.rst +++ b/cdist/conf/type/__start_on_boot/man.rst @@ -12,7 +12,7 @@ This cdist type allows you to enable or disable stuff to be started at boot of your operating system. Warning: This type has not been tested intensively and is not fully -supported (i.e. \*BSD are not implemented). +supported. REQUIRED PARAMETERS From 3c62a88ca7907b15dc501ddb3395c66b6196cb13 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Tue, 10 Dec 2019 19:33:29 +0100 Subject: [PATCH 11/38] ++changelog --- docs/changelog | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/changelog b/docs/changelog index 04ffac62..186dce7a 100644 --- a/docs/changelog +++ b/docs/changelog @@ -9,6 +9,8 @@ next: * Documentation: PreOS english nitpicking (Evil Ham) * Documentation: Add installing from source with signature verification (Darko Poljak) * Core: preos: Support top command logging options, custom conf-dir option and CDIST_PATH env var (Darko Poljak) + * Type __start_on_boot: Docs: remove unsupported *BSD claim (Evil Ham) + * New type: __openldap_server (Evil Ham) 6.2.0: 2019-11-30 * Core: Redefine/reimplement/fix CDIST_ORDER_DEPENDENCY (Darko Poljak) From 2c7b4ddc55a9a2f76ae5b51a6020c0cd14900705 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Thu, 12 Dec 2019 06:18:04 +0100 Subject: [PATCH 12/38] Update cdist man page --- docs/src/man1/cdist.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/man1/cdist.rst b/docs/src/man1/cdist.rst index d6937272..55db82ed 100644 --- a/docs/src/man1/cdist.rst +++ b/docs/src/man1/cdist.rst @@ -915,5 +915,5 @@ such case and display a warning message. An example of such a case: COPYING ------- -Copyright \(C) 2011-2017 Nico Schottelius. Free use of this software is +Copyright \(C) 2011-2019 Nico Schottelius. Free use of this software is granted under the terms of the GNU General Public License v3 or later (GPLv3+). From bd4eee7925356c8e557a75688694feff94cca297 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Thu, 12 Dec 2019 07:00:23 +0100 Subject: [PATCH 13/38] Fix shellcheck reported issues --- cdist/conf/type/__openldap_server/gencode-remote | 4 ++-- cdist/conf/type/__openldap_server/manifest | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/cdist/conf/type/__openldap_server/gencode-remote b/cdist/conf/type/__openldap_server/gencode-remote index 0ac434af..b1e98f8c 100644 --- a/cdist/conf/type/__openldap_server/gencode-remote +++ b/cdist/conf/type/__openldap_server/gencode-remote @@ -4,9 +4,9 @@ manager_dn=$(cat "${__object}/parameter/manager-dn") manager_password=$(cat "${__object}/parameter/manager-password") description=$(cat "${__object}/parameter/description") suffix=$(cat "${__object}/parameter/suffix") -suffix_dc=$(echo -n ${suffix} | awk -F',' '{print $1}' | awk -F'=' '{print $2}') +suffix_dc=$(printf "%s" "${suffix}" | awk -F',' '{print $1}' | awk -F'=' '{print $2}') -SLAPD_IPC=$(cat "${__object}/parameter/slapd-url" | tr '\n' ' ' | awk '{ print $1}') +SLAPD_IPC=$(tr '\n' ' ' < "${__object}/parameter/slapd-url" | awk '{ print $1}') cat </dev/null || true) schemas=$(cat "${__object}/parameter/schema") -slapd_urls=$(cat "${__object}/parameter/slapd-url" | tr '\n' ' ') +slapd_urls=$(tr '\n' ' ' < "${__object}/parameter/slapd-url") tls_cipher_suite=$(cat "${__object}/parameter/tls-cipher-suite" 2>/dev/null || true) @@ -55,7 +55,7 @@ case "${os}" in ;; esac -PKG_MAIN=$(echo ${PKGS} | awk '{print $1;}') +PKG_MAIN=$(echo "${PKGS}" | awk '{print $1;}') # Determine if __letsencrypt_cert is to be used and setup vars accordingly @@ -117,7 +117,7 @@ fi # Install required packages for pkg in ${PKGS}; do - __package ${pkg} + __package "${pkg}" done @@ -147,7 +147,7 @@ case "${os}" in require="__line/rm_slapd_conf" __line add_slapd_conf \ --file ${ETC}/default/slapd \ - --line 'SLAPD_CONF=${SLAPD_DIR}/slapd.conf' \ + --line "SLAPD_CONF=${SLAPD_DIR}/slapd.conf" \ --state present require="__line/rm_slapd_services" __line add_slapd_services \ From 8562871da9a51752fea47308e496e139210edf75 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Thu, 12 Dec 2019 07:35:53 +0100 Subject: [PATCH 14/38] Fix shellcheck exit status shellcheck* targets were always reporting exit status 0. With this fix, if shellcheck fails, then build-helper script exits with 1. --- bin/build-helper | 50 ++++++++++++++++++++++++++++++++---------------- 1 file changed, 33 insertions(+), 17 deletions(-) diff --git a/bin/build-helper b/bin/build-helper index 7cfc4d55..69dee4c7 100755 --- a/bin/build-helper +++ b/bin/build-helper @@ -74,6 +74,7 @@ SHELLCHECKCMD="shellcheck -s sh -f gcc -x" # Skip SC2154 for variables starting with __ since such variables are cdist # environment variables. SHELLCHECK_SKIP=': __.*is referenced but not assigned.*\[SC2154\]' +SHELLCHECKTMP=".shellcheck.tmp" # Change to checkout directory basedir="${0%/*}/../" @@ -431,53 +432,67 @@ eof ;; shellcheck-global-explorers) - find cdist/conf/explorer -type f -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" || exit 0 + # shellcheck disable=SC2086 + find cdist/conf/explorer -type f -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" > "${SHELLCHECKTMP}" + test ! -s "${SHELLCHECKTMP}" || { cat "${SHELLCHECKTMP}"; exit 1; } ;; shellcheck-type-explorers) - find cdist/conf/type -type f -path "*/explorer/*" -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" || exit 0 + # shellcheck disable=SC2086 + find cdist/conf/type -type f -path "*/explorer/*" -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" > "${SHELLCHECKTMP}" + test ! -s "${SHELLCHECKTMP}" || { cat "${SHELLCHECKTMP}"; exit 1; } ;; shellcheck-manifests) - find cdist/conf/type -type f -name manifest -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" || exit 0 + # shellcheck disable=SC2086 + find cdist/conf/type -type f -name manifest -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" > "${SHELLCHECKTMP}" + test ! -s "${SHELLCHECKTMP}" || { cat "${SHELLCHECKTMP}"; exit 1; } ;; shellcheck-local-gencodes) - find cdist/conf/type -type f -name gencode-local -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" || exit 0 + # shellcheck disable=SC2086 + find cdist/conf/type -type f -name gencode-local -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" > "${SHELLCHECKTMP}" + test ! -s "${SHELLCHECKTMP}" || { cat "${SHELLCHECKTMP}"; exit 1; } ;; shellcheck-remote-gencodes) - find cdist/conf/type -type f -name gencode-remote -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" || exit 0 + # shellcheck disable=SC2086 + find cdist/conf/type -type f -name gencode-remote -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" > "${SHELLCHECKTMP}" + test ! -s "${SHELLCHECKTMP}" || { cat "${SHELLCHECKTMP}"; exit 1; } ;; shellcheck-scripts) - ${SHELLCHECKCMD} scripts/cdist-dump scripts/cdist-new-type || exit 0 + # shellcheck disable=SC2086 + ${SHELLCHECKCMD} scripts/cdist-dump scripts/cdist-new-type > "${SHELLCHECKTMP}" + test ! -s "${SHELLCHECKTMP}" || { cat "${SHELLCHECKTMP}"; exit 1; } ;; shellcheck-gencodes) - "$0" shellcheck-local-gencodes - "$0" shellcheck-remote-gencodes + "$0" shellcheck-local-gencodes || exit 1 + "$0" shellcheck-remote-gencodes || exit 1 ;; shellcheck-types) - "$0" shellcheck-type-explorers - "$0" shellcheck-manifests - "$0" shellcheck-gencodes + "$0" shellcheck-type-explorers || exit 1 + "$0" shellcheck-manifests || exit 1 + "$0" shellcheck-gencodes || exit 1 ;; shellcheck) - "$0" shellcheck-global-explorers - "$0" shellcheck-types - "$0" shellcheck-scripts + "$0" shellcheck-global-explorers || exit 1 + "$0" shellcheck-types || exit 1 + "$0" shellcheck-scripts || exit 1 ;; shellcheck-type-files) - find cdist/conf/type -type f -path "*/files/*" -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" || exit 0 + # shellcheck disable=SC2086 + find cdist/conf/type -type f -path "*/files/*" -exec ${SHELLCHECKCMD} {} + | grep -v "${SHELLCHECK_SKIP}" > "${SHELLCHECKTMP}" + test ! -s "${SHELLCHECKTMP}" || { cat "${SHELLCHECKTMP}"; exit 1; } ;; shellcheck-with-files) - "$0" shellcheck - "$0" shellcheck-type-files + "$0" shellcheck || exit 1 + "$0" shellcheck-type-files || exit 1 ;; shellcheck-build-helper) @@ -535,6 +550,7 @@ eof # Temp files rm -f ./*.tmp + rm -f ./.*.tmp ;; distclean) From 9859080217a9dc067250a0fc0f686a2242ef1083 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Thu, 12 Dec 2019 07:43:07 +0100 Subject: [PATCH 15/38] Release 6.3.0 --- docs/changelog | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelog b/docs/changelog index 186dce7a..674e14c7 100644 --- a/docs/changelog +++ b/docs/changelog @@ -1,7 +1,7 @@ Changelog --------- -next: +6.3.0: 2019-12-12 * Type __package_update_index: Fix Alpine part (Dominique Roux) * Documentation: Fix man pages for install types (Darko Poljak) * Documentation: Embed config skeleton instead of rewriting it (Darko Poljak) From f407e8825fc2eb70fa489dc218b46f1f2d92d747 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 15 Dec 2019 22:56:16 +0100 Subject: [PATCH 16/38] [consul/alpine] no init script required to be deployed anymore --- cdist/conf/type/__consul_agent/manifest | 2 +- docs/changelog | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/cdist/conf/type/__consul_agent/manifest b/cdist/conf/type/__consul_agent/manifest index a88d26ed..ee682d72 100755 --- a/cdist/conf/type/__consul_agent/manifest +++ b/cdist/conf/type/__consul_agent/manifest @@ -181,7 +181,7 @@ init_upstart() # Install init script to start on boot case "$os" in - alpine|devuan) + devuan) init_sysvinit debian ;; centos|redhat) diff --git a/docs/changelog b/docs/changelog index 674e14c7..415af0af 100644 --- a/docs/changelog +++ b/docs/changelog @@ -1,6 +1,10 @@ Changelog --------- +next: + * Type __consul_agent: don't deploy init script on Alpine anymore, + it ships with one itself (Nico Schottelius) + 6.3.0: 2019-12-12 * Type __package_update_index: Fix Alpine part (Dominique Roux) * Documentation: Fix man pages for install types (Darko Poljak) From df05abd15b6676d6594b3e8aebff112c2f5c0ce7 Mon Sep 17 00:00:00 2001 From: Steven Armstrong Date: Thu, 19 Dec 2019 12:33:47 +0100 Subject: [PATCH 17/38] bugfix: __install_chroot_umount was not using __chroot_umount/manifest Signed-off-by: Steven Armstrong --- cdist/conf/type/__install_chroot_umount/manifest | 1 + 1 file changed, 1 insertion(+) create mode 120000 cdist/conf/type/__install_chroot_umount/manifest diff --git a/cdist/conf/type/__install_chroot_umount/manifest b/cdist/conf/type/__install_chroot_umount/manifest new file mode 120000 index 00000000..f17af67a --- /dev/null +++ b/cdist/conf/type/__install_chroot_umount/manifest @@ -0,0 +1 @@ +../__chroot_umount/manifest \ No newline at end of file From beb930c0dc6e80d0718029fa8fc5539f373154a2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Timoth=C3=A9e=20Floure?= Date: Tue, 31 Dec 2019 11:05:40 +0100 Subject: [PATCH 18/38] __postgres_*: use delimited identifiers (double quoted) in generated SQL --- cdist/conf/type/__postgres_database/gencode-remote | 6 +++--- cdist/conf/type/__postgres_role/gencode-remote | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/cdist/conf/type/__postgres_database/gencode-remote b/cdist/conf/type/__postgres_database/gencode-remote index 61cfa50d..9f12d215 100755 --- a/cdist/conf/type/__postgres_database/gencode-remote +++ b/cdist/conf/type/__postgres_database/gencode-remote @@ -41,12 +41,12 @@ if [ "$state_should" != "$state_is" ]; then present) owner="" if [ -f "$__object/parameter/owner" ]; then - owner="-O '$(cat "$__object/parameter/owner")'" + owner="-O \"$(cat "$__object/parameter/owner")\"" fi - echo "su - '$postgres_user' -c \"createdb $owner '$name'\"" + echo "su - '$postgres_user' -c \"createdb $owner \"$name\"\"" ;; absent) - echo "su - '$postgres_user' -c \"dropdb '$name'\"" + echo "su - '$postgres_user' -c \"dropdb \"$name\"\"" ;; esac fi diff --git a/cdist/conf/type/__postgres_role/gencode-remote b/cdist/conf/type/__postgres_role/gencode-remote index fd56e85d..0b8852f4 100755 --- a/cdist/conf/type/__postgres_role/gencode-remote +++ b/cdist/conf/type/__postgres_role/gencode-remote @@ -54,7 +54,7 @@ case "$state_should" in [ -n "$password" ] && password="PASSWORD '$password'" - cmd="CREATE ROLE $name WITH $password $booleans" + cmd="CREATE ROLE \"$name\" WITH $password $booleans" echo "su - '$postgres_user' -c \"psql postgres -wc \\\"$cmd\\\"\"" ;; absent) From c32e4040b1b17f1e4c8173f23f1bdb2d5112f227 Mon Sep 17 00:00:00 2001 From: Ander Punnar Date: Tue, 31 Dec 2019 19:16:49 +0200 Subject: [PATCH 19/38] __mysql_*: initial rewrite --- .../conf/type/__mysql_database/explorer/state | 15 ++++ .../conf/type/__mysql_database/gencode-remote | 72 ++++++------------- cdist/conf/type/__mysql_database/man.rst | 49 ------------- cdist/conf/type/__mysql_database/manifest | 26 +++++++ .../__mysql_database/parameter/default/state | 1 + .../type/__mysql_database/parameter/optional | 1 + .../type/__mysql_privileges/explorer/state | 22 ++++++ .../type/__mysql_privileges/gencode-remote | 31 ++++++++ .../__mysql_privileges/parameter/default/host | 1 + .../parameter/default/privileges | 1 + .../parameter/default/state | 1 + .../parameter/default/table | 1 + .../__mysql_privileges/parameter/optional | 4 ++ .../__mysql_privileges/parameter/required | 2 + cdist/conf/type/__mysql_user/explorer/state | 36 ++++++++++ cdist/conf/type/__mysql_user/gencode-remote | 50 +++++++++++++ .../type/__mysql_user/parameter/default/host | 1 + .../type/__mysql_user/parameter/default/state | 1 + .../conf/type/__mysql_user/parameter/optional | 4 ++ 19 files changed, 221 insertions(+), 98 deletions(-) create mode 100755 cdist/conf/type/__mysql_database/explorer/state delete mode 100644 cdist/conf/type/__mysql_database/man.rst create mode 100755 cdist/conf/type/__mysql_database/manifest create mode 100644 cdist/conf/type/__mysql_database/parameter/default/state create mode 100755 cdist/conf/type/__mysql_privileges/explorer/state create mode 100755 cdist/conf/type/__mysql_privileges/gencode-remote create mode 100644 cdist/conf/type/__mysql_privileges/parameter/default/host create mode 100644 cdist/conf/type/__mysql_privileges/parameter/default/privileges create mode 100644 cdist/conf/type/__mysql_privileges/parameter/default/state create mode 100644 cdist/conf/type/__mysql_privileges/parameter/default/table create mode 100644 cdist/conf/type/__mysql_privileges/parameter/optional create mode 100644 cdist/conf/type/__mysql_privileges/parameter/required create mode 100755 cdist/conf/type/__mysql_user/explorer/state create mode 100755 cdist/conf/type/__mysql_user/gencode-remote create mode 100644 cdist/conf/type/__mysql_user/parameter/default/host create mode 100644 cdist/conf/type/__mysql_user/parameter/default/state create mode 100644 cdist/conf/type/__mysql_user/parameter/optional diff --git a/cdist/conf/type/__mysql_database/explorer/state b/cdist/conf/type/__mysql_database/explorer/state new file mode 100755 index 00000000..16cc9ce5 --- /dev/null +++ b/cdist/conf/type/__mysql_database/explorer/state @@ -0,0 +1,15 @@ +#!/bin/sh -e + +if [ -f "$__object/parameter/name" ] +then + name="$( cat "$__object/parameter/name" )" +else + name="$__object_id" +fi + +if [ -n "$( mysql -B -N -e "show databases like '$name'" )" ] +then + echo 'present' +else + echo 'absent' +fi diff --git a/cdist/conf/type/__mysql_database/gencode-remote b/cdist/conf/type/__mysql_database/gencode-remote index 23e51b05..d3692572 100755 --- a/cdist/conf/type/__mysql_database/gencode-remote +++ b/cdist/conf/type/__mysql_database/gencode-remote @@ -1,54 +1,28 @@ #!/bin/sh -e -# -# 2012 Benedikt Koeppel (code@benediktkoeppel.ch) -# -# This file is part of cdist. -# -# cdist is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# cdist is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with cdist. If not, see . -# -# -# if --database was specified -if [ -f "$__object/parameter/name" ]; then - database="$(cat "$__object/parameter/name")" -else # otherwise use the object id as database name - database="$__object_id" +state_is="$( cat "$__object/explorer/state" )" + +state_should="$( cat "$__object/parameter/state" )" + +if [ "$state_is" = "$state_should" ] +then + exit 0 fi -cat <<-EOFF -mysql -u root <<-EOF - CREATE DATABASE IF NOT EXISTS $database -EOF -EOFF - -# if --user was specified -if [ -f "$__object/parameter/user" ]; then - user="$(cat "$__object/parameter/user")" - - # if --password was specified - if [ -f "$__object/parameter/password" ]; then - password="$(cat "$__object/parameter/password")" - cat <<-EOFF - mysql -u root <<-EOF - GRANT ALL PRIVILEGES ON $database.* to '$user'@'localhost' IDENTIFIED BY '$password'; -EOF -EOFF - else - cat <<-EOFF - mysql -u root <<-EOF - GRANT ALL PRIVILEGES ON $database.* to '$user'@'localhost'; -EOF -EOFF - fi +if [ -f "$__object/parameter/name" ] +then + name="$( cat "$__object/parameter/name" )" +else + name="$__object_id" fi + +case "$state_should" in + present) + echo "mysql -e 'create database \`$name\`'" + echo "create database $name" >> "$__messages_out" + ;; + absent) + echo "mysql -e 'drop database \`$name\`'" + echo "drop database $name" >> "$__messages_out" + ;; +esac diff --git a/cdist/conf/type/__mysql_database/man.rst b/cdist/conf/type/__mysql_database/man.rst deleted file mode 100644 index 1e245a08..00000000 --- a/cdist/conf/type/__mysql_database/man.rst +++ /dev/null @@ -1,49 +0,0 @@ -cdist-type__mysql_database(7) -============================= - -NAME ----- -cdist-type__mysql_database - Manage a MySQL database - - -DESCRIPTION ------------ -This cdist type allows you to install a MySQL database. - - -REQUIRED PARAMETERS -------------------- -None. - -OPTIONAL PARAMETERS -------------------- -name - The name of the database to install - defaults to the object id - -user - A user that should have access to the database - -password - The password for the user who manages the database - - -EXAMPLES --------- - -.. code-block:: sh - - __mysql_database "cdist" --name "cdist" --user "myuser" --password "mypwd" - - -AUTHORS -------- -Benedikt Koeppel - - -COPYING -------- -Copyright \(C) 2012 Benedikt Koeppel. You can redistribute it -and/or modify it under the terms of the GNU General Public License as -published by the Free Software Foundation, either version 3 of the -License, or (at your option) any later version. diff --git a/cdist/conf/type/__mysql_database/manifest b/cdist/conf/type/__mysql_database/manifest new file mode 100755 index 00000000..a57c31ce --- /dev/null +++ b/cdist/conf/type/__mysql_database/manifest @@ -0,0 +1,26 @@ +#!/bin/sh -e + +if [ -f "$__object/parameter/user" ] +then + user="$( cat "$__object/parameter/user" )" +fi + +if [ -f "$__object/parameter/password" ] +then + password="$( cat "$__object/parameter/password" )" +fi + +if [ -n "$user" ] && [ -n "$password" ] +then + if [ -f "$__object/parameter/name" ] + then + database="$( cat "$__object/parameter/name" )" + else + database="$__object_id" + fi + + __mysql_user "$user" --password "$password" + + require="__mysql_user/$user" \ + __mysql_privileges "$database/$user" --database "$database" --user "$user" +fi diff --git a/cdist/conf/type/__mysql_database/parameter/default/state b/cdist/conf/type/__mysql_database/parameter/default/state new file mode 100644 index 00000000..e7f6134f --- /dev/null +++ b/cdist/conf/type/__mysql_database/parameter/default/state @@ -0,0 +1 @@ +present diff --git a/cdist/conf/type/__mysql_database/parameter/optional b/cdist/conf/type/__mysql_database/parameter/optional index 756afee7..6c0b1e85 100644 --- a/cdist/conf/type/__mysql_database/parameter/optional +++ b/cdist/conf/type/__mysql_database/parameter/optional @@ -1,3 +1,4 @@ name user password +state diff --git a/cdist/conf/type/__mysql_privileges/explorer/state b/cdist/conf/type/__mysql_privileges/explorer/state new file mode 100755 index 00000000..97674479 --- /dev/null +++ b/cdist/conf/type/__mysql_privileges/explorer/state @@ -0,0 +1,22 @@ +#!/bin/sh -e + +privileges="$( cat "$__object/parameter/privileges" )" + +database="$( cat "$__object/parameter/database" )" + +table="$( cat "$__object/parameter/table" )" + +user="$( cat "$__object/parameter/user" )" + +host="$( cat "$__object/parameter/host" )" + +check_privileges="$( + mysql -B -N -e "show grants for '$user'@'$host'" \ + | grep -Ei "^grant $privileges on .$database.\..$table. to " || true )" + +if [ -n "$check_privileges" ] +then + echo 'present' +else + echo 'absent' +fi diff --git a/cdist/conf/type/__mysql_privileges/gencode-remote b/cdist/conf/type/__mysql_privileges/gencode-remote new file mode 100755 index 00000000..6b2e0fc1 --- /dev/null +++ b/cdist/conf/type/__mysql_privileges/gencode-remote @@ -0,0 +1,31 @@ +#!/bin/sh -e + +state_is="$( cat "$__object/explorer/state" )" + +state_should="$( cat "$__object/parameter/state" )" + +if [ "$state_is" = "$state_should" ] +then + exit 0 +fi + +privileges="$( cat "$__object/parameter/privileges" )" + +database="$( cat "$__object/parameter/database" )" + +table="$( cat "$__object/parameter/table" )" + +user="$( cat "$__object/parameter/user" )" + +host="$( cat "$__object/parameter/host" )" + +case "$state_should" in + present) + echo "mysql -e 'grant $privileges on \`$database\`.\`$table\` to \`$user\`@\`$host\`'" + echo "grant $privileges on $database.$table to $user@$host" >> "$__messages_out" + ;; + absent) + echo "mysql -e 'revoke $privileges on \`$database\`.\`$table\` from \`$user\`@\`$host\`'" + echo "revoke $privileges on $database.$table from $user@$host" >> "$__messages_out" + ;; +esac diff --git a/cdist/conf/type/__mysql_privileges/parameter/default/host b/cdist/conf/type/__mysql_privileges/parameter/default/host new file mode 100644 index 00000000..2fbb50c4 --- /dev/null +++ b/cdist/conf/type/__mysql_privileges/parameter/default/host @@ -0,0 +1 @@ +localhost diff --git a/cdist/conf/type/__mysql_privileges/parameter/default/privileges b/cdist/conf/type/__mysql_privileges/parameter/default/privileges new file mode 100644 index 00000000..5472efad --- /dev/null +++ b/cdist/conf/type/__mysql_privileges/parameter/default/privileges @@ -0,0 +1 @@ +all privileges diff --git a/cdist/conf/type/__mysql_privileges/parameter/default/state b/cdist/conf/type/__mysql_privileges/parameter/default/state new file mode 100644 index 00000000..e7f6134f --- /dev/null +++ b/cdist/conf/type/__mysql_privileges/parameter/default/state @@ -0,0 +1 @@ +present diff --git a/cdist/conf/type/__mysql_privileges/parameter/default/table b/cdist/conf/type/__mysql_privileges/parameter/default/table new file mode 100644 index 00000000..72e8ffc0 --- /dev/null +++ b/cdist/conf/type/__mysql_privileges/parameter/default/table @@ -0,0 +1 @@ +* diff --git a/cdist/conf/type/__mysql_privileges/parameter/optional b/cdist/conf/type/__mysql_privileges/parameter/optional new file mode 100644 index 00000000..d4ed5bc5 --- /dev/null +++ b/cdist/conf/type/__mysql_privileges/parameter/optional @@ -0,0 +1,4 @@ +privileges +table +host +state diff --git a/cdist/conf/type/__mysql_privileges/parameter/required b/cdist/conf/type/__mysql_privileges/parameter/required new file mode 100644 index 00000000..152b4a1e --- /dev/null +++ b/cdist/conf/type/__mysql_privileges/parameter/required @@ -0,0 +1,2 @@ +database +user diff --git a/cdist/conf/type/__mysql_user/explorer/state b/cdist/conf/type/__mysql_user/explorer/state new file mode 100755 index 00000000..c91bb36a --- /dev/null +++ b/cdist/conf/type/__mysql_user/explorer/state @@ -0,0 +1,36 @@ +#!/bin/sh -e + +if [ -f "$__object/parameter/name" ] +then + name="$( cat "$__object/parameter/name" )" +else + name="$__object_id" +fi + +if [ -f "$__object/parameter/password" ] +then + password="$( cat "$__object/parameter/password" )" +else + password='' +fi + +host="$( cat "$__object/parameter/host" )" + +check_user="$( mysql -B -N -e "select user from mysql.user where user = '$name' and host = '$host'" )" + +if [ -n "$check_user" ] +then + if [ -n "$password" ] + then + check_password="$( mysql -B -N -e "select user from mysql.user where user = '$name' and host = '$host' and password = password( '$password' )" )" + fi + + if [ -n "$password" ] && [ -z "$check_password" ] + then + echo 'change-password' + else + echo 'present' + fi +else + echo 'absent' +fi diff --git a/cdist/conf/type/__mysql_user/gencode-remote b/cdist/conf/type/__mysql_user/gencode-remote new file mode 100755 index 00000000..67500716 --- /dev/null +++ b/cdist/conf/type/__mysql_user/gencode-remote @@ -0,0 +1,50 @@ +#!/bin/sh -e + +state_is="$( cat "$__object/explorer/state" )" + +state_should="$( cat "$__object/parameter/state" )" + +if [ "$state_is" = "$state_should" ] +then + exit 0 +fi + +if [ -f "$__object/parameter/name" ] +then + name="$( cat "$__object/parameter/name" )" +else + name="$__object_id" +fi + +host="$( cat "$__object/parameter/host" )" + +if [ -f "$__object/parameter/password" ] +then + password="$( cat "$__object/parameter/password" )" +else + if [ "$state_should" = 'present' ] + then + echo '--password needed' >&2 + exit 1 + else + password='' + fi +fi + +if [ "$state_is" = 'absent' ] && [ "$state_should" = 'present' ] +then + echo "mysql -e 'create user \`$name\`@\`$host\` identified by \"$password\"'" + echo "create user $name@$host" >> "$__messages_out" + +elif [ "$state_is" != 'absent' ] && [ "$state_should" = 'absent' ] +then + echo "mysql -e 'drop user \`$name\`@\`$host\`'" + echo "drop user $name@$host" >> "$__messages_out" + +elif [ "$state_is" = 'change-password' ] +then + # this only works with MySQL 5.7.6 and later or MariaDB 10.1.20 and later + echo "mysql -e 'alter user \`$name\`@\`$host\` identified by \"$password\"'" + echo "mysql -e 'flush privileges'" + echo "change password $name@$host" >> "$__messages_out" +fi diff --git a/cdist/conf/type/__mysql_user/parameter/default/host b/cdist/conf/type/__mysql_user/parameter/default/host new file mode 100644 index 00000000..2fbb50c4 --- /dev/null +++ b/cdist/conf/type/__mysql_user/parameter/default/host @@ -0,0 +1 @@ +localhost diff --git a/cdist/conf/type/__mysql_user/parameter/default/state b/cdist/conf/type/__mysql_user/parameter/default/state new file mode 100644 index 00000000..e7f6134f --- /dev/null +++ b/cdist/conf/type/__mysql_user/parameter/default/state @@ -0,0 +1 @@ +present diff --git a/cdist/conf/type/__mysql_user/parameter/optional b/cdist/conf/type/__mysql_user/parameter/optional new file mode 100644 index 00000000..a286266c --- /dev/null +++ b/cdist/conf/type/__mysql_user/parameter/optional @@ -0,0 +1,4 @@ +name +host +password +state From 9a693537f4192e3cd133e14cd31ab1bdcc792608 Mon Sep 17 00:00:00 2001 From: Ander Punnar Date: Wed, 1 Jan 2020 12:38:12 +0200 Subject: [PATCH 20/38] __mysql_*: add license headers --- .../conf/type/__mysql_database/explorer/state | 18 ++++++++++++++++++ .../conf/type/__mysql_database/gencode-remote | 18 ++++++++++++++++++ cdist/conf/type/__mysql_database/manifest | 18 ++++++++++++++++++ .../type/__mysql_privileges/explorer/state | 18 ++++++++++++++++++ .../type/__mysql_privileges/gencode-remote | 18 ++++++++++++++++++ cdist/conf/type/__mysql_user/explorer/state | 18 ++++++++++++++++++ cdist/conf/type/__mysql_user/gencode-remote | 18 ++++++++++++++++++ 7 files changed, 126 insertions(+) diff --git a/cdist/conf/type/__mysql_database/explorer/state b/cdist/conf/type/__mysql_database/explorer/state index 16cc9ce5..79858695 100755 --- a/cdist/conf/type/__mysql_database/explorer/state +++ b/cdist/conf/type/__mysql_database/explorer/state @@ -1,4 +1,22 @@ #!/bin/sh -e +# +# 2020 Ander Punnar (ander-at-kvlt-dot-ee) +# +# This file is part of cdist. +# +# cdist is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# cdist is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with cdist. If not, see . +# if [ -f "$__object/parameter/name" ] then diff --git a/cdist/conf/type/__mysql_database/gencode-remote b/cdist/conf/type/__mysql_database/gencode-remote index d3692572..1bdb2b11 100755 --- a/cdist/conf/type/__mysql_database/gencode-remote +++ b/cdist/conf/type/__mysql_database/gencode-remote @@ -1,4 +1,22 @@ #!/bin/sh -e +# +# 2020 Ander Punnar (ander-at-kvlt-dot-ee) +# +# This file is part of cdist. +# +# cdist is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# cdist is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with cdist. If not, see . +# state_is="$( cat "$__object/explorer/state" )" diff --git a/cdist/conf/type/__mysql_database/manifest b/cdist/conf/type/__mysql_database/manifest index a57c31ce..628b543c 100755 --- a/cdist/conf/type/__mysql_database/manifest +++ b/cdist/conf/type/__mysql_database/manifest @@ -1,4 +1,22 @@ #!/bin/sh -e +# +# 2020 Ander Punnar (ander-at-kvlt-dot-ee) +# +# This file is part of cdist. +# +# cdist is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# cdist is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with cdist. If not, see . +# if [ -f "$__object/parameter/user" ] then diff --git a/cdist/conf/type/__mysql_privileges/explorer/state b/cdist/conf/type/__mysql_privileges/explorer/state index 97674479..0cfbaacd 100755 --- a/cdist/conf/type/__mysql_privileges/explorer/state +++ b/cdist/conf/type/__mysql_privileges/explorer/state @@ -1,4 +1,22 @@ #!/bin/sh -e +# +# 2020 Ander Punnar (ander-at-kvlt-dot-ee) +# +# This file is part of cdist. +# +# cdist is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# cdist is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with cdist. If not, see . +# privileges="$( cat "$__object/parameter/privileges" )" diff --git a/cdist/conf/type/__mysql_privileges/gencode-remote b/cdist/conf/type/__mysql_privileges/gencode-remote index 6b2e0fc1..bcd362e6 100755 --- a/cdist/conf/type/__mysql_privileges/gencode-remote +++ b/cdist/conf/type/__mysql_privileges/gencode-remote @@ -1,4 +1,22 @@ #!/bin/sh -e +# +# 2020 Ander Punnar (ander-at-kvlt-dot-ee) +# +# This file is part of cdist. +# +# cdist is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# cdist is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with cdist. If not, see . +# state_is="$( cat "$__object/explorer/state" )" diff --git a/cdist/conf/type/__mysql_user/explorer/state b/cdist/conf/type/__mysql_user/explorer/state index c91bb36a..6817ee9d 100755 --- a/cdist/conf/type/__mysql_user/explorer/state +++ b/cdist/conf/type/__mysql_user/explorer/state @@ -1,4 +1,22 @@ #!/bin/sh -e +# +# 2020 Ander Punnar (ander-at-kvlt-dot-ee) +# +# This file is part of cdist. +# +# cdist is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# cdist is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with cdist. If not, see . +# if [ -f "$__object/parameter/name" ] then diff --git a/cdist/conf/type/__mysql_user/gencode-remote b/cdist/conf/type/__mysql_user/gencode-remote index 67500716..5f13bc87 100755 --- a/cdist/conf/type/__mysql_user/gencode-remote +++ b/cdist/conf/type/__mysql_user/gencode-remote @@ -1,4 +1,22 @@ #!/bin/sh -e +# +# 2020 Ander Punnar (ander-at-kvlt-dot-ee) +# +# This file is part of cdist. +# +# cdist is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# cdist is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with cdist. If not, see . +# state_is="$( cat "$__object/explorer/state" )" From 24862e0208705cd9081f5963d1f07e0fcceb23f2 Mon Sep 17 00:00:00 2001 From: Ander Punnar Date: Fri, 3 Jan 2020 18:26:11 +0200 Subject: [PATCH 21/38] __mysql_database: carry over state --- cdist/conf/type/__mysql_database/manifest | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/cdist/conf/type/__mysql_database/manifest b/cdist/conf/type/__mysql_database/manifest index 628b543c..a3c9ed5d 100755 --- a/cdist/conf/type/__mysql_database/manifest +++ b/cdist/conf/type/__mysql_database/manifest @@ -37,8 +37,16 @@ then database="$__object_id" fi - __mysql_user "$user" --password "$password" + state_should="$( cat "$__object/parameter/state" )" + __mysql_user "$user" \ + --password "$password" \ + --state "$state_should" + + # removing user should remove all user's privileges require="__mysql_user/$user" \ - __mysql_privileges "$database/$user" --database "$database" --user "$user" + __mysql_privileges "$database/$user" \ + --database "$database" \ + --user "$user" \ + --state "$state_should" fi From fcc774cb7b2b3f2128dc77d622d7801397d906b7 Mon Sep 17 00:00:00 2001 From: Ander Punnar Date: Fri, 3 Jan 2020 18:33:23 +0200 Subject: [PATCH 22/38] __mysql_database: add manual --- cdist/conf/type/__mysql_database/man.rst | 55 ++++++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 cdist/conf/type/__mysql_database/man.rst diff --git a/cdist/conf/type/__mysql_database/man.rst b/cdist/conf/type/__mysql_database/man.rst new file mode 100644 index 00000000..b3b56b5f --- /dev/null +++ b/cdist/conf/type/__mysql_database/man.rst @@ -0,0 +1,55 @@ +cdist-type__mysql_database(7) +============================= + +NAME +---- +cdist-type__mysql_database - Manage a MySQL database + + +DESCRIPTION +----------- + +Create MySQL database and optionally user with all privileges. + + +OPTIONAL PARAMETERS +------------------- +name + Name of database. Defaults to object id. + +user + Create user and give all privileges to database. + +password + Password for user. + +state + Defaults to present. + If absent and user is also set, both will be removed (with privileges). + + +EXAMPLES +-------- + +.. code-block:: sh + + # just create database + __mysql_database foo + + # create database with respective user with all privileges to database + __mysql_database bar \ + --user name \ + --password secret + + +AUTHORS +------- +Ander Punnar + + +COPYING +------- +Copyright \(C) 2020 Ander Punnar. You can redistribute it and/or modify it +under the terms of the GNU General Public License as published by the Free +Software Foundation, either version 3 of the License, or (at your option) any +later version. From 5e8dc7122d764896247258ae4c9049e222f4d7ff Mon Sep 17 00:00:00 2001 From: Ander Punnar Date: Fri, 3 Jan 2020 18:48:11 +0200 Subject: [PATCH 23/38] __mysql_user: add manual --- cdist/conf/type/__mysql_user/man.rst | 48 ++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 cdist/conf/type/__mysql_user/man.rst diff --git a/cdist/conf/type/__mysql_user/man.rst b/cdist/conf/type/__mysql_user/man.rst new file mode 100644 index 00000000..c2b222d5 --- /dev/null +++ b/cdist/conf/type/__mysql_user/man.rst @@ -0,0 +1,48 @@ +cdist-type__mysql_user(7) +========================= + +NAME +---- +cdist-type__mysql_user - Manage a MySQL user + + +DESCRIPTION +----------- + +Create MySQL user or change password for the user. + + +OPTIONAL PARAMETERS +------------------- +name + Name of user. Defaults to object id. + +host + Host of user. Defaults to localhost. + +password + Password of user. + +state + Defaults to present. + + +EXAMPLES +-------- + +.. code-block:: sh + + __mysql_user user --password secret + + +AUTHORS +------- +Ander Punnar + + +COPYING +------- +Copyright \(C) 2020 Ander Punnar. You can redistribute it and/or modify it +under the terms of the GNU General Public License as published by the Free +Software Foundation, either version 3 of the License, or (at your option) any +later version. From 4329cced82930a336378765b294b9a3be9433991 Mon Sep 17 00:00:00 2001 From: Ander Punnar Date: Fri, 3 Jan 2020 18:55:55 +0200 Subject: [PATCH 24/38] __mysql_privileges: add manual --- cdist/conf/type/__mysql_privileges/man.rst | 57 ++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 cdist/conf/type/__mysql_privileges/man.rst diff --git a/cdist/conf/type/__mysql_privileges/man.rst b/cdist/conf/type/__mysql_privileges/man.rst new file mode 100644 index 00000000..8208d7d4 --- /dev/null +++ b/cdist/conf/type/__mysql_privileges/man.rst @@ -0,0 +1,57 @@ +cdist-type__mysql_privileges(7) +=============================== + +NAME +---- +cdist-type__mysql_privileges - Manage MySQL privileges + + +DESCRIPTION +----------- + +Grant and revoke privileges of MySQL user. + + +REQUIRED PARAMETERS +------------------- +database + Name of database. + +User + Name of user. + + +OPTIONAL PARAMETERS +------------------- +privileges + Defaults to "all". + +table + Defaults to "*". + +host + Defaults to localhost. + +state + "present" grants and "absent" revokes. Defaults to present. + + +EXAMPLES +-------- + +.. code-block:: sh + + __mysql_privileges user-to-db --database db --user user + + +AUTHORS +------- +Ander Punnar + + +COPYING +------- +Copyright \(C) 2020 Ander Punnar. You can redistribute it and/or modify it +under the terms of the GNU General Public License as published by the Free +Software Foundation, either version 3 of the License, or (at your option) any +later version. From 6369bc1ae5e17dd9c0a13d57419107cbdb166245 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Sat, 4 Jan 2020 10:07:53 +0100 Subject: [PATCH 25/38] ++changelog --- docs/changelog | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/changelog b/docs/changelog index 415af0af..8feb0b83 100644 --- a/docs/changelog +++ b/docs/changelog @@ -2,8 +2,9 @@ Changelog --------- next: - * Type __consul_agent: don't deploy init script on Alpine anymore, - it ships with one itself (Nico Schottelius) + * Type __consul_agent: Don't deploy init script on Alpine anymore, it ships with one itself (Nico Schottelius) + * Type __install_chroot_umount: Bugfix: type was not using __chroot_umount/manifest (Steven Armstrong) + * Types __postgres_*: Use double quoted identifiers in generated SQL (fnux) 6.3.0: 2019-12-12 * Type __package_update_index: Fix Alpine part (Dominique Roux) From 72935e0a797461d39f64b9f85bac02ad06347b1d Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Sat, 4 Jan 2020 11:08:48 +0100 Subject: [PATCH 26/38] ++changelog --- docs/changelog | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelog b/docs/changelog index 8feb0b83..d0a3a2de 100644 --- a/docs/changelog +++ b/docs/changelog @@ -4,7 +4,7 @@ Changelog next: * Type __consul_agent: Don't deploy init script on Alpine anymore, it ships with one itself (Nico Schottelius) * Type __install_chroot_umount: Bugfix: type was not using __chroot_umount/manifest (Steven Armstrong) - * Types __postgres_*: Use double quoted identifiers in generated SQL (fnux) + * Types __postgres_*: Use double quoted identifiers in generated SQL (Timothée Floure) 6.3.0: 2019-12-12 * Type __package_update_index: Fix Alpine part (Dominique Roux) From e4596593c08f43284e1fa85d1e353b733bebdc57 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Mon, 30 Dec 2019 09:32:15 +0100 Subject: [PATCH 27/38] Add cdist info command --- cdist/argparse.py | 32 +++++++ cdist/exec/local.py | 42 ++------- cdist/exec/util.py | 25 ++++++ cdist/info.py | 183 ++++++++++++++++++++++++++++++++++++++++ docs/src/man1/cdist.rst | 37 +++++++- 5 files changed, 283 insertions(+), 36 deletions(-) create mode 100644 cdist/info.py diff --git a/cdist/argparse.py b/cdist/argparse.py index 7dc683f3..611c484a 100644 --- a/cdist/argparse.py +++ b/cdist/argparse.py @@ -6,6 +6,7 @@ import collections import functools import cdist.configuration import cdist.preos +import cdist.info # set of beta sub-commands @@ -436,6 +437,37 @@ def get_parsers(): ' should be POSIX compatible shell.')) parser['shell'].set_defaults(func=cdist.shell.Shell.commandline) + # Info + parser['info'] = parser['sub'].add_parser('info') + parser['info'].add_argument( + '-a', '--all', help='Display all info. This is the default.', + action='store_true', default=False) + parser['info'].add_argument( + '-c', '--conf-dir', + help='Add configuration directory (can be repeated).', + action='append') + parser['info'].add_argument( + '-e', '--global-explorers', + help='Display info for global explorers.', action='store_true', + default=False) + parser['info'].add_argument( + '-F', '--fixed-string', + help='Interpret pattern as a fixed string.', action='store_true', + default=False) + parser['info'].add_argument( + '-f', '--full', help='Display full details.', + action='store_true', default=False) + parser['info'].add_argument( + '-g', '--config-file', + help='Use specified custom configuration file.', + dest="config_file", required=False) + parser['info'].add_argument( + '-t', '--types', help='Display info for types.', + action='store_true', default=False) + parser['info'].add_argument( + 'pattern', nargs='?', help='Glob pattern.') + parser['info'].set_defaults(func=cdist.info.Info.commandline) + for p in parser: parser[p].epilog = EPILOG diff --git a/cdist/exec/local.py b/cdist/exec/local.py index f83c85df..ad6c6e36 100644 --- a/cdist/exec/local.py +++ b/cdist/exec/local.py @@ -69,7 +69,6 @@ class Local(object): self.exec_path = exec_path self.custom_initial_manifest = initial_manifest - self._add_conf_dirs = add_conf_dirs self.cache_path_pattern = cache_path_pattern self.quiet_mode = quiet_mode if configuration: @@ -84,16 +83,7 @@ class Local(object): self._init_cache_dir(None) self._init_paths() self._init_object_marker() - self._init_conf_dirs() - - @property - def dist_conf_dir(self): - return os.path.abspath(os.path.join(os.path.dirname(cdist.__file__), - "conf")) - - @property - def home_dir(self): - return cdist.home_dir() + self._init_conf_dirs(add_conf_dirs) def _init_log(self): self.log = logging.getLogger(self.target_host[0]) @@ -140,28 +130,9 @@ class Local(object): # Does not need to be secure - just randomly different from .cdist self.object_marker_name = tempfile.mktemp(prefix='.cdist-', dir='') - def _init_conf_dirs(self): - self.conf_dirs = [] - - self.conf_dirs.append(self.dist_conf_dir) - - # Is the default place for user created explorer, type and manifest - if self.home_dir: - self.conf_dirs.append(self.home_dir) - - # Add directories defined in the CDIST_PATH environment variable - # if 'CDIST_PATH' in os.environ: - # cdist_path_dirs = re.split(r'(?. +# +# + +import cdist +import cdist.configuration +import cdist.core +import cdist.exec.util as util +import os +import glob +import fnmatch + + +class Info(object): + + def __init__(self, conf_dirs, args): + self.conf_dirs = conf_dirs + self.all = args.all + self.display_global_explorers = args.global_explorers + self.display_types = args.types + if not self.display_global_explorers and not self.display_types: + self.all = True + self.fixed_string = args.fixed_string + self._setup_glob_pattern(args.pattern) + self.full = args.full + + def _setup_glob_pattern(self, pattern): + if pattern is None: + self.glob_pattern = '*' + elif ('?' in pattern or '*' in pattern or '[' in pattern or + self.fixed_string): + self.glob_pattern = pattern + else: + self.glob_pattern = '*' + pattern + '*' + + @classmethod + def commandline(cls, args): + cfg = cdist.configuration.Configuration(args) + configuration = cfg.get_config(section='GLOBAL') + conf_dirs = util.resolve_conf_dirs(configuration, + args.conf_dir) + c = cls(conf_dirs, args) + c.run() + + def _get_global_explorers(self, conf_path): + rv = [] + global_explorer_path = os.path.join(conf_path, "explorer", + self.glob_pattern) + if self.fixed_string: + if os.path.exists(global_explorer_path): + rv.append(global_explorer_path) + else: + for explorer in glob.glob(global_explorer_path): + rv.append(explorer) + return rv + + def _should_display_type(self, dir_entry): + if not dir_entry.is_dir(): + return False + if self.glob_pattern is None: + return True + if self.fixed_string: + return dir_entry.name == self.glob_pattern + else: + return fnmatch.fnmatch(dir_entry.name, self.glob_pattern) + + def _get_types(self, conf_path): + rv = [] + types_path = os.path.join(conf_path, "type") + if not os.path.exists(types_path): + return rv + with os.scandir(types_path) as it: + for entry in it: + if self._should_display_type(entry): + rv.append(entry.path) + return rv + + def _display_details(self, title, details, default_values=None, + deprecated=None): + if not details: + return + if isinstance(details, bool): + print("\t{}: {}".format(title, 'yes' if details else 'no')) + elif isinstance(details, str): + print("\t{}: {}".format(title, details)) + elif isinstance(details, list): + dv = dict(default_values) if default_values else {} + dp = dict(deprecated) if deprecated else {} + + print("\t{}:".format(title)) + for x in sorted(details): + print("\t\t{}".format(x), end='') + has_default = x in dv + is_deprecated = x in dp + need_comma = False + if has_default or is_deprecated: + print(" (", end='') + if has_default: + print("default: {}".format(dv[x]), end='') + need_comma = True + if is_deprecated: + print("{}deprecated".format(', ' if need_comma else ''), + end='') + if has_default or is_deprecated: + print(")", end='') + print() + + def _display_type_parameters(self, cdist_type): + self._display_details("required parameters", + cdist_type.required_parameters, + default_values=cdist_type.parameter_defaults, + deprecated=cdist_type.deprecated_parameters) + self._display_details("required multiple parameters", + cdist_type.required_multiple_parameters, + default_values=cdist_type.parameter_defaults, + deprecated=cdist_type.deprecated_parameters) + self._display_details("optional parameters", + cdist_type.optional_parameters, + default_values=cdist_type.parameter_defaults, + deprecated=cdist_type.deprecated_parameters) + self._display_details("optional multiple parameters", + cdist_type.optional_multiple_parameters, + default_values=cdist_type.parameter_defaults, + deprecated=cdist_type.deprecated_parameters) + self._display_details("boolean parameters", + cdist_type.boolean_parameters, + default_values=cdist_type.parameter_defaults, + deprecated=cdist_type.deprecated_parameters) + + def _display_type_characteristics(self, cdist_type): + characteristics = [] + if cdist_type.is_install: + characteristics.append('install') + else: + characteristics.append('config') + if cdist_type.is_singleton: + characteristics.append('singleton') + if cdist_type.is_nonparallel: + characteristics.append('nonparallel') + else: + characteristics.append('parallel') + if cdist_type.deprecated is not None: + characteristics.append('deprecated') + print("\t{}".format(', '.join(characteristics))) + + def _display_type_details(self, type_path): + dirname, basename = os.path.split(type_path) + cdist_type = cdist.core.CdistType(dirname, basename) + + self._display_type_characteristics(cdist_type) + self._display_type_parameters(cdist_type) + + def run(self): + rv = [] + for conf_path in self.conf_dirs: + if self.all or self.display_global_explorers: + rv.extend((x, 'E', ) for x in self._get_global_explorers( + conf_path)) + if self.all or self.display_types: + rv.extend((x, 'T', ) for x in self._get_types(conf_path)) + rv = sorted(rv, key=lambda x: x[0]) + for x, t in rv: + print(x) + if self.full and t == 'T': + self._display_type_details(x) diff --git a/docs/src/man1/cdist.rst b/docs/src/man1/cdist.rst index 55db82ed..66c356ec 100644 --- a/docs/src/man1/cdist.rst +++ b/docs/src/man1/cdist.rst @@ -11,7 +11,7 @@ SYNOPSIS :: - cdist [-h] [-V] {banner,config,install,inventory,preos,shell} ... + cdist [-h] [-V] {banner,config,install,inventory,preos,shell,info} ... cdist banner [-h] [-l LOGLEVEL] [-q] [-v] @@ -84,6 +84,8 @@ SYNOPSIS cdist shell [-h] [-l LOGLEVEL] [-q] [-v] [-s SHELL] + cdist info [-h] [-a] [-c CONF_DIR] [-e] [-F] [-f] [-t] [pattern] + DESCRIPTION ----------- @@ -604,6 +606,39 @@ usage. Its primary use is for debugging type parameters. be POSIX compatible shell. +INFO +---- +Display information for cdist (global explorers, types). + +**pattern** + Glob pattern. If it contains special characters('?', '*', '[') then it is + used as specified, otherwise it is translated to `*pattern*`. + +**-h, --help** + Show help message and exit. + +**-a, --all** + Display all info. This is the default. + +**-c CONF_DIR, --conf-dir CONF_DIR** + Add configuration directory (can be repeated). + +**-e, --global-explorers** + Display info for global explorers. + +**-F, --fixed-string** + Interpret pattern as a fixed string. + +**-f, --full** + Display full details. + +**-g CONFIG_FILE, --config-file CONFIG_FILE** + Use specified custom configuration file. + +**-t, --types** + Display info for types. + + CONFIGURATION ------------- cdist obtains configuration data from the following sources in the following From 7b1192257d8517cc838f84a8c57ac0964d104588 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Sat, 4 Jan 2020 13:17:54 +0100 Subject: [PATCH 28/38] Fix incomplete cdist info synopsis --- docs/src/man1/cdist.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/man1/cdist.rst b/docs/src/man1/cdist.rst index 66c356ec..bc73a0b8 100644 --- a/docs/src/man1/cdist.rst +++ b/docs/src/man1/cdist.rst @@ -84,7 +84,7 @@ SYNOPSIS cdist shell [-h] [-l LOGLEVEL] [-q] [-v] [-s SHELL] - cdist info [-h] [-a] [-c CONF_DIR] [-e] [-F] [-f] [-t] [pattern] + cdist info [-h] [-a] [-c CONF_DIR] [-e] [-F] [-f] [-g CONFIG_FILE] [-t] [pattern] DESCRIPTION From e2015367925c8a2716e6a79f6f2609f2877cd134 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Sat, 4 Jan 2020 13:18:13 +0100 Subject: [PATCH 29/38] ++changelog --- docs/changelog | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/changelog b/docs/changelog index d0a3a2de..7489489e 100644 --- a/docs/changelog +++ b/docs/changelog @@ -5,6 +5,7 @@ next: * Type __consul_agent: Don't deploy init script on Alpine anymore, it ships with one itself (Nico Schottelius) * Type __install_chroot_umount: Bugfix: type was not using __chroot_umount/manifest (Steven Armstrong) * Types __postgres_*: Use double quoted identifiers in generated SQL (Timothée Floure) + * Core: Add cdist info command (Darko Poljak) 6.3.0: 2019-12-12 * Type __package_update_index: Fix Alpine part (Dominique Roux) From bc1990c7c8244b2d00a91bc13dbf796d91c21041 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Sat, 4 Jan 2020 13:44:04 +0100 Subject: [PATCH 30/38] ++changelog --- docs/changelog | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/changelog b/docs/changelog index 7489489e..fc4d69a8 100644 --- a/docs/changelog +++ b/docs/changelog @@ -6,6 +6,8 @@ next: * Type __install_chroot_umount: Bugfix: type was not using __chroot_umount/manifest (Steven Armstrong) * Types __postgres_*: Use double quoted identifiers in generated SQL (Timothée Floure) * Core: Add cdist info command (Darko Poljak) + * New types: __mysql_user, __mysql_privileges (Ander Punnar) + * Type __mysql_database: Rewrite (Ander Punnar) 6.3.0: 2019-12-12 * Type __package_update_index: Fix Alpine part (Dominique Roux) From 7c9dd3b03e31f39cbe758510e2aa1f542eae4825 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Sat, 4 Jan 2020 14:36:14 +0100 Subject: [PATCH 31/38] Release 6.4.0 --- docs/changelog | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelog b/docs/changelog index fc4d69a8..706d76af 100644 --- a/docs/changelog +++ b/docs/changelog @@ -1,7 +1,7 @@ Changelog --------- -next: +6.4.0: 2020-01-04 * Type __consul_agent: Don't deploy init script on Alpine anymore, it ships with one itself (Nico Schottelius) * Type __install_chroot_umount: Bugfix: type was not using __chroot_umount/manifest (Steven Armstrong) * Types __postgres_*: Use double quoted identifiers in generated SQL (Timothée Floure) From d1a64596fe73697d3e6a8e514991ffc173d04772 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Sat, 4 Jan 2020 14:56:05 +0100 Subject: [PATCH 32/38] Update build-helper --- bin/build-helper | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/build-helper b/bin/build-helper index 69dee4c7..ed41e438 100755 --- a/bin/build-helper +++ b/bin/build-helper @@ -370,7 +370,7 @@ eof cat << eof Manual steps post release: - cdist-web - - send mail body generated in mailinglist.tmp and inform Dmitry for deb + - send generated mailinglist.tmp mail - twitter eof ;; From d4bd49bbb598dfd4e4a510a2bf9035dcb4686e4a Mon Sep 17 00:00:00 2001 From: Ander Punnar Date: Sat, 4 Jan 2020 17:43:57 +0200 Subject: [PATCH 33/38] __acl: rename --acl to --entry for the sake of consistency, add compatibility --- cdist/conf/type/__acl/gencode-remote | 5 +++- cdist/conf/type/__acl/man.rst | 28 +++++++++---------- .../conf/type/__acl/parameter/deprecated/acl | 1 + .../type/__acl/parameter/optional_multiple | 1 + 4 files changed, 20 insertions(+), 15 deletions(-) create mode 100644 cdist/conf/type/__acl/parameter/deprecated/acl diff --git a/cdist/conf/type/__acl/gencode-remote b/cdist/conf/type/__acl/gencode-remote index 6dab4d09..f4f0d1e2 100755 --- a/cdist/conf/type/__acl/gencode-remote +++ b/cdist/conf/type/__acl/gencode-remote @@ -28,7 +28,10 @@ acl_path="/$__object_id" acl_is="$( cat "$__object/explorer/acl_is" )" -if [ -f "$__object/parameter/acl" ] +if [ -f "$__object/parameter/entry" ] +then + acl_should="$( cat "$__object/parameter/entry" )" +elif [ -f "$__object/parameter/acl" ] then acl_should="$( cat "$__object/parameter/acl" )" elif diff --git a/cdist/conf/type/__acl/man.rst b/cdist/conf/type/__acl/man.rst index 85e946ce..c3493e49 100644 --- a/cdist/conf/type/__acl/man.rst +++ b/cdist/conf/type/__acl/man.rst @@ -15,7 +15,7 @@ See ``setfacl`` and ``acl`` manpages for more details. REQUIRED MULTIPLE PARAMETERS ---------------------------- -acl +entry Set ACL entry following ``getfacl`` output syntax. @@ -36,8 +36,8 @@ remove DEPRECATED PARAMETERS --------------------- -Parameters ``user``, ``group``, ``mask`` and ``other`` are deprecated and they -will be removed in future versions. Please use ``acl`` parameter instead. +Parameters ``acl``, ``user``, ``group``, ``mask`` and ``other`` are deprecated and they +will be removed in future versions. Please use ``entry`` parameter instead. EXAMPLES @@ -49,27 +49,27 @@ EXAMPLES --default \ --recursive \ --remove \ - --acl user:alice:rwx \ - --acl user:bob:r-x \ - --acl group:project-group:rwx \ - --acl group:some-other-group:r-x \ - --acl mask::r-x \ - --acl other::r-x + --entry user:alice:rwx \ + --entry user:bob:r-x \ + --entry group:project-group:rwx \ + --entry group:some-other-group:r-x \ + --entry mask::r-x \ + --entry other::r-x # give Alice read-only access to subdir, # but don't allow her to see parent content. __acl /srv/project2 \ --remove \ - --acl default:group:secret-project:rwx \ - --acl group:secret-project:rwx \ - --acl user:alice:--x + --entry default:group:secret-project:rwx \ + --entry group:secret-project:rwx \ + --entry user:alice:--x __acl /srv/project2/subdir \ --default \ --remove \ - --acl group:secret-project:rwx \ - --acl user:alice:r-x + --entry group:secret-project:rwx \ + --entry user:alice:r-x AUTHORS diff --git a/cdist/conf/type/__acl/parameter/deprecated/acl b/cdist/conf/type/__acl/parameter/deprecated/acl new file mode 100644 index 00000000..94e14159 --- /dev/null +++ b/cdist/conf/type/__acl/parameter/deprecated/acl @@ -0,0 +1 @@ +see manual for details diff --git a/cdist/conf/type/__acl/parameter/optional_multiple b/cdist/conf/type/__acl/parameter/optional_multiple index 95c25d55..c615d507 100644 --- a/cdist/conf/type/__acl/parameter/optional_multiple +++ b/cdist/conf/type/__acl/parameter/optional_multiple @@ -1,3 +1,4 @@ +entry acl user group From 51ba4a49d8ec79968f79563f994489c619f10bac Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Sat, 4 Jan 2020 18:21:23 +0100 Subject: [PATCH 34/38] ++changelog --- docs/changelog | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/changelog b/docs/changelog index 706d76af..a7bcf9b1 100644 --- a/docs/changelog +++ b/docs/changelog @@ -1,6 +1,9 @@ Changelog --------- +next: + * Type __acl: Add --entry parameter to replace --acl, deprecate --acl (Ander Punnar) + 6.4.0: 2020-01-04 * Type __consul_agent: Don't deploy init script on Alpine anymore, it ships with one itself (Nico Schottelius) * Type __install_chroot_umount: Bugfix: type was not using __chroot_umount/manifest (Steven Armstrong) From 11f569959d6e331d4d5052ca73fb5d83bf9df8e7 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Sat, 11 Jan 2020 14:16:33 +0100 Subject: [PATCH 35/38] Fix missing configuration file usage, support -g PreOS code did not use configuration support. This fix adds support for using cdist configuration, which takes into account cdist configuration file, environment variables and command line options, especially conf_dir. It also adds support for -g, --config-file option, for specifying custom configuration file. --- cdist/preos.py | 24 +++++++++++------------- docs/changelog | 1 + docs/src/man1/cdist.rst | 7 ++++++- 3 files changed, 18 insertions(+), 14 deletions(-) diff --git a/cdist/preos.py b/cdist/preos.py index 378071db..491338d2 100644 --- a/cdist/preos.py +++ b/cdist/preos.py @@ -5,8 +5,9 @@ import inspect import argparse import cdist import logging -import re import cdist.argparse +import cdist.configuration +import cdist.exec.util as util _PREOS_CALL = "commandline" @@ -24,16 +25,6 @@ def extend_plugins_path(dirs): _PLUGINS_PATH.append(preos_dir) -cdist_home = cdist.home_dir() -if cdist_home: - extend_plugins_path((cdist_home, )) -x = 'CDIST_PATH' -if x in os.environ: - vals = re.split(r'(? Date: Sat, 11 Jan 2020 15:26:46 +0100 Subject: [PATCH 36/38] Info command: support tilde expansion --- cdist/exec/util.py | 7 +++++++ cdist/info.py | 8 +++----- cdist/preos.py | 7 ++----- docs/changelog | 1 + 4 files changed, 13 insertions(+), 10 deletions(-) diff --git a/cdist/exec/util.py b/cdist/exec/util.py index 5513f01d..9787f431 100644 --- a/cdist/exec/util.py +++ b/cdist/exec/util.py @@ -24,6 +24,7 @@ import os from tempfile import TemporaryFile import cdist +import cdist.configuration # IMPORTANT: @@ -200,3 +201,9 @@ def resolve_conf_dirs(configuration, add_conf_dirs): conf_dirs.extend(add_conf_dirs) conf_dirs = set(conf_dirs) return conf_dirs + + +def resolve_conf_dirs_from_config_and_args(args): + cfg = cdist.configuration.Configuration(args) + configuration = cfg.get_config(section='GLOBAL') + return resolve_conf_dirs(configuration, args.conf_dir) diff --git a/cdist/info.py b/cdist/info.py index 4c1d3560..b896a3d1 100644 --- a/cdist/info.py +++ b/cdist/info.py @@ -53,10 +53,7 @@ class Info(object): @classmethod def commandline(cls, args): - cfg = cdist.configuration.Configuration(args) - configuration = cfg.get_config(section='GLOBAL') - conf_dirs = util.resolve_conf_dirs(configuration, - args.conf_dir) + conf_dirs = util.resolve_conf_dirs_from_config_and_args(args) c = cls(conf_dirs, args) c.run() @@ -170,7 +167,8 @@ class Info(object): def run(self): rv = [] - for conf_path in self.conf_dirs: + for cp in self.conf_dirs: + conf_path = os.path.expanduser(cp) if self.all or self.display_global_explorers: rv.extend((x, 'E', ) for x in self._get_global_explorers( conf_path)) diff --git a/cdist/preos.py b/cdist/preos.py index 491338d2..e353fe3b 100644 --- a/cdist/preos.py +++ b/cdist/preos.py @@ -101,13 +101,10 @@ class PreOS(object): action='store_true', default=False) parser.add_argument('remainder_args', nargs=argparse.REMAINDER) args = parser.parse_args(argv[1:]) - cdist.argparse.handle_loglevel(args) + st.argparse.handle_loglevel(args) log.debug("preos args : {}".format(args)) - cfg = cdist.configuration.Configuration(args) - configuration = cfg.get_config(section='GLOBAL') - conf_dirs = util.resolve_conf_dirs(configuration, - args.conf_dir) + conf_dirs = util.resolve_conf_dirs_from_config_and_args(args) extend_plugins_path(conf_dirs) sys.path.extend(_PLUGINS_PATH) diff --git a/docs/changelog b/docs/changelog index 1b1a909e..526fc320 100644 --- a/docs/changelog +++ b/docs/changelog @@ -4,6 +4,7 @@ Changelog next: * Type __acl: Add --entry parameter to replace --acl, deprecate --acl (Ander Punnar) * Core: preos: Fix missing configuration file usage, support -g, --config-file option (Darko Poljak) + * Core info command: Support tilde expansion of conf directories (Darko Poljak) 6.4.0: 2020-01-04 * Type __consul_agent: Don't deploy init script on Alpine anymore, it ships with one itself (Nico Schottelius) From 3258fc98e15fedbd98e17f7d0b568a38b8da139c Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Sun, 12 Jan 2020 12:19:49 +0100 Subject: [PATCH 37/38] Fix typo --- cdist/preos.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cdist/preos.py b/cdist/preos.py index e353fe3b..bf2a8e60 100644 --- a/cdist/preos.py +++ b/cdist/preos.py @@ -101,7 +101,7 @@ class PreOS(object): action='store_true', default=False) parser.add_argument('remainder_args', nargs=argparse.REMAINDER) args = parser.parse_args(argv[1:]) - st.argparse.handle_loglevel(args) + cdist.argparse.handle_loglevel(args) log.debug("preos args : {}".format(args)) conf_dirs = util.resolve_conf_dirs_from_config_and_args(args) From ef2f4b9a004369cc390f1271721bc22d6a44f402 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Timoth=C3=A9e=20Floure?= Date: Fri, 17 Jan 2020 11:21:28 +0100 Subject: [PATCH 38/38] __postgres_*: fix forgotten edge cases in delimited identifier escape --- cdist/conf/type/__postgres_database/gencode-remote | 8 ++++++-- cdist/conf/type/__postgres_role/gencode-remote | 10 ++++++---- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/cdist/conf/type/__postgres_database/gencode-remote b/cdist/conf/type/__postgres_database/gencode-remote index 9f12d215..47e6b97c 100755 --- a/cdist/conf/type/__postgres_database/gencode-remote +++ b/cdist/conf/type/__postgres_database/gencode-remote @@ -43,10 +43,14 @@ if [ "$state_should" != "$state_is" ]; then if [ -f "$__object/parameter/owner" ]; then owner="-O \"$(cat "$__object/parameter/owner")\"" fi - echo "su - '$postgres_user' -c \"createdb $owner \"$name\"\"" + cat << EOF +su - '$postgres_user' -c "createdb $owner \"$name\"" +EOF ;; absent) - echo "su - '$postgres_user' -c \"dropdb \"$name\"\"" + cat << EOF +su - '$postgres_user' -c "dropdb \"$name\"" +EOF ;; esac fi diff --git a/cdist/conf/type/__postgres_role/gencode-remote b/cdist/conf/type/__postgres_role/gencode-remote index 0b8852f4..977832c9 100755 --- a/cdist/conf/type/__postgres_role/gencode-remote +++ b/cdist/conf/type/__postgres_role/gencode-remote @@ -53,11 +53,13 @@ case "$state_should" in done [ -n "$password" ] && password="PASSWORD '$password'" - - cmd="CREATE ROLE \"$name\" WITH $password $booleans" - echo "su - '$postgres_user' -c \"psql postgres -wc \\\"$cmd\\\"\"" + cat << EOF +su - '$postgres_user' -c "psql postgres -wc 'CREATE ROLE \"$name\" WITH $password $booleans;'" +EOF ;; absent) - echo "su - '$postgres_user' -c \"dropuser \\\"$name\\\"\"" + cat << EOF +su - '$postgres_user' -c "dropuser \"$name\"" +EOF ;; esac