[__git] Silence SC2012 as it does not apply to this case.

Instead of using stat, we can rely on ls.
Thanks to @ssrq for pointing that out.

README

@@ -1,7 +0,0 @@
-cdist
------
-
-cdist is a usable configuration management system.
-
-For the web documentation have a look at https://www.cdi.st/
-or at docs/src for reStructuredText manual.

README.md

@@ -0,0 +1,31 @@
+# cdist
+
+**cdist** is a usable configuration management system.
+
+It adheres to the [**KISS principle**](https://en.wikipedia.org/wiki/KISS_principle)
+and is being used in small up to enterprise grade environments.
+
+For more information have a look at [**homepage**](https://cdi.st)
+or at **``docs/src``** for manual in **reStructuredText** format.
+
+## Contributing
+
+Merge/Pull requests can be made in both
+[upstream **GitLab**](https://code.ungleich.ch/ungleich-public/cdist/merge_requests)
+(managed by [**ungleich**](https://ungleich.ch))
+and [**GitHub** project](https://github.com/ungleich/cdist/pulls).
+
+Issues can be made and other project management activites happen
+[**only in GitLab**](https://code.ungleich.ch/ungleich-public/cdist)
+(needs [**ungleich** account](https://account.ungleich.ch)).
+
+For community-maintained types there is
+[**cdist-contrib** project](https://code.ungleich.ch/ungleich-public/cdist-contrib).
+
+## Participating
+
+IRC: ``#cdist`` @ freenode
+
+Matrix: ``#cdist:ungleich.ch``
+
+Mattermost: https://chat.ungleich.ch/ungleich/channels/cdist

cdist/__init__.py

@@ -26,6 +26,7 @@ import hashlib
 import cdist.log
 import cdist.version
 
+
 VERSION = cdist.version.VERSION
 
 BANNER = """
@@ -48,6 +49,9 @@ REMOTE_EXEC = "ssh -o User=root"
 REMOTE_CMDS_CLEANUP_PATTERN = "ssh -o User=root -O exit -S {}"
 
 
+MIN_SUPPORTED_PYTHON_VERSION = '3.5'
+
+
 class Error(Exception):
     """Base exception class for this project"""
     pass

cdist/argparse.py

@@ -5,6 +5,7 @@ import logging
 import collections
 import functools
 import cdist.configuration
+import cdist.log
 import cdist.preos
 import cdist.info
 
@@ -125,6 +126,14 @@ def get_parsers():
                   'value.'),
             action='count', default=None)
 
+    parser['colored_output'] = argparse.ArgumentParser(add_help=False)
+    parser['colored_output'].add_argument(
+            '--colors', metavar='WHEN',
+            help="Colorize cdist's output based on log level; "
+                 "WHEN is 'always', 'never', or 'auto'.",
+            action='store', dest='colored_output', required=False,
+            choices=cdist.configuration.ColoredOutputOption.CHOICES)
+
     parser['beta'] = argparse.ArgumentParser(add_help=False)
     parser['beta'].add_argument(
            '-b', '--beta',
@@ -197,6 +206,13 @@ def get_parsers():
                  'supported. Without argument CPU count is used by default. '),
            action='store', dest='jobs',
            const=multiprocessing.cpu_count())
+    parser['config_main'].add_argument(
+           '--log-server',
+           action='store_true',
+           help=('Start a log server for sub processes to use. '
+                 'This is mainly useful when running cdist nested '
+                 'from a code-local script. Log server is alwasy '
+                 'implicitly started for \'install\' command.'))
     parser['config_main'].add_argument(
            '-n', '--dry-run',
            help='Do not execute code.', action='store_true')
@@ -283,6 +299,7 @@ def get_parsers():
             'host', nargs='*', help='Host(s) to operate on.')
     parser['config'] = parser['sub'].add_parser(
             'config', parents=[parser['loglevel'], parser['beta'],
+                               parser['colored_output'],
                                parser['common'],
                                parser['config_main'],
                                parser['inventory_common'],
@@ -301,6 +318,7 @@ def get_parsers():
 
     parser['add-host'] = parser['invsub'].add_parser(
             'add-host', parents=[parser['loglevel'], parser['beta'],
+                                 parser['colored_output'],
                                  parser['common'],
                                  parser['inventory_common']])
     parser['add-host'].add_argument(
@@ -315,6 +333,7 @@ def get_parsers():
 
     parser['add-tag'] = parser['invsub'].add_parser(
             'add-tag', parents=[parser['loglevel'], parser['beta'],
+                                parser['colored_output'],
                                 parser['common'],
                                 parser['inventory_common']])
     parser['add-tag'].add_argument(
@@ -346,6 +365,7 @@ def get_parsers():
 
     parser['del-host'] = parser['invsub'].add_parser(
             'del-host', parents=[parser['loglevel'], parser['beta'],
+                                 parser['colored_output'],
                                  parser['common'],
                                  parser['inventory_common']])
     parser['del-host'].add_argument(
@@ -363,6 +383,7 @@ def get_parsers():
 
     parser['del-tag'] = parser['invsub'].add_parser(
             'del-tag', parents=[parser['loglevel'], parser['beta'],
+                                parser['colored_output'],
                                 parser['common'],
                                 parser['inventory_common']])
     parser['del-tag'].add_argument(
@@ -398,6 +419,7 @@ def get_parsers():
 
     parser['list'] = parser['invsub'].add_parser(
             'list', parents=[parser['loglevel'], parser['beta'],
+                             parser['colored_output'],
                              parser['common'],
                              parser['inventory_common']])
     parser['list'].add_argument(
@@ -430,7 +452,7 @@ def get_parsers():
 
     # Shell
     parser['shell'] = parser['sub'].add_parser(
-            'shell', parents=[parser['loglevel']])
+            'shell', parents=[parser['loglevel'], parser['colored_output']])
     parser['shell'].add_argument(
             '-s', '--shell',
             help=('Select shell to use, defaults to current shell. Used shell'
@@ -478,7 +500,12 @@ def handle_loglevel(args):
     if hasattr(args, 'quiet') and args.quiet:
         args.verbose = _verbosity_level_off
 
-    logging.root.setLevel(_verbosity_level[args.verbose])
+    logging.getLogger().setLevel(_verbosity_level[args.verbose])
+
+
+def handle_log_colors(args):
+    if cdist.configuration.ColoredOutputOption.translate(args.colored_output):
+        cdist.log.CdistFormatter.USE_COLORS = True
 
 
 def parse_and_configure(argv, singleton=True):
@@ -492,6 +519,7 @@ def parse_and_configure(argv, singleton=True):
         raise cdist.Error(str(e))
     # Loglevels are handled globally in here
     handle_loglevel(args)
+    handle_log_colors(args)
 
     log = logging.getLogger("cdist")
 

cdist/conf/explorer/cpu_cores

@@ -32,6 +32,10 @@ case "$os" in
         sysctl -n hw.ncpuonline
     ;;
 
+    "freebsd"|"netbsd")
+        sysctl -n hw.ncpu
+    ;;
+
     *)
         if [ -r /proc/cpuinfo ]; then
             cores="$(grep "core id" /proc/cpuinfo | sort | uniq | wc -l)"

cdist/conf/explorer/machine_type

@@ -2,6 +2,7 @@
 #
 # 2014 Daniel Heule  (hda at sfs.biz)
 # 2014 Thomas Oettli (otho at sfs.biz)
+# 2020 Evilham (contact at evilham.com)
 #
 # This file is part of cdist.
 #
@@ -18,9 +19,51 @@
 # You should have received a copy of the GNU General Public License
 # along with cdist. If not, see <http://www.gnu.org/licenses/>.
 #
-#
 
-# FIXME: other system types (not linux ...)
+os=$("$__explorer/os")
+
+vendor_string_to_machine_type() {
+    for vendor in vmware bochs kvm qemu virtualbox bhyve; do
+        if echo "${1}" | grep -q -i "${vendor}"; then
+            if [ "${vendor}" = "bochs" ] || [ "${vendor}" = "qemu" ]; then
+                vendor="kvm"
+            fi
+            echo "virtual_by_${vendor}"
+            exit
+        fi
+    done
+}
+
+case "$os" in
+    "freebsd")
+        # FreeBSD does not have /proc/cpuinfo even when procfs is used.
+        # Instead there is a sysctl kern.vm_guest.
+        # Which is 'none' if physical, else the virtualisation.
+        vm_guest="$(sysctl -n kern.vm_guest 2>/dev/null || true)"
+        if [ -n "${vm_guest}" ]; then
+            if [ "${vm_guest}" = "none" ]; then
+                echo "physical"
+                exit
+            fi
+            echo "virtual_by_${vm_guest}"
+            exit
+        fi
+    ;;
+
+    "openbsd")
+        # OpenBSD can also use the sysctl's: hw.vendor or hw.product.
+        # Note we can be reasonably sure about a machine being virtualised
+        # as long as we can identify the virtualisation technology.
+        # But not so much about it being physical...
+        # Patches are welcome / reach out if you have better ideas.
+        for sysctl in hw.vendor hw.product; do
+            # This exits if we can make a reasonable judgement
+            vendor_string_to_machine_type "$(sysctl -n "${sysctl}")"
+        done
+    ;;
+
+    *)
+        # Defaulting to linux for compatibility with previous cdist behaviour
 
         if [ -d "/proc/vz" ] && [ ! -d "/proc/bc" ]; then
             echo openvz
@@ -35,7 +78,7 @@ fi
 
         if [ -r /proc/cpuinfo ]; then
             # this should only exist on virtual guest machines,
-    # tested on vmware, xen, kvm
+            # tested on vmware, xen, kvm, bhyve
             if grep -q "hypervisor" /proc/cpuinfo; then
                 # this file is aviable in xen guest systems
                 if [ -r /sys/hypervisor/type ]; then
@@ -44,37 +87,23 @@ if [ -r /proc/cpuinfo ]; then
                         exit
                     fi
                 else
-            if [ -r /sys/class/dmi/id/product_name ]; then
+                    for vendor_file in /sys/class/dmi/id/product_name \
-                if grep -q -i 'vmware' /sys/class/dmi/id/product_name; then
+                                       /sys/class/dmi/id/sys_vendor \
-                    echo "virtual_by_vmware"
+                                       /sys/class/dmi/id/chasis_vendor; do
-                    exit
+                        if [ -r ${vendor_file} ]; then
-                elif grep -q -i 'bochs' /sys/class/dmi/id/product_name; then
+                            # This exits if we can make a reasonable judgement
-                    echo "virtual_by_kvm"
+                            vendor_string_to_machine_type "$(cat "${vendor_file}")"
-                    exit 
-                elif grep -q -i 'virtualbox' /sys/class/dmi/id/product_name; then
-                    echo "virtual_by_virtualbox"
-                    exit 
-                fi
-            fi
-
-            if [ -r /sys/class/dmi/id/sys_vendor ]; then
-                if grep -q -i 'qemu' /sys/class/dmi/id/sys_vendor; then
-                    echo "virtual_by_kvm"
-                    exit
-                fi
-            fi
-
-            if [ -r /sys/class/dmi/id/chassis_vendor ]; then
-                if grep -q -i 'qemu' /sys/class/dmi/id/chassis_vendor; then
-                    echo "virtual_by_kvm"
-                    exit
-                fi
                         fi
+                    done
                 fi
                 echo "virtual_by_unknown"
+                exit
             else
                 echo "physical"
+                exit
             fi
-else
+        fi
+    ;;
+esac
+
 echo "unknown"
-fi

cdist/conf/explorer/memory

@@ -29,7 +29,7 @@ case "$os" in
         echo "$(sysctl -n hw.memsize)/1024" | bc
     ;;
 
-    "openbsd")
+    *"bsd")
         echo "$(sysctl -n hw.physmem) / 1048576" | bc
     ;;
 

cdist/conf/explorer/os

@@ -143,6 +143,11 @@ case "$uname_s" in
 esac
 
 if [ -f /etc/os-release ]; then
+   # after sles15, suse don't provide an /etc/SuSE-release anymore, but there is almost no difference between sles and opensuse leap, so call it suse
+   if grep -q ^ID_LIKE=\"suse\" /etc/os-release 2>/dev/null; then
+      echo suse
+      exit 0
+   fi
    # already lowercase, according to:
    # https://www.freedesktop.org/software/systemd/man/os-release.html
    awk -F= '/^ID=/ { if ($2 ~ /^'"'"'(.*)'"'"'$/ || $2 ~ /^"(.*)"$/) { print substr($2, 2, length($2) - 2) } else { print $2 } }' /etc/os-release

cdist/conf/type/__clean_path/explorer/list

@@ -18,7 +18,12 @@
 # along with cdist. If not, see <http://www.gnu.org/licenses/>.
 #
 
+if [ -f "$__object/parameter/path" ]
+then
+    path="$( cat "$__object/parameter/path" )"
+else
     path="/$__object_id"
+fi
 
 [ ! -d "$path" ] && exit 0
 

cdist/conf/type/__clean_path/gencode-remote

@@ -20,7 +20,12 @@
 
 [ ! -s "$__object/explorer/list" ] && exit 0
 
+if [ -f "$__object/parameter/path" ]
+then
+    path="$( cat "$__object/parameter/path" )"
+else
     path="/$__object_id"
+fi
 
 pattern="$( cat "$__object/parameter/pattern" )"
 

cdist/conf/type/__clean_path/man.rst

@@ -10,7 +10,7 @@ DESCRIPTION
 -----------
 Remove files and directories which match the pattern.
 
-Provided path (as __object_id) must be a directory.
+Provided path must be a directory.
 
 Patterns are passed to ``find``'s ``-regex`` - see ``find(1)`` for more details.
 
@@ -29,6 +29,9 @@ pattern
 
 OPTIONAL PARAMETERS
 -------------------
+path
+   Path which will be cleaned. Defaults to ``$__object_id``.
+
 exclude
    Pattern of files which are excluded from removal.
 
@@ -46,6 +49,11 @@ EXAMPLES
         --exclude '.+\(charset\.conf\|security\.conf\)' \
         --onchange 'service apache2 restart'
 
+    __clean_path apache2-conf-enabled \
+        --path /etc/apache2/conf-enabled \
+        --pattern '.+' \
+        --exclude '.+\(charset\.conf\|security\.conf\)' \
+        --onchange 'service apache2 restart'
 
 AUTHORS
 -------

cdist/conf/type/__clean_path/parameter/optional

@@ -1,2 +1,3 @@
 exclude
 onchange
+path

cdist/conf/type/__cron/man.rst

@@ -21,6 +21,11 @@ command
 
 OPTIONAL PARAMETERS
 -------------------
+**NOTE**: All time-related parameters (``--minute``, ``--hour``, ``--day_of_month``
+``--month`` and ``--day_of_week``) defaults to ``*``, which means to execute it
+**always**. If you set ``--hour 0`` to execute the cronjob only at midnight, it
+will execute **every** minute in the first hour of the morning all days.
+
 state
    Either present or absent. Defaults to present.
 minute

cdist/conf/type/__daemontools_service/man.rst

@@ -40,12 +40,6 @@ run-file
 log-run
    Command to run for log consumption. Default: `multilog t ./main`
 
-owner
-   User to chown to.
-
-group
-   User to chgrp to.
-
 servicedir
    Directory to install into. Default: `/service`
 

cdist/conf/type/__daemontools_service/manifest

@@ -9,8 +9,6 @@ servicedir=$(cat "$__object/parameter/servicedir")
 run=$(cat "$__object/parameter/run")
 runfile=$(cat "$__object/parameter/run-file")
 logrun=$(cat "$__object/parameter/log-run")
-owner=$(cat "$__object/parameter/owner")
-group=$(cat "$__object/parameter/group")
 
 svc=$(cat "$__type/explorer/svc")
 
@@ -27,22 +25,14 @@ badusage() {
 [ -z "$run$runfile" ] && badusage
 [ -n "$run" ] && [ -n "$runfile" ] && badusage
 
-flags=""
+__directory "$servicedir/$name/log/main" --parents
-if [ -n "$owner" ]; then
-  flags="$flags --owner $owner"
-fi
-if [ -n "$group" ]; then
-  flags="$flags --group $group"
-fi
-
-__directory "$servicedir/$name/log/main" --parents $flags
 
 echo "$RUN_PREFIX$run" | require="__directory/$servicedir/$name/log/main" __config_file "$servicedir/$name/run" \
 	--onchange "svc -t '$servicedir/$name' 2>/dev/null" \
-	--mode 755 $flags \
+	--mode 755 \
 	--source "${runfile:--}"
 
 echo "$RUN_PREFIX$logrun" | require="__directory/$servicedir/$name/log/main" __config_file "$servicedir/$name/log/run" \
 	--onchange "svc -t '$servicedir/$name/log' 2>/dev/null" \
-	--mode 755 $flags \
+	--mode 755 \
 	--source "-"

cdist/conf/type/__daemontools_service/parameter/optional

@@ -1,6 +1,4 @@
-group
 log-run
-owner
 run
 run-file
 servicedir

cdist/conf/type/__download/explorer/state

@@ -0,0 +1,20 @@
+#!/bin/sh -e
+
+dst="/$__object_id"
+
+# shellcheck disable=SC2059
+cmd="$( printf "$( cat "$__object/parameter/cmd-sum" )" "$dst" )"
+
+sum="$( cat "$__object/parameter/sum" )"
+
+if [ -f "$dst" ]
+then
+    if [ "$( eval "$cmd" )" = "$sum" ]
+    then
+        echo 'present'
+    else
+        echo 'mismatch'
+    fi
+else
+    echo 'absent'
+fi

cdist/conf/type/__download/gencode-local

@@ -0,0 +1,35 @@
+#!/bin/sh -e
+
+state_is="$( cat "$__object/explorer/state" )"
+
+if [ "$state_is" = 'present' ]
+then
+    exit 0
+fi
+
+url="$( cat "$__object/parameter/url" )"
+
+cmd="$( cat "$__object/parameter/cmd-get" )"
+
+tmp="$( mktemp )"
+
+dst="/$__object_id"
+
+printf "$cmd > %s\n" \
+    "$url" \
+    "$tmp"
+
+if echo "$__target_host" | grep -Eq '^[0-9a-fA-F:]+$'
+then
+    target_host="[$__target_host]"
+else
+    target_host="$__target_host"
+fi
+
+printf '%s %s %s:%s\n' \
+    "$__remote_copy" \
+    "$tmp" \
+    "$target_host" \
+    "$dst"
+
+echo "rm -f '$tmp'"

cdist/conf/type/__download/man.rst

@@ -0,0 +1,66 @@
+cdist-type__download(7)
+=======================
+
+NAME
+----
+cdist-type__download - Download file to local storage and copy it to target host
+
+
+DESCRIPTION
+-----------
+You must use persistent storage in target host for destination file
+(``$__object_id``) because it will be used for checksum calculation
+in order to decide if file must be downloaded.
+
+
+REQUIRED PARAMETERS
+-------------------
+url
+   URL from which to download the file.
+
+sum
+   Checksum of downloaded file.
+
+
+OPTIONAL PARAMETERS
+-------------------
+cmd-get
+   Command used for downloading.
+   Default is ``wget -O- '%s'``.
+   Command must output to ``stdout``.
+
+cmd-sum
+   Command used for checksum calculation.
+   Default is ``md5sum '%s' | awk '{print $1}'``.
+   Command output and ``--sum`` parameter must match.
+
+
+EXAMPLES
+--------
+
+.. code-block:: sh
+
+    __directory /opt/cpma
+
+    require='__directory/opt/cpma' \
+        __download /opt/cpma/cnq3.zip \
+            --url https://cdn.playmorepromode.com/files/cnq3/cnq3-1.51.zip \
+            --sum 46da3021ca9eace277115ec9106c5b46
+
+    require='__download/opt/cpma/cnq3.zip' \
+        __unpack /opt/cpma/cnq3.zip \
+            --move-existing-destination \
+            --destination /opt/cpma/server
+
+
+AUTHORS
+-------
+Ander Punnar <ander-at-kvlt-dot-ee>
+
+
+COPYING
+-------
+Copyright \(C) 2020 Ander Punnar. You can redistribute it
+and/or modify it under the terms of the GNU General Public License as
+published by the Free Software Foundation, either version 3 of the
+License, or (at your option) any later version.

cdist/conf/type/__download/parameter/default/cmd-get

@@ -0,0 +1 @@
+wget -O- '%s'

cdist/conf/type/__download/parameter/default/cmd-sum

@@ -0,0 +1 @@
+md5sum '%s' | awk '{print $1}'

cdist/conf/type/__download/parameter/optional

@@ -0,0 +1,2 @@
+cmd-get
+cmd-sum

cdist/conf/type/__download/parameter/required

@@ -0,0 +1,2 @@
+url
+sum

cdist/conf/type/__file/man.rst

@@ -50,13 +50,13 @@ state
       create or modify it
 
 group
-   Group to chgrp to.
+   Group to chgrp to. Defaults to ``root``.
 
 mode
-   Unix permissions, suitable for chmod.
+   Unix permissions, suitable for chmod. Defaults to a very secure ``0600``.
 
 owner
-   User to chown to.
+   User to chown to. Defaults to ``root``.
 
 source
    If supplied, copy this file from the host running cdist to the target.

cdist/conf/type/__git/explorer/branch

@@ -0,0 +1,21 @@
+#!/bin/sh -e
+
+destination="/$__object_id"
+
+state_should="$(cat "$__object/parameter/state")"
+owner="$(cat "$__object/parameter/owner")"
+
+# If the user did not provide an owner, cdist defaults to root.
+git_user="${owner:-root}"
+
+# Only do something if we are not removing the repo and it is not the first run
+if [ "$state_should" = "present" ] && [ -d "$destination/.git" ]; then
+    # Whenever possible run git as non-root, see history of CVEs.
+    branch="$(su -m "$git_user" -c "git -C '$destination' rev-parse --abbrev-ref HEAD")"
+    if [ "$branch" != "HEAD" ]; then
+        echo "$branch"
+    else
+        # We are using tags
+        su -m "$git_user" -c "git -C '$destination' describe --always --tags --abbrev=0"
+    fi
+fi

cdist/conf/type/__git/explorer/group

@@ -2,4 +2,6 @@
 
 destination="/$__object_id/.git"
 
-stat --print "%G" "${destination}" 2>/dev/null || exit 0
+# See: cdist/conf/type/__file/explorer/stat
+# shellcheck disable=SC2012
+ls -ld "$destination" | awk '{ print $4 }'

cdist/conf/type/__git/explorer/needs-update

@@ -0,0 +1,30 @@
+#!/bin/sh -e
+
+if [ -f "$__object/parameter/no-updates" ]; then
+    # User requested explicitly not to have updates
+    exit
+fi
+
+destination="/$__object_id"
+
+state_should="$(cat "$__object/parameter/state")"
+branch_should="$(cat "$__object/parameter/branch")"
+owner="$(cat "$__object/parameter/owner")"
+
+# If the user did not provide an owner, cdist defaults to root.
+git_user="${owner:-root}"
+
+# Only do something if we are not removing the repo and it is not the first run
+if [ "$state_should" = "present" ] && [ -d "$destination/.git" ]; then
+    # First fetch the remote
+    # Whenever possible run git as non-root, see history of CVEs.
+    su -m "$git_user" -c "git -C '$destination' fetch --quiet"
+    head="$(su -m "$git_user" -c "git -C '$destination' rev-parse HEAD")"
+    # Try first to get the latest commit in the remote current branch,
+    # if it fails try to get the commit for the expected tag name
+    upstream="$(su -m "$git_user" -c "git -C '$destination' rev-parse '@{u}'" 2>/dev/null ||
+        su -m "$git_user" -c "git -C '$destination' rev-parse '${branch_should}^{}'")"
+    if [ "${head}" != "$upstream" ]; then
+        echo "YES"
+    fi
+fi

cdist/conf/type/__git/explorer/owner

@@ -2,4 +2,6 @@
 
 destination="/$__object_id/.git"
 
-stat --print "%U" "${destination}" 2>/dev/null || exit 0
+# See: cdist/conf/type/__file/explorer/stat
+# shellcheck disable=SC2012
+ls -ld "$destination" | awk '{ print $3 }'

cdist/conf/type/__git/gencode-remote

@@ -19,13 +19,19 @@
 #
 #
 
+branch_is=$(cat "$__object/explorer/branch")
 state_is=$(cat "$__object/explorer/state")
 owner_is=$(cat "$__object/explorer/owner")
 group_is=$(cat "$__object/explorer/group")
+needs_update=$(cat "$__object/explorer/needs-update")
+
+if [ -f "$__object/parameter/no-updates" ]; then
+	no_updates="YES"
+fi
 
 state_should=$(cat "$__object/parameter/state")
 
-branch=$(cat "$__object/parameter/branch")
+branch_should=$(cat "$__object/parameter/branch")
 
 source=$(cat "$__object/parameter/source")
 
@@ -38,27 +44,70 @@ mode=$(cat "$__object/parameter/mode")
 [ -f "$__object/parameter/recursive" ] && recursive='--recurse-submodules' || recursive=''
 [ -f "$__object/parameter/shallow" ] && shallow='--depth 1 --shallow-submodules' || shallow=''
 
+if { [ -n "$owner" ] && [ "$owner_is" != "$owner" ]; } || \
+   { [ -n "$group" ] && [ "$group_is" != "$group" ]; }; then
+    needs_chown="YES"
+fi
+
 [ "$state_should" = "$state_is" ] \
- && [ "$owner" = "$owner_is" ] \
+ && [ -z "$needs_chown" ] \
- && [ "$group" = "$group_is" ] \
  && [ -n "$mode" ] && exit 0
 
+# Whenever possible run git as non-root, see history of CVEs.
+if [ -n "$owner" ]; then
+    git_user="$owner"
+else
+    git_user="root"
+fi
+
 case $state_should in
     present)
         if [ "$state_should" != "$state_is" ]; then
-            echo git clone --quiet "$recursive" "$shallow" --branch "$branch" "$source" "$destination"
+            if [ "$git_user" != "root" ]; then
+                # If we execute git as non-root, it is not obvious that we'll be able
+                # to create the output directory, so we have to ensure that.
+                cat << EOF
+if [ ! -d '$destination' ]; then
+    mkdir '$destination'
 fi
-        if { [ -n "$owner" ] && [ "$owner_is" != "$owner" ]; } || \
+chown '${owner}:${group}' '$destination'
-           { [ -n "$group" ] && [ "$group_is" != "$group" ]; }; then
+EOF
+            fi
+            # Actually clone the repository
+            echo "su -m '$git_user' -c \"git clone --quiet $recursive $shallow --branch '$branch_should' '$source' '$destination'\""
+            echo create >> "$__messages_out"
+            repo_changed="YES"
+        elif [ "$branch_should" != "$branch_is" ]; then
+            # User has changed tag / branch, let's update that
+            echo "su -m '$git_user' -c \"git checkout --quiet '$branch_should'\""
+            echo "change $branch_is -> $branch_should" >> "$__messages_out"
+            repo_changed="YES"
+        elif [ -z "$no_updates" ] && [ -n "$needs_update" ]; then
+            # The remote has newer information than our repository.
+            # Fetch was done in the explorer, here we can just pull
+            echo "su -m '$git_user' -c \"git -C '$destination' pull --quiet\""
+            echo update >> "$__messages_out"
+            repo_changed="YES"
+        fi
+        if [ -n "$needs_chown" ]; then
             echo chown -R "${owner}:${group}" "$destination"
+            echo "chown -R '${owner}:${group}'" >> "$__messages_out"
+            repo_changed="YES"
+        fi
+        if [ -f "$__object/parameter/mode-recursive" ]; then
+            mode_recursive="-R"
         fi
         if [ -n "$mode" ]; then
-            echo chmod -R "$mode" "$destination"
+            echo chmod "${mode_recursive}" "$mode" "$destination"
         fi
     ;;
 
     absent)
-        # Handled in manifest
+        # Handled in manifest, except for the change triggers
+        if [ "$state_should" != "$state_is" ]; then
+            echo remove >> "$__messages_out"
+            repo_changed="YES"
+        fi
     ;;
 
     *)
@@ -66,3 +115,7 @@ case $state_should in
         exit 1
     ;;
 esac
+
+if [ -n "$repo_changed" ]; then
+    cat "$__object/parameter/onchange"
+fi

cdist/conf/type/__git/man.rst

@@ -3,12 +3,13 @@ cdist-type__git(7)
 
 NAME
 ----
-cdist-type__git -  Get and or keep git repositories up-to-date
+cdist-type__git -  Get and keep git repositories up-to-date
 
 
 DESCRIPTION
 -----------
-This cdist type allows you to clone git repositories
+This cdist type allows you to clone git repositories, track specific branches
+or tags and keep up to date at your own pace.
 
 
 REQUIRED PARAMETERS
@@ -23,7 +24,7 @@ state
     Either "present" or "absent", defaults to "present"
 
 branch
-    Create this branch by checking out the remote branch of this name
+    Check out this branch or tag to the target directory.
     Default branch is "master"
 
 group
@@ -31,16 +32,45 @@ group
 
 mode
    Unix permissions, suitable for chmod.
+   By default it only applies to the top-level directory.
+   See the mode-recursive parameter as well.
+
+onchange
+   The code to run if the repository is first-cloned, changes or is removed.
 
 owner
    User to chown to.
 
+BOOLEAN PARAMETERS
+------------------
+mode-recursive
+   If present and --mode is passed, the given permissions will be applied
+   recursively to the working directory.
+
+no-updates
+   If present the repository will not be updated after first check out.
+   Notice that this does not affect behaviour if you change the target
+   tag / branch, in which case the type will still ensure the specified one
+   is checked out.
+
 recursive
    Passes the --recurse-submodules flag to git when cloning the repository.
 
 shallow
    Sets --depth=1 and --shallow-submodules for cloning repositories with big history.
 
+MESSAGES
+--------
+change <old_branch> -> <new_branch>
+    The directory tracks old_branch but should track new_branch, it will be changed
+chown -R <owner>:<group>
+    Changed ownership
+create
+    Freshly created the directory with the repository clone
+remove
+    The directory with the repository exists, but state is absent, it will be removed
+update
+    The repository tracks a branch that has been updated on the remote
 
 EXAMPLES
 --------
@@ -52,15 +82,21 @@ EXAMPLES
     # Checkout cdist, stay on branch 2.1
     __git /home/nico/cdist --source git@code.ungleich.ch:ungleich-public/cdist.git --branch 2.1
 
+    # If at a later stage you decide to keep up with the development version,
+    # cdist will change the branch and keep you up to date with:
+    __git /home/nico/cdist --source git@code.ungleich.ch:ungleich-public/cdist.git --branch master
+
 
 AUTHORS
 -------
 Nico Schottelius <nico-cdist--@--schottelius.org>
+Evilham <cvs--@--evilham.com>
 
 
 COPYING
 -------
-Copyright \(C) 2012 Nico Schottelius. You can redistribute it
+Copyright \(C) 2012 Nico Schottelius. 2020 Evilham.
+You can redistribute it
 and/or modify it under the terms of the GNU General Public License as
 published by the Free Software Foundation, either version 3 of the
 License, or (at your option) any later version.

cdist/conf/type/__git/manifest

@@ -32,6 +32,7 @@ mode="$(cat "$__object/parameter/mode")"
 
 case "$state_should" in
     present)
+        # Handled in gencode-remote
         :
     ;;
 

cdist/conf/type/__git/parameter/boolean

@@ -1,2 +1,4 @@
+mode-recursive
+no-updates
 recursive
 shallow

cdist/conf/type/__git/parameter/optional

@@ -1,5 +1,6 @@
 state
 branch
 group
+onchange
 owner
 mode

cdist/conf/type/__group/gencode-remote

@@ -88,7 +88,7 @@ if [ "$state" = "present" ]; then
          fi
       done
       if [ "$os" = "freebsd" ]; then
-         echo pw groupadd "$@" "$name"
+         echo pw groupadd "$name" "$@"
       else
          echo groupadd "$@" "$name"
       fi

cdist/conf/type/__iocage_clone/gencode-remote

@@ -1,152 +0,0 @@
-#!/bin/sh
-
-state="$(cat $__object/parameter/state)"
-template="$(cat $__object/parameter/template)"
-ip4_addr="$(cat $__object/parameter/bridge)|$(cat $__object/parameter/ip)"
-interfaces="none:none"
-defaultrouter="none"
-vnet="off"
-jail_zfs_dataset="$(cat $__object/parameter/jail_zfs_dataset)"
-devfs_ruleset="$(cat $__object/parameter/devfs_ruleset)"
-allow_socket_af="$(cat $__object/parameter/allow_socket_af)"
-mount_procfs="$(cat $__object/parameter/mount_procfs)"
-mount_linprocfs="$(cat $__object/parameter/mount_linprocfs)"
-
-if [ "X$state" = "Xabsent" ]; then
-  cat <<EOF
-    iocage stop $__object_id || true
-    iocage destroy -f $__object_id || true
-    rm -f /iocage/jails/$__object_id
-EOF
-else 
-  cat <<EOF
-get_property_zfs () {
-  zfs get -H -o value \$1 "\$2"
-}
-
-get_property_iocage () {
-  get_property_zfs "org.freebsd.iocage:\$1" "/iocage/jails/\$2"
-}
-
-create_new=0
-if [ ! -d /iocage/jails/"$__object_id" ]; then
-  echo "Jail $__object_id does not exist, going to create."
-  create_new=1
-else
-  base=\$(get_property_zfs origin "/iocage/jails/$__object_id")
-  current_template=\$(get_property_zfs org.freebsd.iocage:tag "\$base")
-  if [ "X\$current_template" != "X$template" ]; then
-    echo "Jail $__object_id has base \$current_template, which is not $template. " >&2
-    create_new=1
-  fi
-fi
-
-if [ \$create_new -eq 0 ]; then
-  if [ "off" == "\$(get_property_iocage jail_zfs "$__object_id")" ]; then
-    current_jail_zfs_dataset=""
-  else
-    current_jail_zfs_dataset="\$(get_property_iocage jail_zfs_dataset "$__object_id")"
-  fi
-fi
-
-configure=0
-if [ \$create_new -eq 1 ]; then
-  configure=1
-elif [ "X$vnet" != "X\$(get_property_iocage vnet "$__object_id")" ]; then
-  configure=1
-elif [ "X$ip4_addr" != "X\$(get_property_iocage ip4_addr "$__object_id")" ]; then
-  configure=1
-elif [ "X$interfaces" != "X\$(get_property_iocage interfaces "$__object_id")" ]; then
-  configure=1
-elif [ "X$defaultrouter" != "X\$(get_property_iocage defaultrouter "$__object_id")" ]; then
-  configure=1
-elif [ "X$mount_procfs" != "X\$(get_property_iocage mount_procfs "$__object_id")" ]; then
-  configure=1
-elif [ "X$devfs_ruleset" != "X\$(get_property_iocage devfs_ruleset "$__object_id")" ]; then
-  configure=1
-elif [ "X$allow_socket_af" != "X\$(get_property_iocage allow_socket_af "$__object_id")" ]; then
-  configure=1
-elif [ "X$jail_zfs_dataset" != "X\$current_jail_zfs_dataset" ]; then
-  configure=1
-fi
-
-if [ \$create_new -eq 1 ]; then
-  echo "Creating jail $__object_id" >&2
-
-  iocage stop $__object_id || true
-  iocage destroy -f $__object_id || true
-  # Without VNETs, we should not need this.
-  # TODO(riso): Use nicer path
-  # /root/cdist/ioc deconfigure $__object_id
-
-  rm -f /iocage/jails/$__object_id
-
-  iocage clone $template tag=$__object_id
-  iocage set boot=on $__object_id
-  UUID=\$(iocage list | grep " $__object_id " | awk "{ print \\\$2; }")
-  rm -f /iocage/jails/$__object_id
-  ln -s /iocage/jails/\$UUID /iocage/jails/$__object_id
-else
-  UUID=\$(iocage list | grep " $__object_id " | awk "{ print \\\$2; }")
-  echo "Jail $__object_id already exists, UUID=\$UUID" >&2
-fi
-
-ROOT="/iocage/jails/\$UUID/root"
-FSTAB="/iocage/jails/\$UUID/fstab"
-rm -f \$FSTAB.new
-touch \$FSTAB.new
-cat $__object/parameter/mount 2>/dev/null | \\
-while read mount; do
-  src=\$(echo \$mount | awk -F: "{ print \\\$1; }")
-  dst_rel=\$(echo \$mount | awk -F: "{ print \\\$2; }")
-  dst="/iocage/jails/\$UUID/root/\$dst_rel"
-  mkdir -p "\$dst"
-  echo "\$src	\$dst	nullfs	rw	0	0" >>\$FSTAB.new
-done
-if [ $mount_linprocfs -eq 1 ]; then
-  echo "linproc /iocage/jails/\$UUID/root/compat/linux/proc linprocfs rw 0 0" >>\$FSTAB.new
-fi
-
-fstab_changed=0
-if diff -q \$FSTAB \$FSTAB.new >/dev/null; then
-  # pass
-else
-  configure=1
-  fstab_changed=1
-fi
-
-if [ \$configure -eq 1 ]; then
-  echo "Configuring jail $__object_id." >&2
-  iocage stop $__object_id || true
-
-  iocage set vnet="$vnet" $__object_id
-  iocage set interfaces="$interfaces" $__object_id
-  iocage set hostname="$__object_id" $__object_id
-  iocage set ip4_addr="$ip4_addr" $__object_id
-  iocage set defaultrouter="$defaultrouter" $__object_id
-  iocage set mount_procfs="$mount_procfs" $__object_id
-  iocage set devfs_ruleset="$devfs_ruleset" $__object_id
-  iocage set allow_socket_af="$allow_socket_af" $__object_id
-  if [ -n "$jail_zfs_dataset" ]; then
-    iocage set jail_zfs=on $__object_id
-    iocage set jail_zfs_dataset="$jail_zfs_dataset" $__object_id
-  else
-    iocage set jail_zfs=off $__object_id
-  fi
-
-  if [ \$fstab_changed -eq 1 ]; then
-    umount -afF \$FSTAB || true
-    mv \$FSTAB.new \$FSTAB
-  fi
-
-  iocage start $__object_id || true
-
-  # Iocage creates new mac address, but arp can have an old mac cached.
-  # TODO(riso): Is this true without VNETs?
-  arp -d -a
-else
-  echo "Jail $__object_id is already configured." >&2
-fi
-rm -f \$FSTAB.new
-EOF
-fi

cdist/conf/type/__iocage_clone/manifest

@@ -1 +0,0 @@
-__package iocage

cdist/conf/type/__iocage_clone/parameter/default/allow_socket_af

@@ -1 +0,0 @@
-0

cdist/conf/type/__iocage_clone/parameter/default/bridge

@@ -1 +0,0 @@
-bridge0

cdist/conf/type/__iocage_clone/parameter/default/devfs_ruleset

@@ -1 +0,0 @@
-4

cdist/conf/type/__iocage_clone/parameter/default/mount_linprocfs

@@ -1 +0,0 @@
-0

cdist/conf/type/__iocage_clone/parameter/default/mount_procfs

@@ -1 +0,0 @@
-0

cdist/conf/type/__iocage_clone/parameter/default/net

@@ -1 +0,0 @@
-24

cdist/conf/type/__iocage_clone/parameter/default/state

@@ -1 +0,0 @@
-present

cdist/conf/type/__iocage_clone/parameter/optional

@@ -1,7 +0,0 @@
-state
-bridge
-jail_zfs_dataset
-mount_procfs
-mount_linprocfs
-devfs_ruleset
-allow_socket_af

cdist/conf/type/__iocage_clone/parameter/optional_multiple

@@ -1 +0,0 @@
-mount

cdist/conf/type/__iocage_clone/parameter/required

@@ -1,2 +0,0 @@
-ip
-template

cdist/conf/type/__letsencrypt_acmetiny/gencode-remote

@@ -1,112 +0,0 @@
-#!/bin/sh -e
-
-ACME_TINY_CERT_REQUEST_DIR="/var/acme-tiny/cert-requests"
-ACME_TINY_ACCOUNT_KEY="/var/acme-tiny/account.key"
-ACME_CHALLENGE_DIR="/srv/www/sites/acme/public/.well-known/acme-challenge"
-
-REALM="${__object_id}"
-EXTRA_DOMAINS=""
-if [ -f "${__object}/parameter/extra-domain" ]; then
-  EXTRA_DOMAINS="$(cat "${__object}/parameter/extra-domain")"
-fi
-
-#TODO: support linux too
-REALMS_DIR="/usr/local/etc/pki/realms"
-REALM_DIR="${REALMS_DIR}/${REALM}"
-REALM_CERT="${REALM_DIR}/default.crt"
-REALM_KEY="${REALM_DIR}/default.key"
-REALM_CERT_REQUEST="${ACME_TINY_CERT_REQUEST_DIR}/${REALM}.csr"
-REALM_CERT_REQUEST_CNF="${ACME_TINY_CERT_REQUEST_DIR}/${REALM}.cnf"
-
-CSR_ALT_NAMES=""
-REALM_CERT_REQUEST_CNF_LINE=""
-if [ -n "${EXTRA_DOMAINS}" ]; then
-  CSR_ALT_NAMES="DNS:${REALM}"
-  for domain in ${EXTRA_DOMAINS}; do
-    CSR_ALT_NAMES="${CSR_ALT_NAMES},DNS:${domain}"
-  done
-  # CSR requests are executed always against .new, only after succeeding .new replaces the .cnf
-  REALM_CERT_REQUEST_CNF_LINE="-reqexts SAN -config '${REALM_CERT_REQUEST_CNF}.new'"
-fi
-
-cat << EOF
-if [ ! -d '${REALM_DIR}' ]; then
-  mkdir -p '${REALM_DIR}'
-fi
-if [ ! -f '${REALM_KEY}' ]; then
-  openssl genrsa 4096 > '${REALM_KEY}'
-fi
-
-if [ ! -d '${ACME_TINY_CERT_REQUEST_DIR}' ]; then
-  mkdir '${ACME_TINY_CERT_REQUEST_DIR}'
-fi
-
-FORCE_CSR_REGEN=""
-if [ -n '${CSR_ALT_NAMES}' ]; then
-  # Generate new config
-  cat /etc/ssl/openssl.cnf > '${REALM_CERT_REQUEST_CNF}.new'
-  printf '[SAN]\nsubjectAltName=${CSR_ALT_NAMES}' >> '${REALM_CERT_REQUEST_CNF}.new'
-  # Compare to previous config if necessary
-  if [ -f '${REALM_CERT_REQUEST_CNF}' ]; then
-    CNF_DIFF=\$(diff -q '${REALM_CERT_REQUEST_CNF}' '${REALM_CERT_REQUEST_CNF}.new' || true)
-    if [ -n "\${CNF_DIFF}" ]; then
-      # Options have changed
-      FORCE_CSR_REGEN="YES"
-    else
-      # Since they match, we won't be using this, clean it
-      rm '${REALM_CERT_REQUEST_CNF}.new'
-    fi
-  else
-    # We never used SAN here, CSR regen needed.
-    FORCE_CSR_REGEN="YES"
-  fi
-else
-  # We used SAN at some point, not any more
-  if [ -f '${REALM_CERT_REQUEST_CNF}' ]; then
-    rm '${REALM_CERT_REQUEST_CNF}'
-    FORCE_CSR_REGEN="YES"
-  fi
-fi
-
-# Create or re-create when params have changed
-if [ ! -f '${REALM_CERT_REQUEST}' -o -n "\${FORCE_CSR_REGEN}" ]; then
-  openssl req -new -sha256 -key '${REALM_KEY}' -subj '/CN=${REALM}' -out '${REALM_CERT_REQUEST}' ${REALM_CERT_REQUEST_CNF_LINE}
-fi
-
-# Check if cert exists, and if so whether or not it's older than a month
-if [ -f '${REALM_CERT}' ]; then
-  MODIFIED_IN_30d="\$(find '${REALM_CERT}' -mtime -30d)"
-  if [ -z "\${MODIFIED_IN_30d}" ]; then
-    # Cert is over a month old, it's fine to regenerate
-    FORCE_CRT_REGEN="YES"
-  fi
-else
-  # This cert doesn't exist
-  FORCE_CRT_REGEN="YES"
-fi
-
-
-# Only request certificate when needed
-# TODO: support linux too
-if [ -n "\${FORCE_CSR_REGEN}" -o -n "\${FORCE_CRT_REGEN}" ]; then
-  doas -u acme-tiny -- acme_tiny \
-    --account '${ACME_TINY_ACCOUNT_KEY}' \
-    --csr '${REALM_CERT_REQUEST}' \
-    --acme-dir '${ACME_CHALLENGE_DIR}' > '${REALM_CERT}.new'
-
-  if [ -s '${REALM_CERT}.new' ]; then
-    mv '${REALM_CERT}.new' '${REALM_CERT}'
-  else
-    echo "Failed to generate cert for realm '${REALM}'."
-    exit 1
-  fi
-fi
-
-cat "${REALM_CERT}" "${REALMS_DIR}/chain.pem" > ${REALM_DIR}/fullchain.pem
-
-if [ -n '${REALM_CERT_REQUEST_CNF_LINE}' -a -f '${REALM_CERT_REQUEST_CNF}.new' ]; then
-  # CSR and cert generation succeded with a new config, put new config in-place.
-  # This is the last thing we do, so we try again next time if sth fails.
-  mv '${REALM_CERT_REQUEST_CNF}.new' '${REALM_CERT_REQUEST_CNF}'
-fi
-EOF

cdist/conf/type/__letsencrypt_acmetiny/manifest

@@ -1 +0,0 @@
-#__letsencrypt_acmetiny_base

cdist/conf/type/__letsencrypt_acmetiny/parameter/optional_multiple

@@ -1 +0,0 @@
-extra-domain

cdist/conf/type/__letsencrypt_acmetiny_base/gencode-remote

@@ -1,12 +0,0 @@
-#!/bin/sh -e
-
-ACME_HOME="/var/acme-tiny"
-ACME_ACCOUNT_KEY="${ACME_HOME}/account.key"
-
-cat << EOF
-if [ ! -f '${ACME_ACCOUNT_KEY}' ]; then
-  openssl genrsa 4096 > '${ACME_ACCOUNT_KEY}'
-  chown acme-tiny:acme-tiny '${ACME_ACCOUNT_KEY}'
-  chmod 640 '${ACME_ACCOUNT_KEY}'
-fi
-EOF

cdist/conf/type/__letsencrypt_acmetiny_base/manifest

@@ -1,227 +0,0 @@
-# Arguments
-ACME_DOMAIN="$(cat "${__object}/parameter/acme_domain" || true)"
-
-if [ -z "${ACME_DOMAIN}" ]; then
-  ACME_DOMAIN="${__target_host}"
-fi
-
-
-# Install needed stuffz
-
-## TODO: consider not depending on nginx? It is... practical though.
-## TODO: Maybe just move this out to a sepecial type?
-__package "nginx"
-
-NGINX_ETC="/usr/local/etc/nginx"
-
-# Setup the acme-challenge snippet
-require="__package/nginx" __directory "${NGINX_ETC}/snippets" --state present
-require="__directory${NGINX_ETC}/snippets" __file "${NGINX_ETC}/snippets/acme-challenge.conf" \
-  --mode 644 \
-  --source - << EOF
-# This file is managed remotely, all changes will be lost
-
-# This was heavily inspired by debops.org.
-
-# Automatic Certificate Management Environment (ACME) support.
-# https://tools.ietf.org/html/draft-ietf-acme-acme-01
-# https://en.wikipedia.org/wiki/Automated_Certificate_Management_Environment
-
-
-# Return the ACME challenge present in the server public root.
-# If not found, switch to global web server root.
-location ^~ /.well-known/acme-challenge/ {
-        default_type "text/plain";
-        try_files \$uri @well-known-acme-challenge;
-}
-
-# Return the ACME challenge present in the global server public root.
-# If not present, redirect request to a specified domain.
-location @well-known-acme-challenge {
-        root /srv/www/sites/acme/public;
-        default_type "text/plain";
-        try_files \$uri @redirect-acme-challenge;
-}
-
-# Redirect the ACME challenge to a different host. If a redirect loop is
-# detected, return 404.
-location @redirect-acme-challenge {
-        if (\$arg_redirect) {
-                return 404;
-        }
-        return 307 \$scheme://${ACME_DOMAIN}\$request_uri?redirect=yes;
-}
-
-# Return 404 if ACME challenge well known path is accessed directly.
-location = /.well-known/acme-challenge/ {
-    return 404;
-}
-EOF
-
-require="__package/nginx" __directory "${NGINX_ETC}/sites-enabled" --state present
-require="__directory${NGINX_ETC}/sites-enabled" __file "${NGINX_ETC}/nginx.conf" \
-  --mode 644 \
-  --source - << EOF
-# This file is managed remotely, all changes will be lost
-
-worker_processes  1;
-
-# This default error log path is compiled-in to make sure configuration parsing
-# errors are logged somewhere, especially during unattended boot when stderr
-# isn't normally logged anywhere. This path will be touched on every nginx
-# start regardless of error log location configured here. See
-# https://trac.nginx.org/nginx/ticket/147 for more info. 
-#
-#error_log  /var/log/nginx/error.log;
-#
-
-#pid        logs/nginx.pid;
-
-
-events {
-        worker_connections  1024;
-}
-
-
-http {
-
-        include       mime.types;
-        default_type  application/octet-stream;
-
-        server_tokens off;
-
-        ssl_session_cache shared:SSL:10m;
-        ssl_session_timeout 5m;
-        sendfile on;
-        tcp_nopush on;
-        tcp_nodelay on;
-        types_hash_max_size 2048;
-        gzip on;
-        gzip_disable "msie6";
-        gzip_comp_level    5;
-        gzip_min_length    256;
-        gzip_proxied       any;
-        gzip_vary          on;
-        gzip_types
-          application/atom+xml
-          application/javascript
-          application/json
-          application/ld+json
-          application/manifest+json
-          application/rss+xml
-          application/vnd.geo+json
-          application/vnd.ms-fontobject
-          application/x-font-ttf
-          application/x-web-app-manifest+json
-          application/xhtml+xml
-          application/xml
-          font/opentype
-          image/bmp
-          image/svg+xml
-          image/x-icon
-          text/cache-manifest
-          text/css
-          text/plain
-          text/vcard
-          text/vnd.rim.location.xloc
-          text/vtt
-          text/x-component
-          text/x-cross-domain-policy;
-
-        # Logging
-        access_log /var/log/nginx/access.log;
-        error_log  /var/log/nginx/error.log;
-
-        #add_header X-Clacks-Overhead "GNU Terry Pratchett";
-
-        # Virtual Hosts Configs
-        include ${NGINX_ETC}/sites-enabled/*.conf;
-}
-EOF
-
-require="__directory${NGINX_ETC}/sites-enabled" __file "${NGINX_ETC}/sites-enabled/welcome.conf" \
-  --mode 644 \
-  --source - << EOF
-# This file is managed remotely, all changes will be lost
-
-# nginx server configuration for:
-#    - https://welcome/
-
-server {
-
-        listen [::]:80;
-
-        server_name welcome;
-
-        root /srv/www/sites/welcome/public;
-
-        include snippets/acme-challenge.conf;
-
-        location / {
-                return 301 https://\$host\$request_uri;
-        }
-}
-EOF
-
-## TODO: this is kinda bad, don't restart every time.
-##       Otherwise this isn't idempotent.
-require="__package/nginx" __service nginx --action onerestart
-require="__package/nginx" __start_on_boot nginx
-
-
-__package "acme-tiny"
-
-# Create acme-tiny user and secure home dir
-ACME_TINY_HOME="/var/acme-tiny"
-require="__package/acme-tiny" __user acme-tiny --system --home ${ACME_TINY_HOME} --comment "acme-tiny client"
-require="__user/acme-tiny" __directory "${ACME_TINY_HOME}" --state present --mode 0750 --owner acme-tiny --group acme-tiny
-
-# Create ACME challenge dirs to be served by nginx
-ACME_PUBLIC_DIR="/srv/www/sites/acme/public"
-ACME_WELLKNOWN_DIR="${ACME_PUBLIC_DIR}/.well-known"
-ACME_CHALLENGE_DIR="${ACME_WELLKNOWN_DIR}/acme-challenge"
-__directory "${ACME_PUBLIC_DIR}" \
-  --parents \
-  --state present \
-  --owner acme-tiny --group www \
-  --mode 2750 # TODO: check whether this does require gid?
-require="__directory${ACME_PUBLIC_DIR}" __directory "${ACME_WELLKNOWN_DIR}" \
-  --state present \
-  --owner acme-tiny --group www \
-  --mode 0750
-require="__directory${ACME_WELLKNOWN_DIR}" __directory "${ACME_CHALLENGE_DIR}" \
-  --state present \
-  --owner acme-tiny --group www \
-  --mode 0750
-
-__package doas
-DOAS_CONF="/usr/local/etc/doas.conf"
-require="__package/doas" __file "${DOAS_CONF}" --mode 0640
-require="__file${DOAS_CONF}" __line "${DOAS_CONF}" \
-  --regex 'root as acme-tiny' \
-  --line 'permit nopass root as acme-tiny'
-
-# Setup CA 
-REALMS_DIR="/usr/local/etc/pki/realms"
-__directory "${REALMS_DIR}" \
-  --parents \
-  --state present \
-  --mode 0755
-
-require="__directory${REALMS_DIR}" __file ${REALMS_DIR}/intermediate.pem \
-  --mode 0644 \
-  --source - << EOF
-$(curl -s https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt)
-EOF
-require="__directory${REALMS_DIR}" __file ${REALMS_DIR}/root.pem \
-  --mode 0644 \
-  --source - << EOF
-$(curl -s https://letsencrypt.org/certs/trustid-x3-root.pem.txt)
-EOF
-require="__directory${REALMS_DIR}" __file ${REALMS_DIR}/chain.pem \
-  --mode 0644 \
-  --source - << EOF
-$(curl -s https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt)
-$(curl -s https://letsencrypt.org/certs/trustid-x3-root.pem.txt)
-EOF
-

cdist/conf/type/__letsencrypt_acmetiny_base/parameter/optional

@@ -1 +0,0 @@
-acme_domain

cdist/conf/type/__letsencrypt_cert/manifest

@@ -90,6 +90,9 @@ if [ -z "${certbot_fullpath}" ]; then
 			__package py27-certbot
 
 			certbot_fullpath=/usr/local/bin/certbot
+		;;
+		ubuntu)
+            __package certbot
             ;;
 		*)
 			echo "Unsupported os: $os" >&2

cdist/conf/type/__link/man.rst

@@ -18,7 +18,7 @@ source
    Specifies the link source.
 
 type
-   Specifies the link type: Either hard or symoblic.
+   Specifies the link type: Either hard or symbolic.
 
 
 OPTIONAL PARAMETERS

cdist/conf/type/__motd/gencode-remote

@@ -22,13 +22,6 @@
 os=$(cat "$__global/explorer/os")
 
 case "$os" in
-    debian|ubuntu|devuan)
-
-        # Debian and Ubuntu need to be updated,
-        # as seen in /etc/init.d/bootlogs
-        echo "uname -snrvm > /var/run/motd"
-        echo "cat /etc/motd.tail >> /var/run/motd"
-    ;;
     freebsd)
         # FreeBSD only updates /etc/motd on boot,
 	# as seen in /etc/rc.d/motd

cdist/conf/type/__motd/manifest

@@ -33,10 +33,6 @@ os=$(cat "$__global/explorer/os")
 
 
 case "$os" in
-   debian|ubuntu|devuan)
-      # Debian-based systems use /etc/motd.tail as a template
-      destination=/etc/motd.tail
-   ;;
    freebsd)
       # FreeBSD uses motd.template to prepend system information on boot
       # (this actually only applies starting with version 13,

cdist/conf/type/__pf_rdr/manifest

@@ -1,40 +0,0 @@
-#!/bin/sh -e
-#
-# 2016 Kamila Součková (coding at kamila.is)
-#
-# This file is part of cdist.
-#
-# cdist is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# cdist is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with cdist. If not, see <http://www.gnu.org/licenses/>.
-#
-
-# TODO it would be cool to print a warning if a generated anchor is unused in pf.conf
-
-ANCHORS_DIR=/etc/pf.d
-
-proto="$(cat "${__object}/parameter/proto")"
-from="$(cat "${__object}/parameter/from")"
-to="$(cat "${__object}/parameter/to")"
-state="$(cat "${__object}/parameter/state")"
-
-# This breaks utterly with IPv6
-from="$(echo ${from} | sed 's/:/ port /')"
-to="$(echo ${to} | sed 's/:/ port /')"
-
-anchor_name="$(echo ${__object_id} | cut -d/ -f1)"
-rule="rdr pass log proto ${proto} from any to ${from} -> ${to}"
-
-__directory "${ANCHORS_DIR}" --parents
-
-require="__directory/${ANCHORS_DIR}" \
-__line __pf_rdr/${__object_id} --state ${state} --line "${rule}" --file ${ANCHORS_DIR}/${anchor_name}

cdist/conf/type/__pf_rdr/parameter/default/proto

@@ -1 +0,0 @@
-tcp

cdist/conf/type/__pf_rdr/parameter/default/state

@@ -1 +0,0 @@
-present

cdist/conf/type/__pf_rdr/parameter/optional

@@ -1,2 +0,0 @@
-proto
-state

cdist/conf/type/__pf_rdr/parameter/required

@@ -1,2 +0,0 @@
-from
-to

cdist/conf/type/__pyvenv/gencode-remote

@@ -1,6 +1,7 @@
 #!/bin/sh -e
 #
 # 2016 Darko Poljak (darko.poljak at gmail.com)
+# 2020 Nico Schotetlius (nico.schottelius at ungleich.ch)
 #
 # This file is part of cdist.
 #
@@ -45,7 +46,7 @@ then
     pyvenv=$(cat "$pyvenvparam")
 else
     case "$os" in
-        alpine) # no pyvenv on alpine - I assume others will follow
+        alpine|ubuntu) # no pyvenv on alpine - I assume others will follow
             pyvenv="python3 -m venv"
             ;;
         *)

cdist/conf/type/__pyvenv/man.rst

@@ -9,7 +9,7 @@ cdist-type__pyvenv - Create or remove python virtual environment
 DESCRIPTION
 -----------
 This cdist type allows you to create or remove python virtual
-environment using pyvenv.
+environment using pyvenv on python3 -m venv.
 It assumes pyvenv is already installed. Concrete package depends
 on concrete OS and/or OS version/distribution.
 Ensure this for e.g. in your init manifest as in the following example:
@@ -76,4 +76,3 @@ COPYING
 -------
 Copyright \(C) 2016 Darko Poljak. Free use of this software is
 granted under the terms of the GNU General Public License v3 or later (GPLv3+).
-

cdist/conf/type/__ssh_authorized_key/man.rst

@@ -15,25 +15,27 @@ This type was created to be used by the __ssh_authorized_keys type.
 REQUIRED PARAMETERS
 -------------------
 file
-   the authorized_keys file to which the given key should be added
+   The authorized_keys file where the given key should be managed.
 
 key
-   a string containing the ssh keytype, base 64 encoded key and optional
+   The ssh key which shall be managed in this authorized_keys file.
-   trailing comment which shall be added to the given authorized_keys file.
+   Must be a string containing the ssh keytype, base 64 encoded key and
+   optional trailing comment which shall be added to the given
+   authorized_keys file.
 
 
 OPTIONAL PARAMETERS
 -------------------
 comment
-   explicit comment instead of the one which may be trailing the given key
+   Use this comment instead of the one which may be trailing in the key.
 
 option
-   an option to set for this authorized_key entry.
+   An option to set for this authorized_key entry.
    Can be specified multiple times.
    See sshd(8) for available options.
 
 state
-   if the given keys should be 'present' or 'absent', defaults to 'present'.
+   If the managed key should be 'present' or 'absent', defaults to 'present'.
 
 
 MESSAGES
@@ -64,7 +66,7 @@ EXAMPLES
 
 SEE ALSO
 --------
-:strong:`cdist__ssh_authorized_keys`\ (7), :strong:`sshd`\ (8)
+:strong:`cdist-type__ssh_authorized_keys`\ (7), :strong:`sshd`\ (8)
 
 
 AUTHORS

cdist/conf/type/__ssh_authorized_keys/explorer/keys

@@ -0,0 +1,9 @@
+#!/bin/sh -e
+
+# shellcheck disable=SC1090
+file="$( . "$__type_explorer/file" )"
+
+if [ -f "$file" ]
+then
+    cat "$file"
+fi

cdist/conf/type/__ssh_authorized_keys/man.rst

@@ -20,42 +20,48 @@ then left to the user to ensure that the file exists and that ownership and
 permissions work with ssh.
 
 
-REQUIRED PARAMETERS
+REQUIRED MULTIPLE PARAMETERS
--------------------
+----------------------------
 key
-   the ssh key which shall be added to this authorized_keys file.
+   An ssh key which shall be managed in this authorized_keys file.
-   Must be a string and can be specified multiple times.
+   Must be a string containing the ssh keytype, base 64 encoded key and
+   optional trailing comment which shall be added to the given
+   authorized_keys file.
+   Can be specified multiple times.
 
 
 OPTIONAL PARAMETERS
 -------------------
 comment
-   explicit comment instead of the one which may be trailing the given key
+   Use this comment instead of the one which may be trailing in each key.
 
 file
-   an alternative destination file, defaults to ~$owner/.ssh/authorized_keys
+   An alternative destination file, defaults to ~$owner/.ssh/authorized_keys.
 
 option
-   an option to set for all created authorized_key entries.
+   An option to set for all authorized_key entries in the key parameter.
    Can be specified multiple times.
    See sshd(8) for available options.
 
 owner
-   the user owning the authorized_keys file, defaults to object_id.
+   The user owning the authorized_keys file, defaults to object_id.
 
 state
-   if the given keys should be 'present' or 'absent', defaults to 'present'.
+   If the given keys should be 'present' or 'absent', defaults to 'present'.
 
 
 BOOLEAN PARAMETERS
 ------------------
 noparent
-   don't create or change ownership and permissions of the directory containing
+   Don't create or change ownership and permissions of the directory containing
-   the authorized_keys file
+   the authorized_keys file.
 
 nofile
-   don't manage existence, ownership and permissions of the the authorized_keys
+   Don't manage existence, ownership and permissions of the the authorized_keys
-   file
+   file.
+
+remove-unknown
+   Remove undefined keys.
 
 
 EXAMPLES
@@ -67,6 +73,12 @@ EXAMPLES
     __ssh_authorized_keys root \
        --key "$(cat ~/.ssh/id_rsa.pub)"
 
+    # same as above, but make sure your key is only key in
+    # root's authorized_keys file
+    __ssh_authorized_keys root \
+       --key "$(cat ~/.ssh/id_rsa.pub)" \
+       --remove-unknown
+
     # allow key to login as user-name
     __ssh_authorized_keys user-name \
        --key "ssh-rsa AXYZAAB3NzaC1yc2..."

cdist/conf/type/__ssh_authorized_keys/manifest

@@ -55,8 +55,12 @@ _cksum() {
    echo "$1" | cksum | cut -d' ' -f 1
 }
 
+_type_and_key() {
+   echo "$1" | tr ' ' '\n' | awk '/^(ssh|ecdsa)-[^ ]+/ { printf $1" "; getline; printf $1 }'
+}
+
 while read -r key; do
-   type_and_key="$(echo "$key" | tr ' ' '\n' | awk '/^(ssh|ecdsa)-[^ ]+/ { printf $1" "; getline; printf $1 }')"
+   type_and_key="$( _type_and_key "$key" )"
    object_id="$(_cksum "$file")-$(_cksum "$type_and_key")"
    set -- "$object_id"
    set -- "$@" --file "$file"
@@ -72,3 +76,24 @@ while read -r key; do
    # Ensure __ssh_authorized_key does not read stdin
    __ssh_authorized_key "$@" < /dev/null
 done < "$__object/parameter/key"
+
+if [ -f "$__object/parameter/remove-unknown" ] &&
+    [ -s "$__object/explorer/keys" ]
+then
+    while read -r key
+    do
+        type_and_key="$( _type_and_key "$key" )"
+
+        if grep -Fq "$type_and_key" "$__object/parameter/key"
+        then
+            continue
+        fi
+
+        __ssh_authorized_key "remove-$( _cksum "$file$key" )" \
+            --file "$file" \
+            --key "$key" \
+            --state absent \
+                < /dev/null
+    done \
+        < "$__object/explorer/keys"
+fi

cdist/conf/type/__ssh_authorized_keys/parameter/boolean

@@ -1,2 +1,3 @@
 noparent
 nofile
+remove-unknown

cdist/conf/type/__ssh_authorized_keys/parameter/optional

@@ -1,5 +1,4 @@
 comment
 file
-option
 owner
 state

cdist/conf/type/__ssh_authorized_keys/parameter/optional_multiple

@@ -0,0 +1 @@
+option

cdist/conf/type/__tinydns/gencode-remote

@@ -1,7 +0,0 @@
-servicename=$__object_id
-user="$(cat "$__object/parameter/user")"
-server_ip="$(cat "$__object/parameter/server-ip")"
-
-cat<<EOF
-test -d /etc/tinydns/$servicename || tinydns-conf $user $user /etc/tinydns/$servicename $server_ip
-EOF

cdist/conf/type/__tinydns/manifest

@@ -1,8 +0,0 @@
-service_name=$__object_id
-user="$(cat "$__object/parameter/user")"
-
-__package djbdns
-__directory /etc/tinydns --mode 755
-__user $user --system --shell /bin/false
-
-require="__daemontools" __link /service/tinydns-$service_name --type symbolic --source /etc/tinydns/$service_name

cdist/conf/type/__tinydns/parameter/required

@@ -1,2 +0,0 @@
-user
-server-ip

cdist/conf/type/__tinydns_host/gencode-remote

@@ -1,9 +0,0 @@
-servicename=$(echo $__object_id | cut -d/ -f1)
-name=$(echo $__object_id | cut -d/ -f2-)
-ip="$(cat "$__object/parameter/ip")"
-
-cat<<EOF
-cd /etc/tinydns/$servicename/root
-grep '=$name:$ip' data 2>/dev/null || ./add-host $name $ip
-make
-EOF

cdist/conf/type/__tinydns_host/parameter/required

@@ -1 +0,0 @@
-ip

cdist/conf/type/__tinydns_ns/gencode-remote

@@ -1,13 +0,0 @@
-set -x
-
-servicename=$(echo $__object_id | cut -d/ -f1)
-name=$(echo $__object_id | cut -d/ -f2-)
-ip="$(cat "$__object/parameter/ip")"
-
-cat<<EOF
-cd /etc/tinydns/$servicename/root
-grep .$host:$ip data 2>/dev/null || ./add-ns $name $ip
-make
-EOF
-
-set +x

cdist/conf/type/__tinydns_ns/parameter/required

@@ -1 +0,0 @@
-ip

cdist/conf/type/__user/gencode-remote

@@ -135,10 +135,18 @@ elif [ "$state" = "absent" ]; then
     if grep -q "^${name}:" "$__object/explorer/passwd"; then
         #user exists, but state != present, so delete it
         if [ -f "$__object/parameter/remove-home" ]; then
+            if [ "$os" = "freebsd" ]; then
+                printf "pw userdel '%s' -r >/dev/null 2>&1\\n" "${name}"
+            else
                 printf "userdel -r '%s' >/dev/null 2>&1\\n" "${name}"
+            fi
             echo "userdel -r" >> "$__messages_out"
+        else
+            if [ "$os" = "freebsd" ]; then
+                printf "pw userdel '%s' >/dev/null 2>&1\\n" "${name}"
             else
                 printf "userdel '%s' >/dev/null 2>&1\\n" "${name}"
+            fi
             echo "userdel" >> "$__messages_out"
         fi
     fi

cdist/config.py

@@ -29,18 +29,20 @@ import time
 import itertools
 import tempfile
 import multiprocessing
-from cdist.mputil import mp_pool_run, mp_sig_handler
 import atexit
 import shutil
 import socket
+
+from cdist.mputil import mp_pool_run, mp_sig_handler
+from cdist import core, inventory
+from cdist.util.remoteutil import inspect_ssh_mux_opts
+
 import cdist
 import cdist.hostsource
 import cdist.exec.local
 import cdist.exec.remote
 import cdist.util.ipaddr as ipaddr
 import cdist.configuration
-from cdist import core, inventory
-from cdist.util.remoteutil import inspect_ssh_mux_opts
 
 
 def graph_check_cycle(graph):
@@ -195,7 +197,6 @@ class Config(object):
     @classmethod
     def commandline(cls, args):
         """Configure remote system"""
-
         if (args.parallel and args.parallel != 1) or args.jobs:
             if args.timestamp:
                 cdist.log.setupTimestampingParallelLogging()
@@ -203,6 +204,7 @@ class Config(object):
                 cdist.log.setupParallelLogging()
         elif args.timestamp:
             cdist.log.setupTimestampingLogging()
+
         log = logging.getLogger("config")
 
         # No new child process if only one host at a time.
@@ -381,10 +383,16 @@ class Config(object):
            If operating in parallel then return tuple (host, True|False, )
            so that main process knows for which host function was successful.
         """
-
         log = logging.getLogger(host)
 
         try:
+            if args.log_server:
+                # Start a log server so that nested `cdist config` runs
+                # have a place to send their logs to.
+                log_server_socket_dir = tempfile.mkdtemp()
+                cls._register_path_for_removal(log_server_socket_dir)
+                cdist.log.setupLogServer(log_server_socket_dir, log)
+
             remote_exec, remote_copy, cleanup_cmd = cls._resolve_remote_cmds(
                 args)
             log.debug("remote_exec for host \"{}\": {}".format(

cdist/configuration.py

@@ -27,6 +27,7 @@ import cdist.argparse
 import re
 import multiprocessing
 import logging
+import sys
 
 
 class Singleton(type):
@@ -246,9 +247,33 @@ class LogLevelOption(OptionBase):
         return VerbosityOption().translate(val)
 
 
+class ColoredOutputOption(BooleanOption):
+    CHOICES = ('always', 'never', 'auto')
+    DEFAULT = 'auto'
+
+    def get_converter(self):
+        return self.translate
+
+    @staticmethod
+    def translate(val):
+        if isinstance(val, bool):
+            return val
+        elif val == 'always':
+            return True
+        elif val == 'never':
+            return False
+        elif val == 'auto':
+            return 'NO_COLOR' not in os.environ and sys.stdout.isatty()
+
+
+ColoredOutputOption.DEFAULT = ColoredOutputOption.translate(
+    ColoredOutputOption.DEFAULT)
+
+
 _ARG_OPTION_MAPPING = {
     'beta': 'beta',
     'cache_path_pattern': 'cache_path_pattern',
+    'colored_output': 'colored_output',
     'conf_dir': 'conf_dir',
     'manifest': 'init_manifest',
     'out_path': 'out_path',
@@ -294,6 +319,7 @@ class Configuration(metaclass=Singleton):
             'remote_shell': StringOption('remote_shell'),
             'cache_path_pattern': StringOption('cache_path_pattern'),
             'conf_dir': ConfDirOption(),
+            'colored_output': ColoredOutputOption('colored_output'),
             'init_manifest': StringOption('init_manifest'),
             'out_path': StringOption('out_path'),
             'remote_out_path': StringOption('remote_out_path'),
@@ -319,6 +345,7 @@ class Configuration(metaclass=Singleton):
         'CDIST_REMOTE_COPY': 'remote_copy',
         'CDIST_INVENTORY_DIR': 'inventory_dir',
         'CDIST_CACHE_PATH_PATTERN': 'cache_path_pattern',
+        'CDIST_COLORED_OUTPUT': 'colored_output',
         '__cdist_log_level': 'verbosity',
     }
     ENV_VAR_BOOLEAN_OPTIONS = ('CDIST_BETA', )
@@ -327,11 +354,10 @@ class Configuration(metaclass=Singleton):
     }
 
     ARG_OPTION_MAPPING = _ARG_OPTION_MAPPING
-    ADJUST_ARG_OPTION_MAPPING = {
+    ADJUST_ARG_OPTION_MAPPING = {v: k for k, v in _ARG_OPTION_MAPPING.items()}
-       _ARG_OPTION_MAPPING[key]: key for key in _ARG_OPTION_MAPPING
-    }
     REQUIRED_DEFAULT_CONFIG_VALUES = {
         'GLOBAL': {
+            'colored_output': 'auto',
             'verbosity': 0,
         },
     }
@@ -484,8 +510,7 @@ class Configuration(metaclass=Singleton):
             newconfig = self._read_config_file(config_file)
             self._update_config_dict(config, newconfig)
         # command line config file
-        if (self.args and 'config_file' in self.args and
+        if (self.args and self.args.get('config_file', None)):
-                self.args['config_file']):
             newconfig = self._read_config_file(self.args['config_file'])
             self._update_config_dict(config, newconfig)
         # command line

cdist/core/code.py

@@ -116,6 +116,10 @@ class Code(object):
         if dry_run:
             self.env['__cdist_dry_run'] = '1'
 
+        if '__cdist_log_server_socket_export' in os.environ:
+            self.env['__cdist_log_server_socket'] = os.environ[
+                '__cdist_log_server_socket_export']
+
     def _run_gencode(self, cdist_object, which):
         cdist_type = cdist_object.cdist_type
         script = os.path.join(self.local.type_path,

cdist/core/manifest.py

@@ -119,6 +119,8 @@ class Manifest(object):
             '__cdist_log_level': util.log_level_env_var_val(self.log),
             '__cdist_log_level_name': util.log_level_name_env_var_val(
                 self.log),
+            '__cdist_colored_log': str(
+                cdist.log.CdistFormatter.USE_COLORS).lower(),
         }
 
         if dry_run:

cdist/emulator.py

@@ -129,6 +129,9 @@ class Emulator(object):
             # if invalid __cdist_log_level value
             logging.root.setLevel(logging.WARNING)
 
+        colored_log = self.env.get('__cdist_colored_log', 'false')
+        cdist.log.CdistFormatter.USE_COLORS = colored_log == 'true'
+
         self.log = logging.getLogger(self.target_host[0])
 
     def commandline(self):

cdist/exec/remote.py

@@ -280,9 +280,6 @@ class Remote(object):
         assert isinstance(command, (list, tuple)), (
                 "list or tuple argument expected, got: %s" % command)
 
-        if return_output and stdout is not subprocess.PIPE:
-            self.log.debug("return_output is True, ignoring stdout")
-
         close_stdout = False
         close_stderr = False
         if self.save_output_streams:

cdist/flock.py

@@ -22,6 +22,7 @@
 import fcntl
 import logging
 import os
+import cdist.log
 
 
 log = logging.getLogger('cdist-flock')

cdist/install.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 #
-# 2013 Steven Armstrong (steven-cdist at armstrong.cc)
+# 2013-2019 Steven Armstrong (steven-cdist at armstrong.cc)
 #
 # This file is part of cdist.
 #
@@ -22,9 +22,21 @@
 
 import cdist.config
 import cdist.core
+import cdist.log
 
 
 class Install(cdist.config.Config):
+
+    @classmethod
+    def onehost(cls, host, host_tags, host_base_path, host_dir_name, args,
+                parallel, configuration, remove_remote_files_dirs=False):
+        # Always start log server during cdist install so that nested
+        # `cdist config` runs have a place to send their logs to.
+        args.log_server = True
+
+        super().onehost(host, host_tags, host_base_path, host_dir_name, args,
+                        parallel, configuration, remove_remote_files_dirs)
+
     def object_list(self):
         """Short name for object list retrieval.
         In install mode, we only care about install objects.

cdist/log.py

@@ -2,6 +2,7 @@
 # -*- coding: utf-8 -*-
 #
 # 2010-2013 Nico Schottelius (nico-cdist at schottelius.org)
+# 2019-2020 Steven Armstrong
 #
 # This file is part of cdist.
 #
@@ -20,9 +21,16 @@
 #
 #
 
-import logging
-import sys
 import datetime
+import logging
+import logging.handlers
+import sys
+import os
+import asyncio
+import contextlib
+import pickle
+import struct
+import threading
 
 
 # Define additional cdist logging levels.
@@ -50,9 +58,32 @@ def _trace(msg, *args, **kwargs):
 logging.trace = _trace
 
 
-class DefaultLog(logging.Logger):
+class CdistFormatter(logging.Formatter):
+    USE_COLORS = False
+    RESET = '\033[0m'
+    COLOR_MAP = {
+        'ERROR': '\033[0;31m',
+        'WARNING': '\033[0;33m',
+        'INFO': '\033[0;94m',
+        'VERBOSE': '\033[0;34m',
+        'DEBUG': '\033[0;90m',
+        'TRACE': '\033[0;37m',
+    }
 
-    FORMAT = '%(levelname)s: %(message)s'
+    def __init__(self, fmt):
+        super().__init__(fmt=fmt)
+
+    def format(self, record):
+        msg = super().format(record)
+        if self.USE_COLORS:
+            color = self.COLOR_MAP.get(record.levelname)
+            if color:
+                msg = color + msg + self.RESET
+        return msg
+
+
+class DefaultLog(logging.Logger):
+    FORMAT = '%(levelname)s: %(name)s: %(message)s'
 
     class StdoutFilter(logging.Filter):
         def filter(self, rec):
@@ -64,10 +95,15 @@ class DefaultLog(logging.Logger):
 
     def __init__(self, name):
         super().__init__(name)
+        self.propagate = False
 
-        formatter = logging.Formatter(self.FORMAT)
+        if '__cdist_log_server_socket' in os.environ:
-
+            log_server_socket = os.environ['__cdist_log_server_socket']
-        self.addFilter(self)
+            socket_handler = logging.handlers.SocketHandler(log_server_socket,
+                                                            None)
+            self.addHandler(socket_handler)
+        else:
+            formatter = CdistFormatter(self.FORMAT)
 
             stdout_handler = logging.StreamHandler(sys.stdout)
             stdout_handler.addFilter(self.StdoutFilter())
@@ -82,13 +118,6 @@ class DefaultLog(logging.Logger):
             self.addHandler(stdout_handler)
             self.addHandler(stderr_handler)
 
-    def filter(self, record):
-        """Prefix messages with logger name"""
-
-        record.msg = self.name + ": " + str(record.msg)
-
-        return True
-
     def verbose(self, msg, *args, **kwargs):
         self.log(logging.VERBOSE, msg, *args, **kwargs)
 
@@ -110,7 +139,7 @@ class TimestampingLog(DefaultLog):
 
 
 class ParallelLog(DefaultLog):
-    FORMAT = '%(levelname)s: [%(process)d]: %(message)s'
+    FORMAT = '%(levelname)s: [%(process)d]: %(name)s: %(message)s'
 
 
 class TimestampingParallelLog(TimestampingLog, ParallelLog):
@@ -137,4 +166,42 @@ def setupParallelLogging():
     logging.setLoggerClass(ParallelLog)
 
 
+async def handle_log_client(reader, writer):
+    while True:
+        chunk = await reader.read(4)
+        if len(chunk) < 4:
+            return
+
+        data_size = struct.unpack('>L', chunk)[0]
+        data = await reader.read(data_size)
+
+        obj = pickle.loads(data)
+        record = logging.makeLogRecord(obj)
+        logger = logging.getLogger(record.name)
+        logger.handle(record)
+
+
+def run_log_server(server_address):
+    # Get a new loop inside the current thread to run the log server.
+    loop = asyncio.new_event_loop()
+    loop.create_task(asyncio.start_unix_server(handle_log_client,
+                                               server_address))
+    loop.run_forever()
+
+
+def setupLogServer(socket_dir, log=logging.getLogger(__name__)):
+    """Run a asyncio based unix socket log server in a background thread.
+    """
+    log_server_socket = os.path.join(socket_dir, 'log-server')
+    log.debug('Starting logging server on: %s', log_server_socket)
+    os.environ['__cdist_log_server_socket_export'] = log_server_socket
+    with contextlib.suppress(FileNotFoundError):
+        os.remove(log_server_socket)
+    t = threading.Thread(target=run_log_server, args=(log_server_socket,))
+    # Deamonizing the thread means we don't have to care about stoping it.
+    # It will die together with the main process.
+    t.daemon = True
+    t.start()
+
+
 setupDefaultLogging()

cdist/sphinxext/manpage.py

@@ -7,6 +7,7 @@ from docutils.io import FileOutput
 from os import path
 from sphinx.util.nodes import inline_all_toctrees
 from sphinx import addnodes
+from sphinx.util import logging
 
 """
     Extension based on sphinx builtin manpage.
@@ -15,6 +16,9 @@ from sphinx import addnodes
 """
 
 
+logger = logging.getLogger(__name__)
+
+
 class ManualPageTranslator(sphinx.writers.manpage.ManualPageTranslator):
 
     def header(self):
@@ -28,7 +32,7 @@ class ManualPageWriter(sphinx.writers.manpage.ManualPageWriter):
     def __init__(self, builder):
         super().__init__(builder)
         self.translator_class = (
-            self.builder.translator_class or ManualPageTranslator)
+            self.builder.get_translator_class() or ManualPageTranslator)
 
 
 class ManualPageBuilder(sphinx.builders.manpage.ManualPageBuilder):
@@ -43,7 +47,7 @@ class ManualPageBuilder(sphinx.builders.manpage.ManualPageBuilder):
             components=(docwriter,),
             read_config_files=True).get_default_values()
 
-        self.info(bold('writing... '), nonl=True)
+        logger.info(bold('writing... '), nonl=True)
 
         for info in self.config.man_pages:
             docname, name, description, authors, section = info
@@ -54,7 +58,7 @@ class ManualPageBuilder(sphinx.builders.manpage.ManualPageBuilder):
                     authors = []
 
             targetname = '%s.%s' % (name, section)
-            self.info(darkgreen(targetname) + ' { ', nonl=True)
+            logger.info(darkgreen(targetname) + ' { ', nonl=True)
             destination = FileOutput(
                 destination_path=path.join(self.outdir, targetname),
                 encoding='utf-8')
@@ -63,7 +67,7 @@ class ManualPageBuilder(sphinx.builders.manpage.ManualPageBuilder):
             docnames = set()
             largetree = inline_all_toctrees(self, docnames, docname, tree,
                                             darkgreen, [docname])
-            self.info('} ', nonl=True)
+            logger.info('} ', nonl=True)
             self.env.resolve_references(largetree, docname, self)
             # remove pending_xref nodes
             for pendingnode in largetree.traverse(addnodes.pending_xref):
@@ -76,7 +80,7 @@ class ManualPageBuilder(sphinx.builders.manpage.ManualPageBuilder):
             largetree.settings.section = section
 
             docwriter.write(largetree, destination)
-        self.info()
+        logger.info("")
 
 
 def setup(app):

cdist/test/configuration/__init__.py

@@ -28,10 +28,12 @@ import argparse
 from cdist import test
 import cdist.argparse as cap
 import logging
+import sys
 
 my_dir = op.abspath(op.dirname(__file__))
 fixtures = op.join(my_dir, 'fixtures')
 interpolation_config_file = op.join(fixtures, "interpolation-test.cfg")
+colored_output_default = 'auto'
 
 
 def newConfigParser():
@@ -153,6 +155,7 @@ class ConfigurationTestCase(test.CdistTestCase):
             'remote_shell': '/bin/sh',
             'inventory_dir': '',
             'cache_path_pattern': '',
+            'colored_output': colored_output_default,
             'conf_dir': '',
             'init_manifest': '',
             'out_path': '',
@@ -184,6 +187,8 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/bin/sh',
                 'inventory_dir': None,
                 'cache_path_pattern': None,
+                'colored_output': cc.ColoredOutputOption.translate(
+                    colored_output_default),
                 'conf_dir': None,
                 'init_manifest': None,
                 'out_path': None,
@@ -390,6 +395,7 @@ class ConfigurationTestCase(test.CdistTestCase):
         args = argparse.Namespace()
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'verbosity': 0,
             },
         }
@@ -440,6 +446,7 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/bin/sh',
                 'inventory_dir': None,
                 'cache_path_pattern': None,
+                'colored_output': colored_output_default,
                 'conf_dir': None,
                 'init_manifest': None,
                 'out_path': None,
@@ -515,6 +522,7 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/usr/bin/sh',
                 'inventory_dir': '/var/db/cdist/inventory',
                 'cache_path_pattern': None,
+                'colored_output': colored_output_default,
                 'conf_dir': ['/opt/cdist', ],
                 'init_manifest': None,
                 'out_path': None,
@@ -556,6 +564,7 @@ class ConfigurationTestCase(test.CdistTestCase):
             'remote_shell': '/bin/sh',
             'inventory_dir': '',
             'cache_path_pattern': '',
+            'colored_output': colored_output_default,
             'conf_dir': '',
             'init_manifest': '',
             'out_path': '',
@@ -579,6 +588,8 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/usr/bin/sh',
                 'inventory_dir': None,
                 'cache_path_pattern': None,
+                'colored_output': cc.ColoredOutputOption.translate(
+                    colored_output_default),
                 'conf_dir': [
                     '/opt/cdist/conf',
                     '/usr/local/share/cdist/conf',
@@ -623,6 +634,7 @@ class ConfigurationTestCase(test.CdistTestCase):
             'remote_shell': '/bin/sh',
             'inventory_dir': '',
             'cache_path_pattern': '',
+            'colored_output': colored_output_default,
             'conf_dir': '',
             'init_manifest': '',
             'out_path': '',
@@ -645,6 +657,7 @@ class ConfigurationTestCase(test.CdistTestCase):
             'local_shell': '/usr/bin/sh',
             'remote_shell': '/usr/bin/sh',
             'inventory_dir': '/var/db/cdist/inventory',
+            'colored_output': colored_output_default,
             'conf_dir': '/opt/cdist',
             'remote_copy': 'myscp',
             'remote_exec': 'myexec',
@@ -663,6 +676,8 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/usr/bin/sh',
                 'inventory_dir': '/var/db/cdist/inventory',
                 'cache_path_pattern': None,
+                'colored_output': cc.ColoredOutputOption.translate(
+                    colored_output_default),
                 'conf_dir': [
                     '/opt/cdist/conf',
                     '/usr/local/share/cdist/conf',
@@ -694,6 +709,7 @@ class ConfigurationTestCase(test.CdistTestCase):
         }
         expected_config = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'verbosity': 0,
             },
         }
@@ -767,6 +783,7 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/usr/bin/sh',
                 'inventory_dir': '/opt/sysadmin/cdist/inventory',
                 'cache_path_pattern': None,
+                'colored_output': colored_output_default,
                 'conf_dir': [
                     '/opt/cdist/conf',
                     '/usr/local/share/cdist/conf',
@@ -865,6 +882,7 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/usr/bin/sh',
                 'inventory_dir': '/var/db/cdist/inventory',
                 'cache_path_pattern': None,
+                'colored_output': colored_output_default,
                 'conf_dir': [
                     '/opt/conf/cdist',
                 ],
@@ -964,6 +982,7 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/usr/bin/sh',
                 'inventory_dir': '/var/db/cdist/inventory',
                 'cache_path_pattern': None,
+                'colored_output': colored_output_default,
                 'conf_dir': [
                     '/opt/conf/cdist',
                 ],
@@ -1063,6 +1082,7 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/usr/bin/sh',
                 'inventory_dir': '/var/db/cdist/inventory',
                 'cache_path_pattern': None,
+                'colored_output': colored_output_default,
                 'conf_dir': [
                     '/opt/conf/cdist',
                 ],
@@ -1095,6 +1115,7 @@ class ConfigurationTestCase(test.CdistTestCase):
             'beta': True,
             'inventory_dir': '/var/db/cdist/inventory',
             'cache_path_pattern': None,
+            'colored_output': colored_output_default,
             'conf_dir': [
                 '/opt/conf/cdist',
             ],
@@ -1125,6 +1146,7 @@ class ConfigurationTestCase(test.CdistTestCase):
         expected_config_dict = {
             'GLOBAL': {
                 'inventory_dir': None,
+                'colored_output': colored_output_default,
                 'conf_dir': None,
                 'verbosity': 0,
             },
@@ -1148,6 +1170,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'verbosity': cap.VERBOSE_DEBUG,
             },
         }
@@ -1185,6 +1208,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'save_output_streams': True,
                 'verbosity': 0,
             },
@@ -1213,6 +1237,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'save_output_streams': False,
                 'verbosity': 0,
             },
@@ -1241,6 +1266,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'save_output_streams': False,
                 'verbosity': 0,
             },
@@ -1269,6 +1295,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'save_output_streams': False,
                 'verbosity': 0,
             },
@@ -1308,6 +1335,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'timestamp': True,
                 'verbosity': 0,
             },
@@ -1336,6 +1364,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'timestamp': True,
                 'verbosity': 0,
             },
@@ -1364,6 +1393,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'timestamp': False,
                 'verbosity': 0,
             },
@@ -1392,6 +1422,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'timestamp': False,
                 'verbosity': 0,
             },

configuration/cdist.cfg.skeleton

@@ -13,17 +13,28 @@
 #     Specify cache path pattern.
 # cache_path_pattern = %h
 #
+# colored_output
+#     Colorize cdist's output. If enabled, cdist will use different colors for
+#     different log levels.
+#     Recognized values are 'always', 'never', and 'auto'.
+#     If the value is 'auto', colors are enabled if stdout is a TTY unless
+#     the NO_COLOR (https://no-color.org/) environment variable is defined.
+# colored_output = auto
+#
 # conf_dir
 #     List of configuration directories separated with the character conventionally
 #     used by the operating system to separate search path components (as in PATH),
 #     such as ':' for POSIX or ';' for Windows.
 #     If also specified at command line then values from command line are
 #     appended to this value.
+#     Notice that this works in a "last one wins" fashion, so if a type is redefined
+#     in multiple conf_dirs, the last one in which it is defined will be used.
+#     Consider using a unique prefix for your own roles if this can be an issue.
 # conf_dir = <dir1>:<dir2>
 #
 # init_manifest
 #     Specify default initial manifest.
-# init_mainfest = <path-to-init-manifst>
+# init_manifest = <path-to-init-manifest>
 #
 # inventory_dir
 #     Specify inventory directory.

docs/changelog

@@ -2,6 +2,33 @@ Changelog
 ---------
 
 next:
+	* New type: __download (Ander Punnar)
+
+6.6.0: 2020-06-17
+	* Type __ssh_authorized_keys: Add option for removing undefined keys (Ander Punnar)
+	* Core: Support colored log output (Evil Ham)
+	* Core: Tune colored log output and respect NO_COLOR env var (Dennis Camera)
+	* Documentation: Fix failing man pages build with newer sphinx versions (Darko Poljak)
+	* Documentation: Fix trivial grammatical mistakes (Jaak Ristioja)
+	* Explorer os: Fix for sles15 (Daniel Heule)
+	* Type __clean_path: Add --path parameter (Ander Punnar)
+	* Core: Increase minimal supported Python version to 3.5 (Darko Poljak)
+	* Core: Add log server for nested logging (Steven Armstrong)
+
+6.5.6: 2020-05-25
+	* Type __pyvenv: Switch to python3 -m venv for Ubuntu (Nico Schottelius)
+	* Type __letsencrypt_cert: Whitelist Ubuntu (Nico Schottelius)
+	* Types __cron, __file, __link: Improve manpages (Matthias Stecher)
+	* Explorer machine_type: Add support for FreeBSD and OpenBSD, and simplify Linux code (Evil Ham)
+	* Type __ssh_authorized_key, __ssh_authorized_keys: Improve manpages (Evil Ham)
+	* Type __ssh_authorized_keys: Fix bug where --option was not multiple (Evil Ham)
+	* Type __motd: Debian/Ubuntu/Devuan use /etc/motd (Ander Punnar)
+	* Type __group: Fix --gid on FreeBSD (Ander Punnar)
+	* Configuration: Fix typos in cdist.cfg.skeleton (Jaak Ristioja)
+	* Type __user: Fix user deletion on FreeBSD (Ander Punnar)
+	* Core: Fix double log lines (Darko Poljak)
+
+6.5.5: 2020-05-01
 	* Core: Fix XDG_CONFIG_HOME config file location (Joachim Desroches)
 	* Type __postgres_database: Add encoding, lc-collate, lc-ctype, template parameters (Timothée Floure)
 	* Type __motd: Improve documentation and support for FreeBSD (Evil Ham)
@@ -10,6 +37,9 @@ next:
 	* New type: __pf_apply_anchor (Kamila Součková, Evil Ham)
 	* Type __pf_ruleset: Refactor (Kamila Součková, Evil Ham)
 	* Type __pf_apply: Deprecate type (Kamila Součková, Evil Ham)
+	* Configuration: Add notes to cdist.cfg.skeleton (Evil Ham)
+	* Explorers cpu_cores, memory: Improve *BSD support (Evil Ham)
+	* Core: Remove debug logging noise (Evil Ham)
 
 6.5.4: 2020-04-11
 	* Explorer init: Do not grep on non-existent init (Steven Armstrong)

docs/src/cdist-bootstrap.rst

@@ -25,7 +25,7 @@ people, have a look at `cdist best practice <cdist-best-practice.html>`_.
 Setup working directory and branch
 ----------------------------------
 I assume you have a fresh copy of the cdist tree in ~/cdist, cloned from
-one of the official urls (see `cdist quickstart <cdist-quickstart.html>`_ if you don't).
+one of the official URLs (see `cdist quickstart <cdist-quickstart.html>`_ if you don't).
 Entering the command "git branch" should show you "* master", which indicates
 you are on the **master** branch.
 

docs/src/cdist-cache.rst

@@ -59,7 +59,7 @@ typeorder
 
 Object cache overview
 ~~~~~~~~~~~~~~~~~~~~~
-Each object under :strong:`object` directory has its own structurue.
+Each object under :strong:`object` directory has its own structure.
 
 code-local
     code generated from gencode-local, present only if something is

docs/src/cdist-configuration.rst

@@ -21,7 +21,7 @@ precedence. Configuration option value read from source with higher
 precedence will overwrite option value, if exists, read from source with
 lower precedence. That means that command-line option wins them all.
 
-Users can decide on the local conifguration file location. It can be either
+Users can decide on the local configuration file location. It can be either
 ~/.cdist.cfg or $XDG_CONFIG_HOME/cdist/cdist.cfg. Note that, if both exist,
 then ~/.cdist.cfg is used.