[__git] Silence SC2012 as it does not apply to this case.

Instead of using stat, we can rely on ls.
Thanks to @ssrq for pointing that out.

README

@@ -1,7 +0,0 @@
-cdist
------
-
-cdist is a usable configuration management system.
-
-For the web documentation have a look at https://www.cdi.st/
-or at docs/src for reStructuredText manual.

README.md

@@ -0,0 +1,31 @@
+# cdist
+
+**cdist** is a usable configuration management system.
+
+It adheres to the [**KISS principle**](https://en.wikipedia.org/wiki/KISS_principle)
+and is being used in small up to enterprise grade environments.
+
+For more information have a look at [**homepage**](https://cdi.st)
+or at **``docs/src``** for manual in **reStructuredText** format.
+
+## Contributing
+
+Merge/Pull requests can be made in both
+[upstream **GitLab**](https://code.ungleich.ch/ungleich-public/cdist/merge_requests)
+(managed by [**ungleich**](https://ungleich.ch))
+and [**GitHub** project](https://github.com/ungleich/cdist/pulls).
+
+Issues can be made and other project management activites happen
+[**only in GitLab**](https://code.ungleich.ch/ungleich-public/cdist)
+(needs [**ungleich** account](https://account.ungleich.ch)).
+
+For community-maintained types there is
+[**cdist-contrib** project](https://code.ungleich.ch/ungleich-public/cdist-contrib).
+
+## Participating
+
+IRC: ``#cdist`` @ freenode
+
+Matrix: ``#cdist:ungleich.ch``
+
+Mattermost: https://chat.ungleich.ch/ungleich/channels/cdist

cdist/__init__.py

@@ -26,6 +26,7 @@ import hashlib
 import cdist.log
 import cdist.version
 
+
 VERSION = cdist.version.VERSION
 
 BANNER = """
@@ -48,6 +49,9 @@ REMOTE_EXEC = "ssh -o User=root"
 REMOTE_CMDS_CLEANUP_PATTERN = "ssh -o User=root -O exit -S {}"
 
 
+MIN_SUPPORTED_PYTHON_VERSION = '3.5'
+
+
 class Error(Exception):
     """Base exception class for this project"""
     pass

cdist/argparse.py

@@ -5,6 +5,7 @@ import logging
 import collections
 import functools
 import cdist.configuration
+import cdist.log
 import cdist.preos
 import cdist.info
 
@@ -125,6 +126,14 @@ def get_parsers():
                   'value.'),
             action='count', default=None)
 
+    parser['colored_output'] = argparse.ArgumentParser(add_help=False)
+    parser['colored_output'].add_argument(
+            '--colors', metavar='WHEN',
+            help="Colorize cdist's output based on log level; "
+                 "WHEN is 'always', 'never', or 'auto'.",
+            action='store', dest='colored_output', required=False,
+            choices=cdist.configuration.ColoredOutputOption.CHOICES)
+
     parser['beta'] = argparse.ArgumentParser(add_help=False)
     parser['beta'].add_argument(
            '-b', '--beta',
@@ -197,6 +206,13 @@ def get_parsers():
                  'supported. Without argument CPU count is used by default. '),
            action='store', dest='jobs',
            const=multiprocessing.cpu_count())
+    parser['config_main'].add_argument(
+           '--log-server',
+           action='store_true',
+           help=('Start a log server for sub processes to use. '
+                 'This is mainly useful when running cdist nested '
+                 'from a code-local script. Log server is alwasy '
+                 'implicitly started for \'install\' command.'))
     parser['config_main'].add_argument(
            '-n', '--dry-run',
            help='Do not execute code.', action='store_true')
@@ -283,6 +299,7 @@ def get_parsers():
             'host', nargs='*', help='Host(s) to operate on.')
     parser['config'] = parser['sub'].add_parser(
             'config', parents=[parser['loglevel'], parser['beta'],
+                               parser['colored_output'],
                                parser['common'],
                                parser['config_main'],
                                parser['inventory_common'],
@@ -301,6 +318,7 @@ def get_parsers():
 
     parser['add-host'] = parser['invsub'].add_parser(
             'add-host', parents=[parser['loglevel'], parser['beta'],
+                                 parser['colored_output'],
                                  parser['common'],
                                  parser['inventory_common']])
     parser['add-host'].add_argument(
@@ -315,6 +333,7 @@ def get_parsers():
 
     parser['add-tag'] = parser['invsub'].add_parser(
             'add-tag', parents=[parser['loglevel'], parser['beta'],
+                                parser['colored_output'],
                                 parser['common'],
                                 parser['inventory_common']])
     parser['add-tag'].add_argument(
@@ -346,6 +365,7 @@ def get_parsers():
 
     parser['del-host'] = parser['invsub'].add_parser(
             'del-host', parents=[parser['loglevel'], parser['beta'],
+                                 parser['colored_output'],
                                  parser['common'],
                                  parser['inventory_common']])
     parser['del-host'].add_argument(
@@ -363,6 +383,7 @@ def get_parsers():
 
     parser['del-tag'] = parser['invsub'].add_parser(
             'del-tag', parents=[parser['loglevel'], parser['beta'],
+                                parser['colored_output'],
                                 parser['common'],
                                 parser['inventory_common']])
     parser['del-tag'].add_argument(
@@ -398,6 +419,7 @@ def get_parsers():
 
     parser['list'] = parser['invsub'].add_parser(
             'list', parents=[parser['loglevel'], parser['beta'],
+                             parser['colored_output'],
                              parser['common'],
                              parser['inventory_common']])
     parser['list'].add_argument(
@@ -430,7 +452,7 @@ def get_parsers():
 
     # Shell
     parser['shell'] = parser['sub'].add_parser(
-            'shell', parents=[parser['loglevel']])
+            'shell', parents=[parser['loglevel'], parser['colored_output']])
     parser['shell'].add_argument(
             '-s', '--shell',
             help=('Select shell to use, defaults to current shell. Used shell'
@@ -478,7 +500,12 @@ def handle_loglevel(args):
     if hasattr(args, 'quiet') and args.quiet:
         args.verbose = _verbosity_level_off
 
-    logging.root.setLevel(_verbosity_level[args.verbose])
+    logging.getLogger().setLevel(_verbosity_level[args.verbose])
+
+
+def handle_log_colors(args):
+    if cdist.configuration.ColoredOutputOption.translate(args.colored_output):
+        cdist.log.CdistFormatter.USE_COLORS = True
 
 
 def parse_and_configure(argv, singleton=True):
@@ -492,6 +519,7 @@ def parse_and_configure(argv, singleton=True):
         raise cdist.Error(str(e))
     # Loglevels are handled globally in here
     handle_loglevel(args)
+    handle_log_colors(args)
 
     log = logging.getLogger("cdist")
 

cdist/conf/explorer/cpu_cores

@@ -32,6 +32,10 @@ case "$os" in
         sysctl -n hw.ncpuonline
     ;;
 
+    "freebsd"|"netbsd")
+        sysctl -n hw.ncpu
+    ;;
+
     *)
         if [ -r /proc/cpuinfo ]; then
             cores="$(grep "core id" /proc/cpuinfo | sort | uniq | wc -l)"

cdist/conf/explorer/machine_type

@@ -2,6 +2,7 @@
 #
 # 2014 Daniel Heule  (hda at sfs.biz)
 # 2014 Thomas Oettli (otho at sfs.biz)
+# 2020 Evilham (contact at evilham.com)
 #
 # This file is part of cdist.
 #
@@ -18,63 +19,91 @@
 # You should have received a copy of the GNU General Public License
 # along with cdist. If not, see <http://www.gnu.org/licenses/>.
 #
-#
 
-# FIXME: other system types (not linux ...)
+os=$("$__explorer/os")
 
-if [ -d "/proc/vz" ] && [ ! -d "/proc/bc" ]; then
-    echo openvz
-    exit
-fi
-
-if [ -e "/proc/1/environ" ] &&
-    tr '\000' '\n' < "/proc/1/environ" | grep -Eiq '^container='; then
-    echo lxc
-    exit
-fi
-
-if [ -r /proc/cpuinfo ]; then
-    # this should only exist on virtual guest machines,
-    # tested on vmware, xen, kvm
-    if grep -q "hypervisor" /proc/cpuinfo; then
-        # this file is aviable in xen guest systems
-        if [ -r /sys/hypervisor/type ]; then
-            if grep -q -i "xen" /sys/hypervisor/type; then
-                echo virtual_by_xen
-                exit 
+vendor_string_to_machine_type() {
+    for vendor in vmware bochs kvm qemu virtualbox bhyve; do
+        if echo "${1}" | grep -q -i "${vendor}"; then
+            if [ "${vendor}" = "bochs" ] || [ "${vendor}" = "qemu" ]; then
+                vendor="kvm"
             fi
-        else
-            if [ -r /sys/class/dmi/id/product_name ]; then
-                if grep -q -i 'vmware' /sys/class/dmi/id/product_name; then
-                    echo "virtual_by_vmware"
-                    exit
-                elif grep -q -i 'bochs' /sys/class/dmi/id/product_name; then
-                    echo "virtual_by_kvm"
-                    exit 
-                elif grep -q -i 'virtualbox' /sys/class/dmi/id/product_name; then
-                    echo "virtual_by_virtualbox"
-                    exit 
-                fi
-            fi
-
-            if [ -r /sys/class/dmi/id/sys_vendor ]; then
-                if grep -q -i 'qemu' /sys/class/dmi/id/sys_vendor; then
-                    echo "virtual_by_kvm"
-                    exit
-                fi
-            fi
-
-            if [ -r /sys/class/dmi/id/chassis_vendor ]; then
-                if grep -q -i 'qemu' /sys/class/dmi/id/chassis_vendor; then
-                    echo "virtual_by_kvm"
-                    exit
-                fi
-            fi                        
+            echo "virtual_by_${vendor}"
+            exit
         fi
-        echo "virtual_by_unknown"
-    else
-        echo "physical"
-    fi
-else
-    echo "unknown"
-fi
+    done
+}
+
+case "$os" in
+    "freebsd")
+        # FreeBSD does not have /proc/cpuinfo even when procfs is used.
+        # Instead there is a sysctl kern.vm_guest.
+        # Which is 'none' if physical, else the virtualisation.
+        vm_guest="$(sysctl -n kern.vm_guest 2>/dev/null || true)"
+        if [ -n "${vm_guest}" ]; then
+            if [ "${vm_guest}" = "none" ]; then
+                echo "physical"
+                exit
+            fi
+            echo "virtual_by_${vm_guest}"
+            exit
+        fi
+    ;;
+
+    "openbsd")
+        # OpenBSD can also use the sysctl's: hw.vendor or hw.product.
+        # Note we can be reasonably sure about a machine being virtualised
+        # as long as we can identify the virtualisation technology.
+        # But not so much about it being physical...
+        # Patches are welcome / reach out if you have better ideas.
+        for sysctl in hw.vendor hw.product; do
+            # This exits if we can make a reasonable judgement
+            vendor_string_to_machine_type "$(sysctl -n "${sysctl}")"
+        done
+    ;;
+
+    *)
+        # Defaulting to linux for compatibility with previous cdist behaviour
+
+        if [ -d "/proc/vz" ] && [ ! -d "/proc/bc" ]; then
+            echo openvz
+            exit
+        fi
+
+        if [ -e "/proc/1/environ" ] &&
+            tr '\000' '\n' < "/proc/1/environ" | grep -Eiq '^container='; then
+            echo lxc
+            exit
+        fi
+
+        if [ -r /proc/cpuinfo ]; then
+            # this should only exist on virtual guest machines,
+            # tested on vmware, xen, kvm, bhyve
+            if grep -q "hypervisor" /proc/cpuinfo; then
+                # this file is aviable in xen guest systems
+                if [ -r /sys/hypervisor/type ]; then
+                    if grep -q -i "xen" /sys/hypervisor/type; then
+                        echo virtual_by_xen
+                        exit
+                    fi
+                else
+                    for vendor_file in /sys/class/dmi/id/product_name \
+                                       /sys/class/dmi/id/sys_vendor \
+                                       /sys/class/dmi/id/chasis_vendor; do
+                        if [ -r ${vendor_file} ]; then
+                            # This exits if we can make a reasonable judgement
+                            vendor_string_to_machine_type "$(cat "${vendor_file}")"
+                        fi
+                    done
+                fi
+                echo "virtual_by_unknown"
+                exit
+            else
+                echo "physical"
+                exit
+            fi
+        fi
+    ;;
+esac
+
+echo "unknown"

cdist/conf/explorer/memory

@@ -29,7 +29,7 @@ case "$os" in
         echo "$(sysctl -n hw.memsize)/1024" | bc
     ;;
 
-    "openbsd")
+    *"bsd")
         echo "$(sysctl -n hw.physmem) / 1048576" | bc
     ;;
 

cdist/conf/explorer/os

@@ -143,6 +143,11 @@ case "$uname_s" in
 esac
 
 if [ -f /etc/os-release ]; then
+   # after sles15, suse don't provide an /etc/SuSE-release anymore, but there is almost no difference between sles and opensuse leap, so call it suse
+   if grep -q ^ID_LIKE=\"suse\" /etc/os-release 2>/dev/null; then
+      echo suse
+      exit 0
+   fi
    # already lowercase, according to:
    # https://www.freedesktop.org/software/systemd/man/os-release.html
    awk -F= '/^ID=/ { if ($2 ~ /^'"'"'(.*)'"'"'$/ || $2 ~ /^"(.*)"$/) { print substr($2, 2, length($2) - 2) } else { print $2 } }' /etc/os-release

cdist/conf/type/__clean_path/explorer/list

@@ -18,7 +18,12 @@
 # along with cdist. If not, see <http://www.gnu.org/licenses/>.
 #
 
-path="/$__object_id"
+if [ -f "$__object/parameter/path" ]
+then
+    path="$( cat "$__object/parameter/path" )"
+else
+    path="/$__object_id"
+fi
 
 [ ! -d "$path" ] && exit 0
 

cdist/conf/type/__clean_path/gencode-remote

@@ -20,7 +20,12 @@
 
 [ ! -s "$__object/explorer/list" ] && exit 0
 
-path="/$__object_id"
+if [ -f "$__object/parameter/path" ]
+then
+    path="$( cat "$__object/parameter/path" )"
+else
+    path="/$__object_id"
+fi
 
 pattern="$( cat "$__object/parameter/pattern" )"
 

cdist/conf/type/__clean_path/man.rst

@@ -10,7 +10,7 @@ DESCRIPTION
 -----------
 Remove files and directories which match the pattern.
 
-Provided path (as __object_id) must be a directory.
+Provided path must be a directory.
 
 Patterns are passed to ``find``'s ``-regex`` - see ``find(1)`` for more details.
 
@@ -29,6 +29,9 @@ pattern
 
 OPTIONAL PARAMETERS
 -------------------
+path
+   Path which will be cleaned. Defaults to ``$__object_id``.
+
 exclude
    Pattern of files which are excluded from removal.
 
@@ -46,6 +49,11 @@ EXAMPLES
         --exclude '.+\(charset\.conf\|security\.conf\)' \
         --onchange 'service apache2 restart'
 
+    __clean_path apache2-conf-enabled \
+        --path /etc/apache2/conf-enabled \
+        --pattern '.+' \
+        --exclude '.+\(charset\.conf\|security\.conf\)' \
+        --onchange 'service apache2 restart'
 
 AUTHORS
 -------

cdist/conf/type/__clean_path/parameter/optional

@@ -1,2 +1,3 @@
 exclude
 onchange
+path

cdist/conf/type/__cron/man.rst

@@ -21,6 +21,11 @@ command
 
 OPTIONAL PARAMETERS
 -------------------
+**NOTE**: All time-related parameters (``--minute``, ``--hour``, ``--day_of_month``
+``--month`` and ``--day_of_week``) defaults to ``*``, which means to execute it
+**always**. If you set ``--hour 0`` to execute the cronjob only at midnight, it
+will execute **every** minute in the first hour of the morning all days.
+
 state
    Either present or absent. Defaults to present.
 minute

cdist/conf/type/__download/explorer/state

@@ -0,0 +1,20 @@
+#!/bin/sh -e
+
+dst="/$__object_id"
+
+# shellcheck disable=SC2059
+cmd="$( printf "$( cat "$__object/parameter/cmd-sum" )" "$dst" )"
+
+sum="$( cat "$__object/parameter/sum" )"
+
+if [ -f "$dst" ]
+then
+    if [ "$( eval "$cmd" )" = "$sum" ]
+    then
+        echo 'present'
+    else
+        echo 'mismatch'
+    fi
+else
+    echo 'absent'
+fi

cdist/conf/type/__download/gencode-local

@@ -0,0 +1,35 @@
+#!/bin/sh -e
+
+state_is="$( cat "$__object/explorer/state" )"
+
+if [ "$state_is" = 'present' ]
+then
+    exit 0
+fi
+
+url="$( cat "$__object/parameter/url" )"
+
+cmd="$( cat "$__object/parameter/cmd-get" )"
+
+tmp="$( mktemp )"
+
+dst="/$__object_id"
+
+printf "$cmd > %s\n" \
+    "$url" \
+    "$tmp"
+
+if echo "$__target_host" | grep -Eq '^[0-9a-fA-F:]+$'
+then
+    target_host="[$__target_host]"
+else
+    target_host="$__target_host"
+fi
+
+printf '%s %s %s:%s\n' \
+    "$__remote_copy" \
+    "$tmp" \
+    "$target_host" \
+    "$dst"
+
+echo "rm -f '$tmp'"

cdist/conf/type/__download/man.rst

@@ -0,0 +1,66 @@
+cdist-type__download(7)
+=======================
+
+NAME
+----
+cdist-type__download - Download file to local storage and copy it to target host
+
+
+DESCRIPTION
+-----------
+You must use persistent storage in target host for destination file
+(``$__object_id``) because it will be used for checksum calculation
+in order to decide if file must be downloaded.
+
+
+REQUIRED PARAMETERS
+-------------------
+url
+   URL from which to download the file.
+
+sum
+   Checksum of downloaded file.
+
+
+OPTIONAL PARAMETERS
+-------------------
+cmd-get
+   Command used for downloading.
+   Default is ``wget -O- '%s'``.
+   Command must output to ``stdout``.
+
+cmd-sum
+   Command used for checksum calculation.
+   Default is ``md5sum '%s' | awk '{print $1}'``.
+   Command output and ``--sum`` parameter must match.
+
+
+EXAMPLES
+--------
+
+.. code-block:: sh
+
+    __directory /opt/cpma
+
+    require='__directory/opt/cpma' \
+        __download /opt/cpma/cnq3.zip \
+            --url https://cdn.playmorepromode.com/files/cnq3/cnq3-1.51.zip \
+            --sum 46da3021ca9eace277115ec9106c5b46
+
+    require='__download/opt/cpma/cnq3.zip' \
+        __unpack /opt/cpma/cnq3.zip \
+            --move-existing-destination \
+            --destination /opt/cpma/server
+
+
+AUTHORS
+-------
+Ander Punnar <ander-at-kvlt-dot-ee>
+
+
+COPYING
+-------
+Copyright \(C) 2020 Ander Punnar. You can redistribute it
+and/or modify it under the terms of the GNU General Public License as
+published by the Free Software Foundation, either version 3 of the
+License, or (at your option) any later version.

cdist/conf/type/__download/parameter/default/cmd-get

@@ -0,0 +1 @@
+wget -O- '%s'

cdist/conf/type/__download/parameter/default/cmd-sum

@@ -0,0 +1 @@
+md5sum '%s' | awk '{print $1}'

cdist/conf/type/__download/parameter/optional

@@ -0,0 +1,2 @@
+cmd-get
+cmd-sum

cdist/conf/type/__download/parameter/required

@@ -0,0 +1,2 @@
+url
+sum

cdist/conf/type/__file/man.rst

@@ -50,13 +50,13 @@ state
       create or modify it
 
 group
-   Group to chgrp to.
+   Group to chgrp to. Defaults to ``root``.
 
 mode
-   Unix permissions, suitable for chmod.
+   Unix permissions, suitable for chmod. Defaults to a very secure ``0600``.
 
 owner
-   User to chown to.
+   User to chown to. Defaults to ``root``.
 
 source
    If supplied, copy this file from the host running cdist to the target.

cdist/conf/type/__git/explorer/branch

@@ -0,0 +1,21 @@
+#!/bin/sh -e
+
+destination="/$__object_id"
+
+state_should="$(cat "$__object/parameter/state")"
+owner="$(cat "$__object/parameter/owner")"
+
+# If the user did not provide an owner, cdist defaults to root.
+git_user="${owner:-root}"
+
+# Only do something if we are not removing the repo and it is not the first run
+if [ "$state_should" = "present" ] && [ -d "$destination/.git" ]; then
+    # Whenever possible run git as non-root, see history of CVEs.
+    branch="$(su -m "$git_user" -c "git -C '$destination' rev-parse --abbrev-ref HEAD")"
+    if [ "$branch" != "HEAD" ]; then
+        echo "$branch"
+    else
+        # We are using tags
+        su -m "$git_user" -c "git -C '$destination' describe --always --tags --abbrev=0"
+    fi
+fi

cdist/conf/type/__git/explorer/group

@@ -2,4 +2,6 @@
 
 destination="/$__object_id/.git"
 
-stat --print "%G" "${destination}" 2>/dev/null || exit 0
+# See: cdist/conf/type/__file/explorer/stat
+# shellcheck disable=SC2012
+ls -ld "$destination" | awk '{ print $4 }'

cdist/conf/type/__git/explorer/needs-update

@@ -0,0 +1,30 @@
+#!/bin/sh -e
+
+if [ -f "$__object/parameter/no-updates" ]; then
+    # User requested explicitly not to have updates
+    exit
+fi
+
+destination="/$__object_id"
+
+state_should="$(cat "$__object/parameter/state")"
+branch_should="$(cat "$__object/parameter/branch")"
+owner="$(cat "$__object/parameter/owner")"
+
+# If the user did not provide an owner, cdist defaults to root.
+git_user="${owner:-root}"
+
+# Only do something if we are not removing the repo and it is not the first run
+if [ "$state_should" = "present" ] && [ -d "$destination/.git" ]; then
+    # First fetch the remote
+    # Whenever possible run git as non-root, see history of CVEs.
+    su -m "$git_user" -c "git -C '$destination' fetch --quiet"
+    head="$(su -m "$git_user" -c "git -C '$destination' rev-parse HEAD")"
+    # Try first to get the latest commit in the remote current branch,
+    # if it fails try to get the commit for the expected tag name
+    upstream="$(su -m "$git_user" -c "git -C '$destination' rev-parse '@{u}'" 2>/dev/null ||
+        su -m "$git_user" -c "git -C '$destination' rev-parse '${branch_should}^{}'")"
+    if [ "${head}" != "$upstream" ]; then
+        echo "YES"
+    fi
+fi

cdist/conf/type/__git/explorer/owner

@@ -2,4 +2,6 @@
 
 destination="/$__object_id/.git"
 
-stat --print "%U" "${destination}" 2>/dev/null || exit 0
+# See: cdist/conf/type/__file/explorer/stat
+# shellcheck disable=SC2012
+ls -ld "$destination" | awk '{ print $3 }'

cdist/conf/type/__git/gencode-remote

@@ -19,13 +19,19 @@
 #
 #
 
+branch_is=$(cat "$__object/explorer/branch")
 state_is=$(cat "$__object/explorer/state")
 owner_is=$(cat "$__object/explorer/owner")
 group_is=$(cat "$__object/explorer/group")
+needs_update=$(cat "$__object/explorer/needs-update")
+
+if [ -f "$__object/parameter/no-updates" ]; then
+	no_updates="YES"
+fi
 
 state_should=$(cat "$__object/parameter/state")
 
-branch=$(cat "$__object/parameter/branch")
+branch_should=$(cat "$__object/parameter/branch")
 
 source=$(cat "$__object/parameter/source")
 
@@ -38,27 +44,70 @@ mode=$(cat "$__object/parameter/mode")
 [ -f "$__object/parameter/recursive" ] && recursive='--recurse-submodules' || recursive=''
 [ -f "$__object/parameter/shallow" ] && shallow='--depth 1 --shallow-submodules' || shallow=''
 
+if { [ -n "$owner" ] && [ "$owner_is" != "$owner" ]; } || \
+   { [ -n "$group" ] && [ "$group_is" != "$group" ]; }; then
+    needs_chown="YES"
+fi
+
 [ "$state_should" = "$state_is" ] \
- && [ "$owner" = "$owner_is" ] \
- && [ "$group" = "$group_is" ] \
+ && [ -z "$needs_chown" ] \
  && [ -n "$mode" ] && exit 0
 
+# Whenever possible run git as non-root, see history of CVEs.
+if [ -n "$owner" ]; then
+    git_user="$owner"
+else
+    git_user="root"
+fi
+
 case $state_should in
     present)
         if [ "$state_should" != "$state_is" ]; then
-            echo git clone --quiet "$recursive" "$shallow" --branch "$branch" "$source" "$destination"
+            if [ "$git_user" != "root" ]; then
+                # If we execute git as non-root, it is not obvious that we'll be able
+                # to create the output directory, so we have to ensure that.
+                cat << EOF
+if [ ! -d '$destination' ]; then
+    mkdir '$destination'
+fi
+chown '${owner}:${group}' '$destination'
+EOF
+            fi
+            # Actually clone the repository
+            echo "su -m '$git_user' -c \"git clone --quiet $recursive $shallow --branch '$branch_should' '$source' '$destination'\""
+            echo create >> "$__messages_out"
+            repo_changed="YES"
+        elif [ "$branch_should" != "$branch_is" ]; then
+            # User has changed tag / branch, let's update that
+            echo "su -m '$git_user' -c \"git checkout --quiet '$branch_should'\""
+            echo "change $branch_is -> $branch_should" >> "$__messages_out"
+            repo_changed="YES"
+        elif [ -z "$no_updates" ] && [ -n "$needs_update" ]; then
+            # The remote has newer information than our repository.
+            # Fetch was done in the explorer, here we can just pull
+            echo "su -m '$git_user' -c \"git -C '$destination' pull --quiet\""
+            echo update >> "$__messages_out"
+            repo_changed="YES"
         fi
-        if { [ -n "$owner" ] && [ "$owner_is" != "$owner" ]; } || \
-           { [ -n "$group" ] && [ "$group_is" != "$group" ]; }; then
+        if [ -n "$needs_chown" ]; then
             echo chown -R "${owner}:${group}" "$destination"
+            echo "chown -R '${owner}:${group}'" >> "$__messages_out"
+            repo_changed="YES"
+        fi
+        if [ -f "$__object/parameter/mode-recursive" ]; then
+            mode_recursive="-R"
         fi
         if [ -n "$mode" ]; then
-            echo chmod -R "$mode" "$destination"
+            echo chmod "${mode_recursive}" "$mode" "$destination"
         fi
     ;;
 
     absent)
-        # Handled in manifest
+        # Handled in manifest, except for the change triggers
+        if [ "$state_should" != "$state_is" ]; then
+            echo remove >> "$__messages_out"
+            repo_changed="YES"
+        fi
     ;;
 
     *)
@@ -66,3 +115,7 @@ case $state_should in
         exit 1
     ;;
 esac
+
+if [ -n "$repo_changed" ]; then
+    cat "$__object/parameter/onchange"
+fi

cdist/conf/type/__git/man.rst

@@ -3,12 +3,13 @@ cdist-type__git(7)
 
 NAME
 ----
-cdist-type__git -  Get and or keep git repositories up-to-date
+cdist-type__git -  Get and keep git repositories up-to-date
 
 
 DESCRIPTION
 -----------
-This cdist type allows you to clone git repositories
+This cdist type allows you to clone git repositories, track specific branches
+or tags and keep up to date at your own pace.
 
 
 REQUIRED PARAMETERS
@@ -23,7 +24,7 @@ state
     Either "present" or "absent", defaults to "present"
 
 branch
-    Create this branch by checking out the remote branch of this name
+    Check out this branch or tag to the target directory.
     Default branch is "master"
 
 group
@@ -31,16 +32,45 @@ group
 
 mode
    Unix permissions, suitable for chmod.
+   By default it only applies to the top-level directory.
+   See the mode-recursive parameter as well.
+
+onchange
+   The code to run if the repository is first-cloned, changes or is removed.
 
 owner
    User to chown to.
 
+BOOLEAN PARAMETERS
+------------------
+mode-recursive
+   If present and --mode is passed, the given permissions will be applied
+   recursively to the working directory.
+
+no-updates
+   If present the repository will not be updated after first check out.
+   Notice that this does not affect behaviour if you change the target
+   tag / branch, in which case the type will still ensure the specified one
+   is checked out.
+
 recursive
    Passes the --recurse-submodules flag to git when cloning the repository.
 
 shallow
    Sets --depth=1 and --shallow-submodules for cloning repositories with big history.
 
+MESSAGES
+--------
+change <old_branch> -> <new_branch>
+    The directory tracks old_branch but should track new_branch, it will be changed
+chown -R <owner>:<group>
+    Changed ownership
+create
+    Freshly created the directory with the repository clone
+remove
+    The directory with the repository exists, but state is absent, it will be removed
+update
+    The repository tracks a branch that has been updated on the remote
 
 EXAMPLES
 --------
@@ -52,15 +82,21 @@ EXAMPLES
     # Checkout cdist, stay on branch 2.1
     __git /home/nico/cdist --source git@code.ungleich.ch:ungleich-public/cdist.git --branch 2.1
 
+    # If at a later stage you decide to keep up with the development version,
+    # cdist will change the branch and keep you up to date with:
+    __git /home/nico/cdist --source git@code.ungleich.ch:ungleich-public/cdist.git --branch master
+
 
 AUTHORS
 -------
 Nico Schottelius <nico-cdist--@--schottelius.org>
+Evilham <cvs--@--evilham.com>
 
 
 COPYING
 -------
-Copyright \(C) 2012 Nico Schottelius. You can redistribute it
+Copyright \(C) 2012 Nico Schottelius. 2020 Evilham.
+You can redistribute it
 and/or modify it under the terms of the GNU General Public License as
 published by the Free Software Foundation, either version 3 of the
 License, or (at your option) any later version.

cdist/conf/type/__git/manifest

@@ -32,6 +32,7 @@ mode="$(cat "$__object/parameter/mode")"
 
 case "$state_should" in
     present)
+        # Handled in gencode-remote
         :
     ;;
 

cdist/conf/type/__git/parameter/boolean

@@ -1,2 +1,4 @@
+mode-recursive
+no-updates
 recursive
 shallow

cdist/conf/type/__git/parameter/optional

@@ -1,5 +1,6 @@
 state
 branch
 group
+onchange
 owner
 mode

cdist/conf/type/__group/gencode-remote

@@ -88,7 +88,7 @@ if [ "$state" = "present" ]; then
          fi
       done
       if [ "$os" = "freebsd" ]; then
-         echo pw groupadd "$@" "$name"
+         echo pw groupadd "$name" "$@"
       else
          echo groupadd "$@" "$name"
       fi

cdist/conf/type/__letsencrypt_cert/manifest

@@ -91,6 +91,9 @@ if [ -z "${certbot_fullpath}" ]; then
 
 			certbot_fullpath=/usr/local/bin/certbot
 		;;
+		ubuntu)
+            __package certbot
+            ;;
 		*)
 			echo "Unsupported os: $os" >&2
 			exit 1

cdist/conf/type/__link/man.rst

@@ -18,7 +18,7 @@ source
    Specifies the link source.
 
 type
-   Specifies the link type: Either hard or symoblic.
+   Specifies the link type: Either hard or symbolic.
 
 
 OPTIONAL PARAMETERS

cdist/conf/type/__motd/gencode-remote

@@ -22,13 +22,6 @@
 os=$(cat "$__global/explorer/os")
 
 case "$os" in
-    debian|ubuntu|devuan)
-
-        # Debian and Ubuntu need to be updated,
-        # as seen in /etc/init.d/bootlogs
-        echo "uname -snrvm > /var/run/motd"
-        echo "cat /etc/motd.tail >> /var/run/motd"
-    ;;
     freebsd)
         # FreeBSD only updates /etc/motd on boot,
 	# as seen in /etc/rc.d/motd

cdist/conf/type/__motd/manifest

@@ -33,10 +33,6 @@ os=$(cat "$__global/explorer/os")
 
 
 case "$os" in
-   debian|ubuntu|devuan)
-      # Debian-based systems use /etc/motd.tail as a template
-      destination=/etc/motd.tail
-   ;;
    freebsd)
       # FreeBSD uses motd.template to prepend system information on boot
       # (this actually only applies starting with version 13,

cdist/conf/type/__pyvenv/gencode-remote

@@ -1,6 +1,7 @@
 #!/bin/sh -e
 #
 # 2016 Darko Poljak (darko.poljak at gmail.com)
+# 2020 Nico Schotetlius (nico.schottelius at ungleich.ch)
 #
 # This file is part of cdist.
 #
@@ -45,7 +46,7 @@ then
     pyvenv=$(cat "$pyvenvparam")
 else
     case "$os" in
-        alpine) # no pyvenv on alpine - I assume others will follow
+        alpine|ubuntu) # no pyvenv on alpine - I assume others will follow
             pyvenv="python3 -m venv"
             ;;
         *)

cdist/conf/type/__pyvenv/man.rst

@@ -9,7 +9,7 @@ cdist-type__pyvenv - Create or remove python virtual environment
 DESCRIPTION
 -----------
 This cdist type allows you to create or remove python virtual
-environment using pyvenv.
+environment using pyvenv on python3 -m venv.
 It assumes pyvenv is already installed. Concrete package depends
 on concrete OS and/or OS version/distribution.
 Ensure this for e.g. in your init manifest as in the following example:
@@ -57,7 +57,7 @@ EXAMPLES
 
     __pyvenv /home/services/djangoenv
 
-    # Use specific pyvenv 
+    # Use specific pyvenv
     __pyvenv /home/foo/fooenv --pyvenv /usr/local/bin/pyvenv-3.4
 
     # Create python virtualenv for user foo.
@@ -76,4 +76,3 @@ COPYING
 -------
 Copyright \(C) 2016 Darko Poljak. Free use of this software is
 granted under the terms of the GNU General Public License v3 or later (GPLv3+).
-

cdist/conf/type/__ssh_authorized_key/man.rst

@@ -15,25 +15,27 @@ This type was created to be used by the __ssh_authorized_keys type.
 REQUIRED PARAMETERS
 -------------------
 file
-   the authorized_keys file to which the given key should be added
+   The authorized_keys file where the given key should be managed.
 
 key
-   a string containing the ssh keytype, base 64 encoded key and optional
-   trailing comment which shall be added to the given authorized_keys file.
+   The ssh key which shall be managed in this authorized_keys file.
+   Must be a string containing the ssh keytype, base 64 encoded key and
+   optional trailing comment which shall be added to the given
+   authorized_keys file.
 
 
 OPTIONAL PARAMETERS
 -------------------
 comment
-   explicit comment instead of the one which may be trailing the given key
+   Use this comment instead of the one which may be trailing in the key.
 
 option
-   an option to set for this authorized_key entry.
+   An option to set for this authorized_key entry.
    Can be specified multiple times.
    See sshd(8) for available options.
 
 state
-   if the given keys should be 'present' or 'absent', defaults to 'present'.
+   If the managed key should be 'present' or 'absent', defaults to 'present'.
 
 
 MESSAGES
@@ -64,7 +66,7 @@ EXAMPLES
 
 SEE ALSO
 --------
-:strong:`cdist__ssh_authorized_keys`\ (7), :strong:`sshd`\ (8)
+:strong:`cdist-type__ssh_authorized_keys`\ (7), :strong:`sshd`\ (8)
 
 
 AUTHORS

cdist/conf/type/__ssh_authorized_keys/explorer/keys

@@ -0,0 +1,9 @@
+#!/bin/sh -e
+
+# shellcheck disable=SC1090
+file="$( . "$__type_explorer/file" )"
+
+if [ -f "$file" ]
+then
+    cat "$file"
+fi

cdist/conf/type/__ssh_authorized_keys/man.rst

@@ -20,42 +20,48 @@ then left to the user to ensure that the file exists and that ownership and
 permissions work with ssh.
 
 
-REQUIRED PARAMETERS
--------------------
+REQUIRED MULTIPLE PARAMETERS
+----------------------------
 key
-   the ssh key which shall be added to this authorized_keys file.
-   Must be a string and can be specified multiple times.
+   An ssh key which shall be managed in this authorized_keys file.
+   Must be a string containing the ssh keytype, base 64 encoded key and
+   optional trailing comment which shall be added to the given
+   authorized_keys file.
+   Can be specified multiple times.
 
 
 OPTIONAL PARAMETERS
 -------------------
 comment
-   explicit comment instead of the one which may be trailing the given key
+   Use this comment instead of the one which may be trailing in each key.
 
 file
-   an alternative destination file, defaults to ~$owner/.ssh/authorized_keys
+   An alternative destination file, defaults to ~$owner/.ssh/authorized_keys.
 
 option
-   an option to set for all created authorized_key entries.
+   An option to set for all authorized_key entries in the key parameter.
    Can be specified multiple times.
    See sshd(8) for available options.
 
 owner
-   the user owning the authorized_keys file, defaults to object_id.
+   The user owning the authorized_keys file, defaults to object_id.
 
 state
-   if the given keys should be 'present' or 'absent', defaults to 'present'.
+   If the given keys should be 'present' or 'absent', defaults to 'present'.
 
 
 BOOLEAN PARAMETERS
 ------------------
 noparent
-   don't create or change ownership and permissions of the directory containing
-   the authorized_keys file
+   Don't create or change ownership and permissions of the directory containing
+   the authorized_keys file.
 
 nofile
-   don't manage existence, ownership and permissions of the the authorized_keys
-   file
+   Don't manage existence, ownership and permissions of the the authorized_keys
+   file.
+
+remove-unknown
+   Remove undefined keys.
 
 
 EXAMPLES
@@ -67,6 +73,12 @@ EXAMPLES
     __ssh_authorized_keys root \
        --key "$(cat ~/.ssh/id_rsa.pub)"
 
+    # same as above, but make sure your key is only key in
+    # root's authorized_keys file
+    __ssh_authorized_keys root \
+       --key "$(cat ~/.ssh/id_rsa.pub)" \
+       --remove-unknown
+
     # allow key to login as user-name
     __ssh_authorized_keys user-name \
        --key "ssh-rsa AXYZAAB3NzaC1yc2..."

cdist/conf/type/__ssh_authorized_keys/manifest

@@ -55,8 +55,12 @@ _cksum() {
    echo "$1" | cksum | cut -d' ' -f 1
 }
 
+_type_and_key() {
+   echo "$1" | tr ' ' '\n' | awk '/^(ssh|ecdsa)-[^ ]+/ { printf $1" "; getline; printf $1 }'
+}
+
 while read -r key; do
-   type_and_key="$(echo "$key" | tr ' ' '\n' | awk '/^(ssh|ecdsa)-[^ ]+/ { printf $1" "; getline; printf $1 }')"
+   type_and_key="$( _type_and_key "$key" )"
    object_id="$(_cksum "$file")-$(_cksum "$type_and_key")"
    set -- "$object_id"
    set -- "$@" --file "$file"
@@ -72,3 +76,24 @@ while read -r key; do
    # Ensure __ssh_authorized_key does not read stdin
    __ssh_authorized_key "$@" < /dev/null
 done < "$__object/parameter/key"
+
+if [ -f "$__object/parameter/remove-unknown" ] &&
+    [ -s "$__object/explorer/keys" ]
+then
+    while read -r key
+    do
+        type_and_key="$( _type_and_key "$key" )"
+
+        if grep -Fq "$type_and_key" "$__object/parameter/key"
+        then
+            continue
+        fi
+
+        __ssh_authorized_key "remove-$( _cksum "$file$key" )" \
+            --file "$file" \
+            --key "$key" \
+            --state absent \
+                < /dev/null
+    done \
+        < "$__object/explorer/keys"
+fi

cdist/conf/type/__ssh_authorized_keys/parameter/boolean

@@ -1,2 +1,3 @@
 noparent
 nofile
+remove-unknown

cdist/conf/type/__ssh_authorized_keys/parameter/optional

@@ -1,5 +1,4 @@
 comment
 file
-option
 owner
 state

cdist/conf/type/__ssh_authorized_keys/parameter/optional_multiple

@@ -0,0 +1 @@
+option

cdist/conf/type/__user/gencode-remote

@@ -135,11 +135,19 @@ elif [ "$state" = "absent" ]; then
     if grep -q "^${name}:" "$__object/explorer/passwd"; then
         #user exists, but state != present, so delete it
         if [ -f "$__object/parameter/remove-home" ]; then
-            printf "userdel -r '%s' >/dev/null 2>&1\\n" "${name}"
-	    echo "userdel -r" >> "$__messages_out"
+            if [ "$os" = "freebsd" ]; then
+                printf "pw userdel '%s' -r >/dev/null 2>&1\\n" "${name}"
+            else
+                printf "userdel -r '%s' >/dev/null 2>&1\\n" "${name}"
+            fi
+            echo "userdel -r" >> "$__messages_out"
         else
-            printf "userdel '%s' >/dev/null 2>&1\\n" "${name}"
-	    echo "userdel" >> "$__messages_out"
+            if [ "$os" = "freebsd" ]; then
+                printf "pw userdel '%s' >/dev/null 2>&1\\n" "${name}"
+            else
+                printf "userdel '%s' >/dev/null 2>&1\\n" "${name}"
+            fi
+            echo "userdel" >> "$__messages_out"
         fi
     fi
 else

cdist/config.py

@@ -29,18 +29,20 @@ import time
 import itertools
 import tempfile
 import multiprocessing
-from cdist.mputil import mp_pool_run, mp_sig_handler
 import atexit
 import shutil
 import socket
+
+from cdist.mputil import mp_pool_run, mp_sig_handler
+from cdist import core, inventory
+from cdist.util.remoteutil import inspect_ssh_mux_opts
+
 import cdist
 import cdist.hostsource
 import cdist.exec.local
 import cdist.exec.remote
 import cdist.util.ipaddr as ipaddr
 import cdist.configuration
-from cdist import core, inventory
-from cdist.util.remoteutil import inspect_ssh_mux_opts
 
 
 def graph_check_cycle(graph):
@@ -195,7 +197,6 @@ class Config(object):
     @classmethod
     def commandline(cls, args):
         """Configure remote system"""
-
         if (args.parallel and args.parallel != 1) or args.jobs:
             if args.timestamp:
                 cdist.log.setupTimestampingParallelLogging()
@@ -203,6 +204,7 @@ class Config(object):
                 cdist.log.setupParallelLogging()
         elif args.timestamp:
             cdist.log.setupTimestampingLogging()
+
         log = logging.getLogger("config")
 
         # No new child process if only one host at a time.
@@ -381,10 +383,16 @@ class Config(object):
            If operating in parallel then return tuple (host, True|False, )
            so that main process knows for which host function was successful.
         """
-
         log = logging.getLogger(host)
 
         try:
+            if args.log_server:
+                # Start a log server so that nested `cdist config` runs
+                # have a place to send their logs to.
+                log_server_socket_dir = tempfile.mkdtemp()
+                cls._register_path_for_removal(log_server_socket_dir)
+                cdist.log.setupLogServer(log_server_socket_dir, log)
+
             remote_exec, remote_copy, cleanup_cmd = cls._resolve_remote_cmds(
                 args)
             log.debug("remote_exec for host \"{}\": {}".format(

cdist/configuration.py

@@ -27,6 +27,7 @@ import cdist.argparse
 import re
 import multiprocessing
 import logging
+import sys
 
 
 class Singleton(type):
@@ -246,9 +247,33 @@ class LogLevelOption(OptionBase):
         return VerbosityOption().translate(val)
 
 
+class ColoredOutputOption(BooleanOption):
+    CHOICES = ('always', 'never', 'auto')
+    DEFAULT = 'auto'
+
+    def get_converter(self):
+        return self.translate
+
+    @staticmethod
+    def translate(val):
+        if isinstance(val, bool):
+            return val
+        elif val == 'always':
+            return True
+        elif val == 'never':
+            return False
+        elif val == 'auto':
+            return 'NO_COLOR' not in os.environ and sys.stdout.isatty()
+
+
+ColoredOutputOption.DEFAULT = ColoredOutputOption.translate(
+    ColoredOutputOption.DEFAULT)
+
+
 _ARG_OPTION_MAPPING = {
     'beta': 'beta',
     'cache_path_pattern': 'cache_path_pattern',
+    'colored_output': 'colored_output',
     'conf_dir': 'conf_dir',
     'manifest': 'init_manifest',
     'out_path': 'out_path',
@@ -294,6 +319,7 @@ class Configuration(metaclass=Singleton):
             'remote_shell': StringOption('remote_shell'),
             'cache_path_pattern': StringOption('cache_path_pattern'),
             'conf_dir': ConfDirOption(),
+            'colored_output': ColoredOutputOption('colored_output'),
             'init_manifest': StringOption('init_manifest'),
             'out_path': StringOption('out_path'),
             'remote_out_path': StringOption('remote_out_path'),
@@ -319,6 +345,7 @@ class Configuration(metaclass=Singleton):
         'CDIST_REMOTE_COPY': 'remote_copy',
         'CDIST_INVENTORY_DIR': 'inventory_dir',
         'CDIST_CACHE_PATH_PATTERN': 'cache_path_pattern',
+        'CDIST_COLORED_OUTPUT': 'colored_output',
         '__cdist_log_level': 'verbosity',
     }
     ENV_VAR_BOOLEAN_OPTIONS = ('CDIST_BETA', )
@@ -327,11 +354,10 @@ class Configuration(metaclass=Singleton):
     }
 
     ARG_OPTION_MAPPING = _ARG_OPTION_MAPPING
-    ADJUST_ARG_OPTION_MAPPING = {
-       _ARG_OPTION_MAPPING[key]: key for key in _ARG_OPTION_MAPPING
-    }
+    ADJUST_ARG_OPTION_MAPPING = {v: k for k, v in _ARG_OPTION_MAPPING.items()}
     REQUIRED_DEFAULT_CONFIG_VALUES = {
         'GLOBAL': {
+            'colored_output': 'auto',
             'verbosity': 0,
         },
     }
@@ -484,8 +510,7 @@ class Configuration(metaclass=Singleton):
             newconfig = self._read_config_file(config_file)
             self._update_config_dict(config, newconfig)
         # command line config file
-        if (self.args and 'config_file' in self.args and
-                self.args['config_file']):
+        if (self.args and self.args.get('config_file', None)):
             newconfig = self._read_config_file(self.args['config_file'])
             self._update_config_dict(config, newconfig)
         # command line

cdist/core/code.py

@@ -116,6 +116,10 @@ class Code(object):
         if dry_run:
             self.env['__cdist_dry_run'] = '1'
 
+        if '__cdist_log_server_socket_export' in os.environ:
+            self.env['__cdist_log_server_socket'] = os.environ[
+                '__cdist_log_server_socket_export']
+
     def _run_gencode(self, cdist_object, which):
         cdist_type = cdist_object.cdist_type
         script = os.path.join(self.local.type_path,

cdist/core/manifest.py

@@ -119,6 +119,8 @@ class Manifest(object):
             '__cdist_log_level': util.log_level_env_var_val(self.log),
             '__cdist_log_level_name': util.log_level_name_env_var_val(
                 self.log),
+            '__cdist_colored_log': str(
+                cdist.log.CdistFormatter.USE_COLORS).lower(),
         }
 
         if dry_run:

cdist/emulator.py

@@ -129,6 +129,9 @@ class Emulator(object):
             # if invalid __cdist_log_level value
             logging.root.setLevel(logging.WARNING)
 
+        colored_log = self.env.get('__cdist_colored_log', 'false')
+        cdist.log.CdistFormatter.USE_COLORS = colored_log == 'true'
+
         self.log = logging.getLogger(self.target_host[0])
 
     def commandline(self):

cdist/exec/remote.py

@@ -280,9 +280,6 @@ class Remote(object):
         assert isinstance(command, (list, tuple)), (
                 "list or tuple argument expected, got: %s" % command)
 
-        if return_output and stdout is not subprocess.PIPE:
-            self.log.debug("return_output is True, ignoring stdout")
-
         close_stdout = False
         close_stderr = False
         if self.save_output_streams:

cdist/flock.py

@@ -22,6 +22,7 @@
 import fcntl
 import logging
 import os
+import cdist.log
 
 
 log = logging.getLogger('cdist-flock')

cdist/install.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 #
-# 2013 Steven Armstrong (steven-cdist at armstrong.cc)
+# 2013-2019 Steven Armstrong (steven-cdist at armstrong.cc)
 #
 # This file is part of cdist.
 #
@@ -22,9 +22,21 @@
 
 import cdist.config
 import cdist.core
+import cdist.log
 
 
 class Install(cdist.config.Config):
+
+    @classmethod
+    def onehost(cls, host, host_tags, host_base_path, host_dir_name, args,
+                parallel, configuration, remove_remote_files_dirs=False):
+        # Always start log server during cdist install so that nested
+        # `cdist config` runs have a place to send their logs to.
+        args.log_server = True
+
+        super().onehost(host, host_tags, host_base_path, host_dir_name, args,
+                        parallel, configuration, remove_remote_files_dirs)
+
     def object_list(self):
         """Short name for object list retrieval.
         In install mode, we only care about install objects.

cdist/log.py

@@ -2,6 +2,7 @@
 # -*- coding: utf-8 -*-
 #
 # 2010-2013 Nico Schottelius (nico-cdist at schottelius.org)
+# 2019-2020 Steven Armstrong
 #
 # This file is part of cdist.
 #
@@ -20,9 +21,16 @@
 #
 #
 
-import logging
-import sys
 import datetime
+import logging
+import logging.handlers
+import sys
+import os
+import asyncio
+import contextlib
+import pickle
+import struct
+import threading
 
 
 # Define additional cdist logging levels.
@@ -50,9 +58,32 @@ def _trace(msg, *args, **kwargs):
 logging.trace = _trace
 
 
-class DefaultLog(logging.Logger):
+class CdistFormatter(logging.Formatter):
+    USE_COLORS = False
+    RESET = '\033[0m'
+    COLOR_MAP = {
+        'ERROR': '\033[0;31m',
+        'WARNING': '\033[0;33m',
+        'INFO': '\033[0;94m',
+        'VERBOSE': '\033[0;34m',
+        'DEBUG': '\033[0;90m',
+        'TRACE': '\033[0;37m',
+    }
 
-    FORMAT = '%(levelname)s: %(message)s'
+    def __init__(self, fmt):
+        super().__init__(fmt=fmt)
+
+    def format(self, record):
+        msg = super().format(record)
+        if self.USE_COLORS:
+            color = self.COLOR_MAP.get(record.levelname)
+            if color:
+                msg = color + msg + self.RESET
+        return msg
+
+
+class DefaultLog(logging.Logger):
+    FORMAT = '%(levelname)s: %(name)s: %(message)s'
 
     class StdoutFilter(logging.Filter):
         def filter(self, rec):
@@ -64,30 +95,28 @@ class DefaultLog(logging.Logger):
 
     def __init__(self, name):
         super().__init__(name)
+        self.propagate = False
 
-        formatter = logging.Formatter(self.FORMAT)
+        if '__cdist_log_server_socket' in os.environ:
+            log_server_socket = os.environ['__cdist_log_server_socket']
+            socket_handler = logging.handlers.SocketHandler(log_server_socket,
+                                                            None)
+            self.addHandler(socket_handler)
+        else:
+            formatter = CdistFormatter(self.FORMAT)
 
-        self.addFilter(self)
+            stdout_handler = logging.StreamHandler(sys.stdout)
+            stdout_handler.addFilter(self.StdoutFilter())
+            stdout_handler.setLevel(logging.TRACE)
+            stdout_handler.setFormatter(formatter)
 
-        stdout_handler = logging.StreamHandler(sys.stdout)
-        stdout_handler.addFilter(self.StdoutFilter())
-        stdout_handler.setLevel(logging.TRACE)
-        stdout_handler.setFormatter(formatter)
+            stderr_handler = logging.StreamHandler(sys.stderr)
+            stderr_handler.addFilter(self.StderrFilter())
+            stderr_handler.setLevel(logging.ERROR)
+            stderr_handler.setFormatter(formatter)
 
-        stderr_handler = logging.StreamHandler(sys.stderr)
-        stderr_handler.addFilter(self.StderrFilter())
-        stderr_handler.setLevel(logging.ERROR)
-        stderr_handler.setFormatter(formatter)
-
-        self.addHandler(stdout_handler)
-        self.addHandler(stderr_handler)
-
-    def filter(self, record):
-        """Prefix messages with logger name"""
-
-        record.msg = self.name + ": " + str(record.msg)
-
-        return True
+            self.addHandler(stdout_handler)
+            self.addHandler(stderr_handler)
 
     def verbose(self, msg, *args, **kwargs):
         self.log(logging.VERBOSE, msg, *args, **kwargs)
@@ -110,7 +139,7 @@ class TimestampingLog(DefaultLog):
 
 
 class ParallelLog(DefaultLog):
-    FORMAT = '%(levelname)s: [%(process)d]: %(message)s'
+    FORMAT = '%(levelname)s: [%(process)d]: %(name)s: %(message)s'
 
 
 class TimestampingParallelLog(TimestampingLog, ParallelLog):
@@ -137,4 +166,42 @@ def setupParallelLogging():
     logging.setLoggerClass(ParallelLog)
 
 
+async def handle_log_client(reader, writer):
+    while True:
+        chunk = await reader.read(4)
+        if len(chunk) < 4:
+            return
+
+        data_size = struct.unpack('>L', chunk)[0]
+        data = await reader.read(data_size)
+
+        obj = pickle.loads(data)
+        record = logging.makeLogRecord(obj)
+        logger = logging.getLogger(record.name)
+        logger.handle(record)
+
+
+def run_log_server(server_address):
+    # Get a new loop inside the current thread to run the log server.
+    loop = asyncio.new_event_loop()
+    loop.create_task(asyncio.start_unix_server(handle_log_client,
+                                               server_address))
+    loop.run_forever()
+
+
+def setupLogServer(socket_dir, log=logging.getLogger(__name__)):
+    """Run a asyncio based unix socket log server in a background thread.
+    """
+    log_server_socket = os.path.join(socket_dir, 'log-server')
+    log.debug('Starting logging server on: %s', log_server_socket)
+    os.environ['__cdist_log_server_socket_export'] = log_server_socket
+    with contextlib.suppress(FileNotFoundError):
+        os.remove(log_server_socket)
+    t = threading.Thread(target=run_log_server, args=(log_server_socket,))
+    # Deamonizing the thread means we don't have to care about stoping it.
+    # It will die together with the main process.
+    t.daemon = True
+    t.start()
+
+
 setupDefaultLogging()

cdist/sphinxext/manpage.py

@@ -7,6 +7,7 @@ from docutils.io import FileOutput
 from os import path
 from sphinx.util.nodes import inline_all_toctrees
 from sphinx import addnodes
+from sphinx.util import logging
 
 """
     Extension based on sphinx builtin manpage.
@@ -15,6 +16,9 @@ from sphinx import addnodes
 """
 
 
+logger = logging.getLogger(__name__)
+
+
 class ManualPageTranslator(sphinx.writers.manpage.ManualPageTranslator):
 
     def header(self):
@@ -28,7 +32,7 @@ class ManualPageWriter(sphinx.writers.manpage.ManualPageWriter):
     def __init__(self, builder):
         super().__init__(builder)
         self.translator_class = (
-            self.builder.translator_class or ManualPageTranslator)
+            self.builder.get_translator_class() or ManualPageTranslator)
 
 
 class ManualPageBuilder(sphinx.builders.manpage.ManualPageBuilder):
@@ -43,7 +47,7 @@ class ManualPageBuilder(sphinx.builders.manpage.ManualPageBuilder):
             components=(docwriter,),
             read_config_files=True).get_default_values()
 
-        self.info(bold('writing... '), nonl=True)
+        logger.info(bold('writing... '), nonl=True)
 
         for info in self.config.man_pages:
             docname, name, description, authors, section = info
@@ -54,7 +58,7 @@ class ManualPageBuilder(sphinx.builders.manpage.ManualPageBuilder):
                     authors = []
 
             targetname = '%s.%s' % (name, section)
-            self.info(darkgreen(targetname) + ' { ', nonl=True)
+            logger.info(darkgreen(targetname) + ' { ', nonl=True)
             destination = FileOutput(
                 destination_path=path.join(self.outdir, targetname),
                 encoding='utf-8')
@@ -63,7 +67,7 @@ class ManualPageBuilder(sphinx.builders.manpage.ManualPageBuilder):
             docnames = set()
             largetree = inline_all_toctrees(self, docnames, docname, tree,
                                             darkgreen, [docname])
-            self.info('} ', nonl=True)
+            logger.info('} ', nonl=True)
             self.env.resolve_references(largetree, docname, self)
             # remove pending_xref nodes
             for pendingnode in largetree.traverse(addnodes.pending_xref):
@@ -76,7 +80,7 @@ class ManualPageBuilder(sphinx.builders.manpage.ManualPageBuilder):
             largetree.settings.section = section
 
             docwriter.write(largetree, destination)
-        self.info()
+        logger.info("")
 
 
 def setup(app):

cdist/test/configuration/__init__.py

@@ -28,10 +28,12 @@ import argparse
 from cdist import test
 import cdist.argparse as cap
 import logging
+import sys
 
 my_dir = op.abspath(op.dirname(__file__))
 fixtures = op.join(my_dir, 'fixtures')
 interpolation_config_file = op.join(fixtures, "interpolation-test.cfg")
+colored_output_default = 'auto'
 
 
 def newConfigParser():
@@ -153,6 +155,7 @@ class ConfigurationTestCase(test.CdistTestCase):
             'remote_shell': '/bin/sh',
             'inventory_dir': '',
             'cache_path_pattern': '',
+            'colored_output': colored_output_default,
             'conf_dir': '',
             'init_manifest': '',
             'out_path': '',
@@ -184,6 +187,8 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/bin/sh',
                 'inventory_dir': None,
                 'cache_path_pattern': None,
+                'colored_output': cc.ColoredOutputOption.translate(
+                    colored_output_default),
                 'conf_dir': None,
                 'init_manifest': None,
                 'out_path': None,
@@ -390,6 +395,7 @@ class ConfigurationTestCase(test.CdistTestCase):
         args = argparse.Namespace()
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'verbosity': 0,
             },
         }
@@ -440,6 +446,7 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/bin/sh',
                 'inventory_dir': None,
                 'cache_path_pattern': None,
+                'colored_output': colored_output_default,
                 'conf_dir': None,
                 'init_manifest': None,
                 'out_path': None,
@@ -515,6 +522,7 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/usr/bin/sh',
                 'inventory_dir': '/var/db/cdist/inventory',
                 'cache_path_pattern': None,
+                'colored_output': colored_output_default,
                 'conf_dir': ['/opt/cdist', ],
                 'init_manifest': None,
                 'out_path': None,
@@ -556,6 +564,7 @@ class ConfigurationTestCase(test.CdistTestCase):
             'remote_shell': '/bin/sh',
             'inventory_dir': '',
             'cache_path_pattern': '',
+            'colored_output': colored_output_default,
             'conf_dir': '',
             'init_manifest': '',
             'out_path': '',
@@ -579,6 +588,8 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/usr/bin/sh',
                 'inventory_dir': None,
                 'cache_path_pattern': None,
+                'colored_output': cc.ColoredOutputOption.translate(
+                    colored_output_default),
                 'conf_dir': [
                     '/opt/cdist/conf',
                     '/usr/local/share/cdist/conf',
@@ -623,6 +634,7 @@ class ConfigurationTestCase(test.CdistTestCase):
             'remote_shell': '/bin/sh',
             'inventory_dir': '',
             'cache_path_pattern': '',
+            'colored_output': colored_output_default,
             'conf_dir': '',
             'init_manifest': '',
             'out_path': '',
@@ -645,6 +657,7 @@ class ConfigurationTestCase(test.CdistTestCase):
             'local_shell': '/usr/bin/sh',
             'remote_shell': '/usr/bin/sh',
             'inventory_dir': '/var/db/cdist/inventory',
+            'colored_output': colored_output_default,
             'conf_dir': '/opt/cdist',
             'remote_copy': 'myscp',
             'remote_exec': 'myexec',
@@ -663,6 +676,8 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/usr/bin/sh',
                 'inventory_dir': '/var/db/cdist/inventory',
                 'cache_path_pattern': None,
+                'colored_output': cc.ColoredOutputOption.translate(
+                    colored_output_default),
                 'conf_dir': [
                     '/opt/cdist/conf',
                     '/usr/local/share/cdist/conf',
@@ -694,6 +709,7 @@ class ConfigurationTestCase(test.CdistTestCase):
         }
         expected_config = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'verbosity': 0,
             },
         }
@@ -767,6 +783,7 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/usr/bin/sh',
                 'inventory_dir': '/opt/sysadmin/cdist/inventory',
                 'cache_path_pattern': None,
+                'colored_output': colored_output_default,
                 'conf_dir': [
                     '/opt/cdist/conf',
                     '/usr/local/share/cdist/conf',
@@ -865,6 +882,7 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/usr/bin/sh',
                 'inventory_dir': '/var/db/cdist/inventory',
                 'cache_path_pattern': None,
+                'colored_output': colored_output_default,
                 'conf_dir': [
                     '/opt/conf/cdist',
                 ],
@@ -964,6 +982,7 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/usr/bin/sh',
                 'inventory_dir': '/var/db/cdist/inventory',
                 'cache_path_pattern': None,
+                'colored_output': colored_output_default,
                 'conf_dir': [
                     '/opt/conf/cdist',
                 ],
@@ -1063,6 +1082,7 @@ class ConfigurationTestCase(test.CdistTestCase):
                 'remote_shell': '/usr/bin/sh',
                 'inventory_dir': '/var/db/cdist/inventory',
                 'cache_path_pattern': None,
+                'colored_output': colored_output_default,
                 'conf_dir': [
                     '/opt/conf/cdist',
                 ],
@@ -1095,6 +1115,7 @@ class ConfigurationTestCase(test.CdistTestCase):
             'beta': True,
             'inventory_dir': '/var/db/cdist/inventory',
             'cache_path_pattern': None,
+            'colored_output': colored_output_default,
             'conf_dir': [
                 '/opt/conf/cdist',
             ],
@@ -1125,6 +1146,7 @@ class ConfigurationTestCase(test.CdistTestCase):
         expected_config_dict = {
             'GLOBAL': {
                 'inventory_dir': None,
+                'colored_output': colored_output_default,
                 'conf_dir': None,
                 'verbosity': 0,
             },
@@ -1148,6 +1170,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'verbosity': cap.VERBOSE_DEBUG,
             },
         }
@@ -1185,6 +1208,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'save_output_streams': True,
                 'verbosity': 0,
             },
@@ -1213,6 +1237,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'save_output_streams': False,
                 'verbosity': 0,
             },
@@ -1241,6 +1266,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'save_output_streams': False,
                 'verbosity': 0,
             },
@@ -1269,6 +1295,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'save_output_streams': False,
                 'verbosity': 0,
             },
@@ -1308,6 +1335,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'timestamp': True,
                 'verbosity': 0,
             },
@@ -1336,6 +1364,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'timestamp': True,
                 'verbosity': 0,
             },
@@ -1364,6 +1393,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'timestamp': False,
                 'verbosity': 0,
             },
@@ -1392,6 +1422,7 @@ class ConfigurationTestCase(test.CdistTestCase):
 
         expected_config_dict = {
             'GLOBAL': {
+                'colored_output': colored_output_default,
                 'timestamp': False,
                 'verbosity': 0,
             },

configuration/cdist.cfg.skeleton

@@ -13,17 +13,28 @@
 #     Specify cache path pattern.
 # cache_path_pattern = %h
 #
+# colored_output
+#     Colorize cdist's output. If enabled, cdist will use different colors for
+#     different log levels.
+#     Recognized values are 'always', 'never', and 'auto'.
+#     If the value is 'auto', colors are enabled if stdout is a TTY unless
+#     the NO_COLOR (https://no-color.org/) environment variable is defined.
+# colored_output = auto
+#
 # conf_dir
 #     List of configuration directories separated with the character conventionally
 #     used by the operating system to separate search path components (as in PATH),
 #     such as ':' for POSIX or ';' for Windows.
 #     If also specified at command line then values from command line are
 #     appended to this value.
+#     Notice that this works in a "last one wins" fashion, so if a type is redefined
+#     in multiple conf_dirs, the last one in which it is defined will be used.
+#     Consider using a unique prefix for your own roles if this can be an issue.
 # conf_dir = <dir1>:<dir2>
 #
 # init_manifest
 #     Specify default initial manifest.
-# init_mainfest = <path-to-init-manifst>
+# init_manifest = <path-to-init-manifest>
 #
 # inventory_dir
 #     Specify inventory directory.
@@ -42,7 +53,7 @@
 #
 # out_path
 #     Directory to save cdist output in.
-# out_path = 
+# out_path =
 #
 # parallel
 #     Process hosts in parallel. If -1 then the default, number of CPU's in
@@ -68,6 +79,6 @@
 # remote_shell = /bin/sh
 #
 # verbosity
-#     Set verbosity level. Valid values are: 
+#     Set verbosity level. Valid values are:
 #     ERROR, WARNING, INFO, VERBOSE, DEBUG, TRACE and OFF.
 # verbosity = INFO

docs/changelog

@@ -2,6 +2,33 @@ Changelog
 ---------
 
 next:
+	* New type: __download (Ander Punnar)
+
+6.6.0: 2020-06-17
+	* Type __ssh_authorized_keys: Add option for removing undefined keys (Ander Punnar)
+	* Core: Support colored log output (Evil Ham)
+	* Core: Tune colored log output and respect NO_COLOR env var (Dennis Camera)
+	* Documentation: Fix failing man pages build with newer sphinx versions (Darko Poljak)
+	* Documentation: Fix trivial grammatical mistakes (Jaak Ristioja)
+	* Explorer os: Fix for sles15 (Daniel Heule)
+	* Type __clean_path: Add --path parameter (Ander Punnar)
+	* Core: Increase minimal supported Python version to 3.5 (Darko Poljak)
+	* Core: Add log server for nested logging (Steven Armstrong)
+
+6.5.6: 2020-05-25
+	* Type __pyvenv: Switch to python3 -m venv for Ubuntu (Nico Schottelius)
+	* Type __letsencrypt_cert: Whitelist Ubuntu (Nico Schottelius)
+	* Types __cron, __file, __link: Improve manpages (Matthias Stecher)
+	* Explorer machine_type: Add support for FreeBSD and OpenBSD, and simplify Linux code (Evil Ham)
+	* Type __ssh_authorized_key, __ssh_authorized_keys: Improve manpages (Evil Ham)
+	* Type __ssh_authorized_keys: Fix bug where --option was not multiple (Evil Ham)
+	* Type __motd: Debian/Ubuntu/Devuan use /etc/motd (Ander Punnar)
+	* Type __group: Fix --gid on FreeBSD (Ander Punnar)
+	* Configuration: Fix typos in cdist.cfg.skeleton (Jaak Ristioja)
+	* Type __user: Fix user deletion on FreeBSD (Ander Punnar)
+	* Core: Fix double log lines (Darko Poljak)
+
+6.5.5: 2020-05-01
 	* Core: Fix XDG_CONFIG_HOME config file location (Joachim Desroches)
 	* Type __postgres_database: Add encoding, lc-collate, lc-ctype, template parameters (Timothée Floure)
 	* Type __motd: Improve documentation and support for FreeBSD (Evil Ham)
@@ -10,6 +37,9 @@ next:
 	* New type: __pf_apply_anchor (Kamila Součková, Evil Ham)
 	* Type __pf_ruleset: Refactor (Kamila Součková, Evil Ham)
 	* Type __pf_apply: Deprecate type (Kamila Součková, Evil Ham)
+	* Configuration: Add notes to cdist.cfg.skeleton (Evil Ham)
+	* Explorers cpu_cores, memory: Improve *BSD support (Evil Ham)
+	* Core: Remove debug logging noise (Evil Ham)
 
 6.5.4: 2020-04-11
 	* Explorer init: Do not grep on non-existent init (Steven Armstrong)

docs/src/cdist-bootstrap.rst

@@ -25,7 +25,7 @@ people, have a look at `cdist best practice <cdist-best-practice.html>`_.
 Setup working directory and branch
 ----------------------------------
 I assume you have a fresh copy of the cdist tree in ~/cdist, cloned from
-one of the official urls (see `cdist quickstart <cdist-quickstart.html>`_ if you don't).
+one of the official URLs (see `cdist quickstart <cdist-quickstart.html>`_ if you don't).
 Entering the command "git branch" should show you "* master", which indicates
 you are on the **master** branch.
 

docs/src/cdist-cache.rst

@@ -59,7 +59,7 @@ typeorder
 
 Object cache overview
 ~~~~~~~~~~~~~~~~~~~~~
-Each object under :strong:`object` directory has its own structurue.
+Each object under :strong:`object` directory has its own structure.
 
 code-local
     code generated from gencode-local, present only if something is

docs/src/cdist-configuration.rst

@@ -21,7 +21,7 @@ precedence. Configuration option value read from source with higher
 precedence will overwrite option value, if exists, read from source with
 lower precedence. That means that command-line option wins them all.
 
-Users can decide on the local conifguration file location. It can be either
+Users can decide on the local configuration file location. It can be either
 ~/.cdist.cfg or $XDG_CONFIG_HOME/cdist/cdist.cfg. Note that, if both exist,
 then ~/.cdist.cfg is used.
 

docs/src/cdist-install.rst

@@ -9,15 +9,15 @@ Source Host
 
 This is the machine from which you will configure target hosts.
 
- * /bin/sh: A posix like shell (for instance bash, dash, zsh)
- * Python >= 3.2
+ * /bin/sh: A POSIX like shell (for instance bash, dash, zsh)
+ * Python >= 3.5
  * SSH client
  * sphinx (for building html docs and/or the man pages)
 
 Target Hosts
 ~~~~~~~~~~~~
 
- * /bin/sh: A posix like shell (for instance bash, dash, zsh)
+ * /bin/sh: A POSIX like shell (for instance bash, dash, zsh)
  * SSH server
 
 Install cdist

docs/src/cdist-real-world.rst

@@ -198,7 +198,7 @@ We require package uWSGI present in order to create **/etc/uwsgi/apps-enabled/$u
 Installation of uWSGI also creates configuration layout: **/etc/uwsgi/apps-enabled**.
 If this directory does not exist then **__file** type would error.
 We also use stdin as file content source. For details see `Input from stdin <cdist-type.html#input-from-stdin>`_.
-For feading stdin we use here-document (**<<** operator). It allows redirection of subsequent
+For feeding stdin we use here-document (**<<** operator). It allows redirection of subsequent
 lines read by the shell to the input of a command until a line containing only the delimiter
 and a newline, with no blank characters in between (EOF in our case).
 
@@ -546,7 +546,7 @@ we have changed our **wsgi.py** file uWSGI reloads the application.
 
 Our application selects and lists items from **items** table.
 
-Openning application
+Opening application
 ~~~~~~~~~~~~~~~~~~~~
 
 Finally try the application::

docs/src/cdist-reference.rst.sh

@@ -34,7 +34,7 @@ dest="$__cdist_abs_mydir/$filename"
 cd "$__cdist_abs_mydir"
 
 exec > "$dest"
-cat << eof 
+cat << eof
 Reference
 =========
 Variable, path and type reference for cdist
@@ -51,7 +51,7 @@ eof
     done
 )
 
-cat << eof 
+cat << eof
 
 Paths
 -----
@@ -187,13 +187,13 @@ usable within a object directory:
 
 files
     This directory is reserved for user data and will not be used
-    by cdist at any time. It can be used freely by the type 
+    by cdist at any time. It can be used freely by the type
     (for instance to store template results).
 changed
     This empty file exists in an object directory, if the object has
     code to be executed (either remote or local).
 stdin
-    This file exists and contains data, if data was provided on stdin 
+    This file exists and contains data, if data was provided on stdin
     when the type was called.
 
 
@@ -222,65 +222,89 @@ __cdist_log_level, __cdist_log_level_name
     | TRACE          | 5               |
     +----------------+-----------------+
 
+    Available for: initial manifest, explorer, type manifest, type explorer,
+    type gencode.
+__cdist_colored_log
+    whether or not cdist's log has colors enabled.
+    Is set to the string ``true`` if cdist's output is using colors,
+    otherwise the variable contains the string ``false``.
+
     Available for: initial manifest, explorer, type manifest, type explorer,
     type gencode.
 __cdist_dry_run
     Is set only when doing dry run (``-n`` flag).
+
     Available for: initial manifest, explorer, type manifest, type explorer,
     type gencode.
 __explorer
     Directory that contains all global explorers.
+
     Available for: initial manifest, explorer, type explorer, shell.
 __files
     Directory that contains content from the "files" subdirectories
     from the configuration directories.
+
     Available for: initial manifest, type manifest, type gencode, shell.
 __manifest
     Directory that contains the initial manifest.
+
     Available for: initial manifest, type manifest, shell.
 __global
     Directory that contains generic output like explorer.
+
     Available for: initial manifest, type manifest, type gencode, shell.
 __messages_in
     File to read messages from.
+
     Available for: initial manifest, type manifest, type gencode.
 __messages_out
     File to write messages.
+
     Available for: initial manifest, type manifest, type gencode.
 __object
     Directory that contains the current object.
+
     Available for: type manifest, type explorer, type gencode and code scripts.
 __object_id
     The type unique object id.
+
     Available for: type manifest, type explorer, type gencode and code scripts.
-    Note: The leading and the trailing "/" will always be stripped (caused by
-    the filesystem database and ensured by the core).
-    Note: Double slashes ("//") will not be fixed and result in an error.
+
+    | Note: The leading and the trailing "/" will always be stripped (caused by
+      the filesystem database and ensured by the core).
+    | Note: Double slashes ("//") will not be fixed and result in an error.
 __object_name
     The full qualified name of the current object.
+
     Available for: type manifest, type explorer, type gencode.
 __target_host
     The host we are deploying to. This is primary variable. It's content is
     literally the one user passed in.
+
     Available for: explorer, initial manifest, type explorer, type manifest, type gencode, shell.
 __target_hostname
     The hostname of host we are deploying to. This variable is derived from
     **__target_host** (using **socket.getaddrinfo(__target_host)** and then
     **socket.gethostbyaddr()**).
+
     Available for: explorer, initial manifest, type explorer, type manifest, type gencode, shell.
 __target_fqdn
     The fully qualified domain name of the host we are deploying to.
     This variable is derived from **__target_host**
     (using **socket.getfqdn()**).
+
     Available for: explorer, initial manifest, type explorer, type manifest, type gencode, shell.
 __target_host_tags
     Comma separated list of target host tags.
+
     Available for: explorer, initial manifest, type explorer, type manifest, type gencode, shell.
 __type
     Path to the current type.
+
     Available for: type manifest, type gencode.
 __type_explorer
     Directory that contains the type explorers.
+
     Available for: type explorer.
 
 Environment variables (for writing)
@@ -344,6 +368,11 @@ CDIST_INVENTORY_DIR
 CDIST_BETA
     Enable beta functionalities.
 
+CDIST_COLORED_OUTPUT
+    Colorize cdist's output. If enabled, cdist will use different colors for
+    different log levels.
+    Recognized values are 'always', 'never', and 'auto' (the default).
+
 CDIST_CACHE_PATH_PATTERN
     Custom cache path pattern.
 eof

docs/src/cdist-support.rst

@@ -3,7 +3,7 @@ Support
 
 Chat
 ~~~~
-Chat with us: `ungleich chat <https://chat.ungleich.ch/ungleich/channels/cdist>`_.
+Chat with us on `#cdist:ungleich.ch <https://ungleich.ch/u/projects/open-chat/>`_.
 
 Mailing list
 ~~~~~~~~~~~~

docs/src/cdist-upgrade.rst

@@ -14,7 +14,7 @@ To upgrade cdist in the current branch use
     make man
     export MANPATH=$MANPATH:$(pwd -P)/doc/man
 
-If you stay on a version branche (i.e. 1.0, 1.1., ...), nothing should break.
+If you stay on a version branch (i.e. 1.0, 1.1., ...), nothing should break.
 The master branch on the other hand is the development branch and may not be
 working, break your setup or eat the tree in your garden.
 
@@ -61,7 +61,7 @@ After that, you can go back and continue the upgrade:
 Update the python package
 -------------------------
 
-To upgrade to the lastet version do
+To upgrade to the latest version do
 
 .. code-block:: sh
 
@@ -158,7 +158,7 @@ Updating from 1.5 to 1.6
 * If you used **\_\_package_apt --preseed**, you need to use the new
   type **\_\_debconf_set_selections** instead.
 * The **\_\_package** types accepted either --state deinstalled or
-  --state uninstaaled. Starting with 1.6, it was made consistently
+  --state uninstalled. Starting with 1.6, it was made consistently
   to --state removed.
 
 Updating from 1.3 to 1.5

docs/src/cdist-why.rst

@@ -21,7 +21,7 @@ Not only is shell scripting widely known by system engineers,
 but it is also a very powerful language. Here are some features
 which make daily work easy:
 
- * Configuration can react dynamicly on explored values
+ * Configuration can react dynamically on explored values
  * High level string manipulation (using sed, awk, grep)
  * Conditional support (**if, case**)
  * Loop support (**for, while**)
@@ -44,7 +44,7 @@ Cdist requires very little on a target system. Even better,
 in almost all cases all dependencies are usually fulfilled.
 Cdist does not require an agent or high level programming
 languages on the target host: it will run on any host that
-has a **ssh server running** and a posix compatible shell
+has a **ssh server running** and a POSIX compatible shell
 (**/bin/sh**). Compared to other configuration management systems,
 it does not require to open up an additional port.
 

docs/src/man1/cdist.rst

@@ -15,52 +15,51 @@ SYNOPSIS
 
     cdist banner [-h] [-l LOGLEVEL] [-q] [-v]
 
-    cdist config [-h] [-l LOGLEVEL] [-q] [-v] [-b] [-g CONFIG_FILE] [-4]
-                 [-6] [-C CACHE_PATH_PATTERN] [-c CONF_DIR] [-i MANIFEST]
-                 [-j [JOBS]] [-n] [-o OUT_PATH] [-P]
-                 [-R [{tar,tgz,tbz2,txz}]] [-r REMOTE_OUT_PATH]
-                 [--remote-copy REMOTE_COPY] [--remote-exec REMOTE_EXEC]
-                 [-S] [-I INVENTORY_DIR] [-A] [-a] [-f HOSTFILE]
-                 [-p [HOST_MAX]] [-s] [-t]
+    cdist config [-h] [-l LOGLEVEL] [-q] [-v] [-b] [--colors WHEN]
+                 [-g CONFIG_FILE] [-4] [-6] [-C CACHE_PATH_PATTERN]
+                 [-c CONF_DIR] [-i MANIFEST] [-j [JOBS]] [--log-server]
+                 [-n] [-o OUT_PATH] [-P] [-R [{tar,tgz,tbz2,txz}]]
+                 [-r REMOTE_OUT_PATH] [--remote-copy REMOTE_COPY]
+                 [--remote-exec REMOTE_EXEC] [-S] [-I INVENTORY_DIR] [-A]
+                 [-a] [-f HOSTFILE] [-p [HOST_MAX]] [-s] [-t]
                  [host [host ...]]
 
-    cdist install [-h] [-l LOGLEVEL] [-q] [-v] [-b] [-g CONFIG_FILE] [-4]
-                  [-6] [-C CACHE_PATH_PATTERN] [-c CONF_DIR] [-i MANIFEST]
-                  [-j [JOBS]] [-n] [-o OUT_PATH] [-P]
-                  [-R [{tar,tgz,tbz2,txz}]] [-r REMOTE_OUT_PATH]
-                  [--remote-copy REMOTE_COPY] [--remote-exec REMOTE_EXEC]
-                  [-S] [-I INVENTORY_DIR] [-A] [-a] [-f HOSTFILE]
-                  [-p [HOST_MAX]] [-s] [-t]
+    cdist install [-h] [-l LOGLEVEL] [-q] [-v] [-b] [--colors WHEN]
+                  [-g CONFIG_FILE] [-4] [-6] [-C CACHE_PATH_PATTERN]
+                  [-c CONF_DIR] [-i MANIFEST] [-j [JOBS]] [--log-server]
+                  [-n] [-o OUT_PATH] [-P] [-R [{tar,tgz,tbz2,txz}]]
+                  [-r REMOTE_OUT_PATH] [--remote-copy REMOTE_COPY]
+                  [--remote-exec REMOTE_EXEC] [-S] [-I INVENTORY_DIR] [-A]
+                  [-a] [-f HOSTFILE] [-p [HOST_MAX]] [-s] [-t]
                   [host [host ...]]
 
     cdist inventory [-h] {add-host,add-tag,del-host,del-tag,list} ...
 
-    cdist inventory add-host [-h] [-l LOGLEVEL] [-q] [-v] [-b]
-                             [-g CONFIG_FILE] [-I INVENTORY_DIR]
-                             [-f HOSTFILE]
+    cdist inventory add-host [-h] [-l LOGLEVEL] [-q] [-v] [-b] [--colors WHEN]
+                             [-g CONFIG_FILE] [-I INVENTORY_DIR] [-f HOSTFILE]
                              [host [host ...]]
 
-    cdist inventory add-tag [-h] [-l LOGLEVEL] [-q] [-v] [-b]
-                            [-g CONFIG_FILE] [-I INVENTORY_DIR]
-                            [-f HOSTFILE] [-T TAGFILE] [-t TAGLIST]
+    cdist inventory add-tag [-h] [-l LOGLEVEL] [-q] [-v] [-b] [--colors WHEN]
+                            [-g CONFIG_FILE] [-I INVENTORY_DIR] [-f HOSTFILE]
+                            [-T TAGFILE] [-t TAGLIST]
                             [host [host ...]]
 
-    cdist inventory del-host [-h] [-l LOGLEVEL] [-q] [-v] [-b]
+    cdist inventory del-host [-h] [-l LOGLEVEL] [-q] [-v] [-b] [--colors WHEN]
                              [-g CONFIG_FILE] [-I INVENTORY_DIR] [-a]
                              [-f HOSTFILE]
                              [host [host ...]]
 
-    cdist inventory del-tag [-h] [-l LOGLEVEL] [-q] [-v] [-b]
+    cdist inventory del-tag [-h] [-l LOGLEVEL] [-q] [-v] [-b] [--colors WHEN]
                             [-g CONFIG_FILE] [-I INVENTORY_DIR] [-a]
                             [-f HOSTFILE] [-T TAGFILE] [-t TAGLIST]
                             [host [host ...]]
 
-    cdist inventory list [-h] [-l LOGLEVEL] [-q] [-v] [-b] [-g CONFIG_FILE]
-                         [-I INVENTORY_DIR] [-a] [-f HOSTFILE] [-H] [-t]
+    cdist inventory list [-h] [-l LOGLEVEL] [-q] [-v] [-b] [--colors WHEN]
+                         [-g CONFIG_FILE] [-I INVENTORY_DIR] [-a] [-f HOSTFILE]
+                         [-H] [-t]
                          [host [host ...]]
 
-    cdist preos [-h] [-l LOGLEVEL] [-q] [-v] [-c CONF_DIR] [-g CONFIG_FILE]
-                [-L]
+    cdist preos [-h] [-l LOGLEVEL] [-q] [-v] [-c CONF_DIR] [-g CONFIG_FILE] [-L]
                 [preos] ...
 
     cdist preos [preos-options] debian [-h] [-l LOGLEVEL] [-q] [-v] [-b] [-a ARCH] [-B]
@@ -84,9 +83,10 @@ SYNOPSIS
                                        [-S SCRIPT] [-s SUITE] [-y REMOTE_COPY]
                                        target_dir
 
-    cdist shell [-h] [-l LOGLEVEL] [-q] [-v] [-s SHELL]
+    cdist shell [-h] [-l LOGLEVEL] [-q] [-v] [--colors WHEN] [-s SHELL]
 
-    cdist info [-h] [-a] [-c CONF_DIR] [-e] [-F] [-f] [-g CONFIG_FILE] [-t] [pattern]
+    cdist info [-h] [-a] [-c CONF_DIR] [-e] [-F] [-f] [-g CONFIG_FILE] [-t]
+               [pattern]
 
 
 DESCRIPTION
@@ -95,7 +95,7 @@ cdist is the frontend executable to the cdist configuration management.
 It supports different subcommands as explained below.
 
 It is written in Python so it requires :strong:`python`\ (1) to be installed.
-It requires a minimal Python version 3.2.
+It requires a minimal Python version 3.5.
 
 GENERAL
 -------
@@ -104,6 +104,14 @@ All commands accept the following options:
 **-h, --help**
     Show the help screen.
 
+**--colors WHEN**
+    Colorize cdist's output. If enabled, cdist will use different colors for
+    different log levels.
+    WHEN recognizes the values 'always', 'never', and 'auto' (the default).
+
+    If the value is 'auto', colored output is enabled if stdout is a TTY
+    unless the NO_COLOR (https://no-color.org/) environment variable is defined.
+
 **-l LOGLEVEL, --log-level LOGLEVEL**
     Set the specified verbosity level. The levels, in
     order from the lowest to the highest, are: ERROR (-1),
@@ -156,7 +164,7 @@ Install command is currently in beta.
 
 **-b, --beta**
     Enable beta functionality.
-    
+
 **-C CACHE_PATH_PATTERN, --cache-path-pattern CACHE_PATH_PATTERN**
     Specify custom cache path pattern. If it is not set then
     default hostdir is used. For more info on format see
@@ -179,7 +187,7 @@ Install command is currently in beta.
 
 **-I INVENTORY_DIR, --inventory INVENTORY_DIR**
     Use specified custom inventory directory. Inventory
-    directory is set up by the following rules: if cdist 
+    directory is set up by the following rules: if cdist
     configuration resolves this value then specified
     directory is used, if HOME env var is set then
     ~/.cdit/inventory is used, otherwise distribution
@@ -194,6 +202,12 @@ Install command is currently in beta.
     are supported. Without argument CPU count is used by
     default.
 
+**--log-server**
+    Start a log server for sub processes to use. This is
+    mainly useful when running cdist nested from a code-
+    local script. Log server is always implicitly started
+    for 'install' command.
+
 **-n, --dry-run**
     Do not execute code.
 
@@ -294,7 +308,7 @@ Add host(s) to inventory database.
 
 **-I INVENTORY_DIR, --inventory INVENTORY_DIR**
     Use specified custom inventory directory. Inventory
-    directory is set up by the following rules: if cdist 
+    directory is set up by the following rules: if cdist
     configuration resolves this value then specified
     directory is used, if HOME env var is set then
     ~/.cdit/inventory is used, otherwise distribution
@@ -324,7 +338,7 @@ Add tag(s) to inventory database.
 
 **-I INVENTORY_DIR, --inventory INVENTORY_DIR**
     Use specified custom inventory directory. Inventory
-    directory is set up by the following rules: if cdist 
+    directory is set up by the following rules: if cdist
     configuration resolves this value then specified
     directory is used, if HOME env var is set then
     ~/.cdit/inventory is used, otherwise distribution
@@ -367,7 +381,7 @@ Delete host(s) from inventory database.
 
 **-I INVENTORY_DIR, --inventory INVENTORY_DIR**
     Use specified custom inventory directory. Inventory
-    directory is set up by the following rules: if cdist 
+    directory is set up by the following rules: if cdist
     configuration resolves this value then specified
     directory is used, if HOME env var is set then
     ~/.cdit/inventory is used, otherwise distribution
@@ -401,7 +415,7 @@ Delete tag(s) from inventory database.
 
 **-I INVENTORY_DIR, --inventory INVENTORY_DIR**
     Use specified custom inventory directory. Inventory
-    directory is set up by the following rules: if cdist 
+    directory is set up by the following rules: if cdist
     configuration resolves this value then specified
     directory is used, if HOME env var is set then
     ~/.cdit/inventory is used, otherwise distribution
@@ -448,7 +462,7 @@ List inventory database.
 
 **-I INVENTORY_DIR, --inventory INVENTORY_DIR**
     Use specified custom inventory directory. Inventory
-    directory is set up by the following rules: if cdist 
+    directory is set up by the following rules: if cdist
     configuration resolves this value then specified
     directory is used, if HOME env var is set then
     ~/.cdit/inventory is used, otherwise distribution
@@ -673,6 +687,9 @@ The possible keywords and their meanings are as follows:
 :strong:`cache_path_pattern`
     Specify cache path pattern.
 
+:strong:`colored_output`
+    Colorize cdist's output. cf. the :code:`--colors` option.
+
 :strong:`conf_dir`
     List of configuration directories separated with the character conventionally
     used by the operating system to separate search path components (as in PATH),
@@ -726,7 +743,7 @@ The possible keywords and their meanings are as follows:
     in the format: YYYYMMDDHHMMSS.us.
 
 :strong:`verbosity`
-    Set verbosity level. Valid values are: 
+    Set verbosity level. Valid values are:
     'ERROR', 'WARNING', 'INFO', 'VERBOSE', 'DEBUG', 'TRACE' and 'OFF'.
 
 
@@ -758,7 +775,7 @@ cdist/preos
 NOTES
 -----
 cdist detects if host is specified by IPv6 address. If so then remote_copy
-command is executed with host address enclosed in square brackets 
+command is executed with host address enclosed in square brackets
 (see :strong:`scp`\ (1)).
 
 EXAMPLES
@@ -893,6 +910,9 @@ CDIST_BETA
 CDIST_CACHE_PATH_PATTERN
     Custom cache path pattern.
 
+CDIST_COLORED_OUTPUT
+    Colorize cdist's output. cf. the :code:`--colors` option.
+
 CDIST_CONFIG_FILE
     Custom configuration file.
 
@@ -955,5 +975,5 @@ such case and display a warning message. An example of such a case:
 
 COPYING
 -------
-Copyright \(C) 2011-2019 Nico Schottelius. Free use of this software is
+Copyright \(C) 2011-2020 Nico Schottelius. Free use of this software is
 granted under the terms of the GNU General Public License v3 or later (GPLv3+).

scripts/cdist

@@ -60,10 +60,9 @@ def commandline():
 
 
 if __name__ == "__main__":
-    cdistpythonversion = '3.2'
-    if sys.version < cdistpythonversion:
+    if sys.version < cdist.MIN_SUPPORTED_PYTHON_VERSION:
         print('Python >= {} is required on the source host.'.format(
-                cdistpythonversion), file=sys.stderr)
+                cdist.MIN_SUPPORTED_PYTHON_VERSIO), file=sys.stderr)
         sys.exit(1)
 
     exit_code = 0