Release cdist 6.5.6

This commit is contained in:
Darko Poljak 2020-05-25 11:23:25 +02:00
parent a817358ddb
commit a7a8c05e1d
445 changed files with 123362 additions and 0 deletions

View file

@ -1,6 +1,20 @@
Changelog
---------
6.5.6: 2020-05-25
~~~~~~~~~~~~~~~~~
* Type __pyvenv: Switch to python3 -m venv for Ubuntu (Nico Schottelius)
* Type __letsencrypt_cert: Whitelist Ubuntu (Nico Schottelius)
* Types __cron, __file, __link: Improve manpages (Matthias Stecher)
* Explorer machine_type: Add support for FreeBSD and OpenBSD, and simplify Linux code (Evil Ham)
* Type __ssh_authorized_key, __ssh_authorized_keys: Improve manpages (Evil Ham)
* Type __ssh_authorized_keys: Fix bug where --option was not multiple (Evil Ham)
* Type __motd: Debian/Ubuntu/Devuan use /etc/motd (Ander Punnar)
* Type __group: Fix --gid on FreeBSD (Ander Punnar)
* Configuration: Fix typos in cdist.cfg.skeleton (Jaak Ristioja)
* Type __user: Fix user deletion on FreeBSD (Ander Punnar)
* Core: Fix double log lines (Darko Poljak)
6.5.5: 2020-05-01
~~~~~~~~~~~~~~~~~
* Core: Fix XDG_CONFIG_HOME config file location (Joachim Desroches)

View file

@ -8,6 +8,7 @@ cdist manual
**All versions**
* `6.5.6 <manual/6.5.6>`_
* `6.5.5 <manual/6.5.5>`_
* `6.5.4 <manual/6.5.4>`_
* `6.5.3 <manual/6.5.3>`_

View file

@ -0,0 +1,4 @@
# Sphinx build info version 1
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
config: 0c1a90bce34fe6e656fea2a631050203
tags: 645f666f9bcd5a90fca523b33c5a78b7

View file

@ -0,0 +1,278 @@
Best practice
=============
Practices used in real environments
Passwordless connections
------------------------
It is recommended to run cdist with public key authentication.
This requires a private/public key pair and the entry
"PermitRootLogin without-password" in the sshd server.
See sshd_config(5) and ssh-keygen(1).
Speeding up ssh connections
---------------------------
When connecting to a new host, the initial delay with ssh connections
is pretty big. As cdist makes many connections to each host successive
connections can be sped up by "sharing of multiple sessions over a single
network connection" (quote from ssh_config(5)). This is also called "connection
multiplexing".
Cdist implements this since v4.0.0 by executing ssh with the appropriate
options (`-o ControlMaster=auto -o ControlPath=/tmp/<tmpdir>/s -o
ControlPersist=2h`).
Note that the sshd_config on the server can configure the maximum number of
parallel multiplexed connections this with `MaxSessions N` (N defaults to 10
for OpenSSH v7.4).
Speeding up shell execution
----------------------------
On the source host, ensure that /bin/sh is *not* bash: bash is quite slow for
script execution. Instead, you could use dash after installing it::
ln -sf /bin/dash /bin/sh
Multi master or environment setups
----------------------------------
If you plan to distribute cdist among servers or use different
environments, you can do so easily with the included version
control git. For instance if you plan to use the typical three
environments production, integration and development, you can
realise this with git branches::
# Go to cdist checkout
cd /path/to/cdist
# Create branches
git branch development
git branch integration
git branch production
# Make use of a branch, for instance production
git checkout production
Similar if you want to have cdist checked out at multiple machines,
you can clone it multiple times::
machine-a % git clone git://your-git-server/cdist
machine-b % git clone git://your-git-server/cdist
Separating work by groups
-------------------------
If you are working with different groups on one cdist-configuration,
you can delegate to other manifests and have the groups edit only
their manifests. You can use the following snippet in
**conf/manifests/init**::
# Include other groups
sh -e "$__manifest/systems"
sh -e "$__manifest/cbrg"
Maintaining multiple configurations
-----------------------------------
When you need to manage multiple sites with cdist, like company_a, company_b
and private for instance, you can easily use git for this purpose.
Including a possible common base that is reused across the different sites::
# create branches
git branch company_a company_b common private
# make stuff for company a
git checkout company_a
# work, commit, etc.
# make stuff for company b
git checkout company_b
# work, commit, etc.
# make stuff relevant for all sites
git checkout common
# work, commit, etc.
# change to private and include latest common stuff
git checkout private
git merge common
The following **.git/config** is taken from a real world scenario::
# Track upstream, merge from time to time
[remote "upstream"]
url = git://git.schottelius.org/cdist
fetch = +refs/heads/*:refs/remotes/upstream/*
# Same as upstream, but works when being offline
[remote "local"]
fetch = +refs/heads/*:refs/remotes/local/*
url = /home/users/nico/p/cdist
# Remote containing various ETH internal branches
[remote "eth"]
url = sans.ethz.ch:/home/services/sans/git/cdist-eth
fetch = +refs/heads/*:refs/remotes/eth/*
# Public remote that contains my private changes to cdist upstream
[remote "nico"]
url = git.schottelius.org:/home/services/git/cdist-nico
fetch = +refs/heads/*:refs/remotes/nico/*
# The "nico" branch will be synced with the remote nico, branch master
[branch "nico"]
remote = nico
merge = refs/heads/master
# ETH stable contains rock solid configurations used in various places
[branch "eth-stable"]
remote = eth
merge = refs/heads/stable
Have a look at git-remote(1) to adjust the remote configuration, which allows
Multiple developers with different trust
----------------------------------------
If you are working in an environment that requires different people to
work on the same configuration, but having different privileges, you can
implement this scenario with a gateway host and sudo:
- Create a dedicated user (for instance **cdist**)
- Setup the ssh-pubkey for this user that has the right to configure all hosts
- Create a wrapper to update the cdist configuration in ~cdist/cdist
- Allow every developer to execute this script via sudo as the user cdist
- Allow run of cdist as user cdist on specific hosts on a per user/group basis.
- f.i. nico ALL=(ALL) NOPASSWD: /home/cdist/bin/cdist config hostabc
For more details consult sudoers(5)
Templating
----------
* create directory files/ in your type (convention)
* create the template as an executable file like files/basic.conf.sh, it will output text using shell variables for the values
.. code-block:: sh
#!/bin/sh
# in the template, use cat << eof (here document) to output the text
# and use standard shell variables in the template
# output everything in the template script to stdout
cat << EOF
server {
listen 80;
server_name $SERVERNAME;
root $ROOT;
access_log /var/log/nginx/$SERVERNAME_access.log
error_log /var/log/nginx/$SERVERNAME_error.log
}
EOF
* in the manifest, export the relevant variables and add the following lines to your manifest:
.. code-block:: console
# export variables needed for the template
export SERVERNAME='test"
export ROOT='/var/www/test'
# render the template
mkdir -p "$__object/files"
"$__type/files/basic.conf.sh" > "$__object/files/basic.conf"
# send the rendered template
__file /etc/nginx/sites-available/test.conf \
--state present
--source "$__object/files/basic.conf"
Testing a new type
------------------
If you want to test a new type on a node, you can tell cdist to only use an
object of this type: Use the '--initial-manifest' parameter
with - (stdin) as argument and feed object into stdin
of cdist:
.. code-block:: sh
# Singleton type without parameter
echo __ungleich_munin_server | cdist --initial-manifest - munin.panter.ch
# Singleton type with parameter
echo __ungleich_munin_node --allow 1.2.3.4 | \
cdist --initial-manifest - rails-19.panter.ch
# Normal type
echo __file /tmp/stdintest --mode 0644 | \
cdist --initial-manifest - cdist-dev-01.ungleich.ch
Other content in cdist repository
---------------------------------
Usually the cdist repository contains all configuration
items. Sometimes you may have additional resources that
you would like to store in your central configuration
repository (like password files from KeepassX,
Libreoffice diagrams, etc.).
It is recommended to use a subfolder named "non-cdist"
in the repository for such content: It allows you to
easily distinguish what is used by cdist and what is not
and also to store all important files in one
repository.
Notes on CDIST_ORDER_DEPENDENCY
-------------------------------
With CDIST_ORDER_DEPENDENCY all types are executed in the order in which they
are created in the manifest. The current created object automatically depends
on the previously created object.
It essentially helps you to build up blocks of code that build upon each other
(like first creating the directory xyz than the file below the directory).
This can be helpful, but one must be aware of its side effects.
CDIST_ORDER_DEPENDENCY kills parallelization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Suppose you have defined CDIST_ORDER_DEPENDENCY and then, among other things,
you specify creation of three, by nature independent, files.
**init**
.. code-block:: sh
CDIST_ORDER_DEPENDENCY=1
export CDIST_ORDER_DEPENDENCY
...
__file /tmp/file1
__file /tmp/file2
__file /tmp/file3
...
Due to defined CDIST_ORDER_DEPENDENCY cdist will execute them in specified order.
It is better to use CDIST_ORDER_DEPENDENCY in well defined blocks:
**init**
.. code-block:: sh
CDIST_ORDER_DEPENDENCY=1
export CDIST_ORDER_DEPENDENCY
...
unset CDIST_ORDER_DEPENDENCY
__file /tmp/file1
__file /tmp/file2
__file /tmp/file3
CDIST_ORDER_DEPENDENCY=1
export CDIST_ORDER_DEPENDENCY
...
unset CDIST_ORDER_DEPENDENCY

View file

@ -0,0 +1,118 @@
Bootstrap
=========
This document describes the usual steps recommended for a new
cdist setup. It is recommended that you have read and understood
`cdist quickstart <cdist-quickstart.html>`_ before digging into this.
Location
---------
First of all, you should think about where to store your configuration
database and who will be accessing or changing it. Secondly you have to
think about where to configure your hosts from, which may be a different
location.
For starters, having cdist (which includes the configuration database) on
your notebook should be fine.
Additionally an external copy of the git repository the configuration
relies on is recommended, for use as backup as well as to allow easy collaboration
with others.
For more sophisticated setups developing cdist configurations with multiple
people, have a look at `cdist best practice <cdist-best-practice.html>`_.
Setup working directory and branch
----------------------------------
I assume you have a fresh copy of the cdist tree in ~/cdist, cloned from
one of the official urls (see `cdist quickstart <cdist-quickstart.html>`_ if you don't).
Entering the command "git branch" should show you "* master", which indicates
you are on the **master** branch.
The master branch reflects the latest development of cdist. As this is the
development branch, it may or may not work. There are also version branches
available, which are kept in a stable state. Let's use **git branch -r**
to list all branches::
cdist% git branch -r
origin/1.0
origin/1.1
origin/1.2
origin/1.3
origin/1.4
origin/1.5
origin/1.6
origin/1.7
origin/2.0
origin/HEAD -> origin/master
origin/archive_shell_function_approach
origin/master
So **2.0** is the latest version branch in this example.
All versions (2.0.x) within one version branch (2.0) are compatible to each
other and won't break your configuration when updating.
It's up to you to decide which branch you want to base your own work on:
master contains more recent changes, newer types, but may also break.
The version branches are stable, but may lack the latest features.
Your decision can be changed later on, but may result in merge conflicts,
which you will need to solve.
Let's assume you want latest stuff and select the master branch as base for
your own work. Now it's time to create your branch, which contains your
local changes. I usually name it by the company/area I am working for:
ethz-systems, localch, customerX, ... But this is pretty much up to you.
In this tutorial I use the branch **mycompany**::
cdist% git checkout -b mycompany origin/master
Branch mycompany set up to track remote branch master from origin.
Switched to a new branch 'mycompany'
cdist-user% git branch
master
* mycompany
From now on, you can use git as usual to commit your changes in your own branch.
Publishing the configuration
----------------------------
Usually a development machine like a notebook should be considered
temporary only. For this reason and to enable shareability, the configuration
should be published to another device as early as possible. The following
example shows how to publish the configuration to another host that is
reachable via ssh and has git installed::
# Create bare git repository on the host named "loch"
cdist% ssh loch "GIT_DIR=/home/nutzer/cdist git init"
Initialized empty Git repository in /home/nutzer/cdist/
# Add remote git repo to git config
cdist% git remote add loch loch:/home/nutzer/cdist
# Configure the mycompany branch to push to loch
cdist% git config branch.mycompany.remote loch
# Configure mycompany branch to push into remote master branch
cdist% git config branch.mycompany.merge refs/heads/master
# Push mycompany branch to remote branch master initially
cdist% git push loch mycompany:refs/heads/master
Now you have setup the git repository to synchronise the **mycompany**
branch with the **master** branch on the host **loch**. Thus you can commit
as usual in your branch and push out changes by entering **git push**.
Updating from origin
--------------------
Whenever you want to update your cdist installation, you can use git to do so::
# Update git repository with latest changes from origin
cdist% git fetch origin
# Update current branch with master branch from origin
cdist% git merge origin/master
# Alternative: Update current branch with 2.0 branch from origin
cdist% git merge origin/2.0

View file

@ -0,0 +1,98 @@
Local cache overview
====================
Description
-----------
While executing, cdist stores data to local cache. Currently this feature is
one way only. That means that cdist does not use stored data for future runs.
Anyway, those data can be used for debugging cdist, debugging types and
debugging after host configuration fails.
Local cache is saved under $HOME/.cdist/cache directory, one directory entry
for each host. Subdirectory path is specified by
:strong:`-C/--cache-path-pattern` option, :strong:`cache_path_pattern`
configuration option or by using :strong:`CDIST_CACHE_PATH_PATTERN`
environment variable.
For more info on cache path pattern see :strong:`CACHE PATH PATTERN FORMAT`
section in cdist man page.
Cache overview
--------------
As noted above each configured host has got its subdirectory in local cache.
Entries in host's cache directory are as follows.
bin
directory with cdist type emulators
conf
dynamically determined cdist conf directory, union of all specified
conf directories
explorer
directory containing global explorer named files containing explorer output
after running on target host
messages
file containing messages
object
directory containing subdirectory for each cdist object
object_marker
object marker for this particular cdist run
stderr
directory containing init manifest and remote stderr stream output
stdout
directory containing init manifest and remote stdout stream output
target_host
file containing target host of this cdist run, as specified when running
cdist
typeorder
file containing types in order of execution.
Object cache overview
~~~~~~~~~~~~~~~~~~~~~
Each object under :strong:`object` directory has its own structurue.
code-local
code generated from gencode-local, present only if something is
generated
code-remote
code generated from gencode-remote, present only if something is
generated
explorer
directory containing type explorer named files containing explorer output
after running on target host
files
directory with object files created during type execution
parameter
directory containing type parameter named files containing parameter
values
source
this type's source (init manifest)
state
this type execution state ('done' when finished)
stderr
directory containing type's manifest, gencode-* and code-* stderr stream
outputs
stdin
this type stdin content
stdout
directory containing type's manifest, gencode-* and code-* stdout stream
outputs.

View file

@ -0,0 +1,41 @@
Configuration
=============
Description
-----------
cdist obtains configuration data from the following sources in the following
order:
#. command-line options
#. configuration file specified at command-line using -g command line option
#. configuration file specified in CDIST_CONFIG_FILE environment variable
#. environment variables
#. user's configuration file (first one found of ~/.cdist.cfg, $XDG_CONFIG_HOME/cdist/cdist.cfg, in specified order)
#. in-distribution configuration file (cdist/conf/cdist.cfg)
#. system-wide configuration file (/etc/cdist.cfg)
if one exists.
Configuration source with lower ordering number from above has a higher
precedence. Configuration option value read from source with higher
precedence will overwrite option value, if exists, read from source with
lower precedence. That means that command-line option wins them all.
Users can decide on the local conifguration file location. It can be either
~/.cdist.cfg or $XDG_CONFIG_HOME/cdist/cdist.cfg. Note that, if both exist,
then ~/.cdist.cfg is used.
For a per-project configuration, particular environment variables or better,
CDIST_CONFIG_FILE environment variable or -g CONFIG_FILE command line option,
can be used.
Config file format
------------------
cdist configuration file is in the INI file format. Currently it supports
only [GLOBAL] section.
Here you can find configuration file skeleton:
.. literalinclude:: cdist.cfg.skeleton
:language: ini

View file

@ -0,0 +1,54 @@
Explorer
========
Description
-----------
Explorers are small shell scripts, which will be executed on the target
host. The aim of each explorer is to give hints to types on how to act on the
target system. An explorer outputs the result to stdout, which is usually
a one liner, but may be empty or multi line especially in the case of
type explorers.
There are general explorers, which are run in an early stage, and
type explorers. Both work almost exactly the same way, with the difference
that the values of the general explorers are stored in a general location and
the type specific below the object.
Explorers can reuse other explorers on the target system by calling
::
$__explorer/<explorer_name> (general and type explorer)
or
::
$__type_explorer/<explorer name> (type explorer).
In case of significant errors, the explorer may exit non-zero and return an
error message on stderr, which will cause cdist to abort.
You can also use stderr for debugging purposes while developing a new
explorer.
Examples
--------
A very simple explorer may look like this::
hostname
Which is in practise the **hostname** explorer.
A type explorer, which could check for the status of a package may look like this:
.. code-block:: sh
if [ -f "$__object/parameter/name" ]; then
name="$(cat "$__object/parameter/name")"
else
name="$__object_id"
fi
# Expect dpkg failing, if package is not known / installed
dpkg -s "$name" 2>/dev/null || exit 0

View file

@ -0,0 +1,48 @@
Features
========
But cdist ticks differently, here is the feature set that makes it unique:
Simplicity
There is only one type to extend cdist called **type**
Design
+ Type and core cleanly separated
+ Sticks completely to the KISS (keep it simple and stupid) paradigm
+ Meaningful error messages - do not lose time debugging error messages
+ Consistency in behaviour, naming and documentation
+ No surprise factor: Only do what is obviously clear, no magic
+ Define target state, do not focus on methods or scripts
+ Push architecture: Instantly apply your changes
Small core
cdist's core is very small - less code, less bugs
Fast development
Focus on straightforwardness of type creation is a main development objective
Batteries included: A lot of requirements can be solved using standard types
Modern Programming Language
cdist is written in Python
Requirements, Scalability
No central server needed, cdist operates in push mode and can be run from any computer
Requirements, Scalability, Upgrade
cdist only needs to be updated on the master, not on the target hosts
Requirements, Security
Uses well-know `SSH <http://www.openssh.com/>`_ as transport protocol
Requirements, Simplicity
Requires only shell and SSH server on the target
UNIX
Reuse of existing tools like cat, find, mv, ...
UNIX, familiar environment, documentation
Is available as manpages and HTML
UNIX, simplicity, familiar environment
cdist is configured in POSIX shell

View file

@ -0,0 +1,145 @@
Hacking
=======
Welcome
-------
Welcome dear hacker! I invite you to a tour of pointers to
get into the usable configuration management system, cdist.
The first thing to know is probably that cdist is brought to
you by people who care about how code looks like and who think
twice before merging or implementing a feature: Less features
with good usability are far better than the opposite.
Reporting bugs
--------------
If you believe you've found a bug and verified that it is
in the latest version, drop a mail to the cdist mailing list,
subject prefixed with "[BUG] " or create an issue on code.ungleich.ch.
Coding conventions (everywhere)
-------------------------------
If something should be improved or needs to be fixed, add the word FIXME
nearby, so grepping for FIXME gives all positions that need to be fixed.
Indentation is 4 spaces (welcome to the python world).
How to submit stuff for inclusion into upstream cdist
-----------------------------------------------------
If you did some cool changes to cdist, which you think might be of benefit to other
cdist users, you're welcome to propose inclusion into upstream.
There are some requirements to ensure your changes don't break other peoples
work nor kill the authors brain:
- All files should contain the usual header (Author, Copying, etc.)
- Code submission must be done via git
- Do not add cdist/conf/manifest/init - This file should only be touched in your
private branch!
- Code to be included should be branched of the upstream "master" branch
- Exception: Bugfixes to a version branch
- On a merge request, always name the branch I should pull from
- Always ensure **all** manpages build. Use **./build man** to test.
- If you developed more than **one** feature, consider submitting them in
separate branches. This way one feature can already be included, even if
the other needs to be improved.
As soon as your work meets these requirements, write a mail
for inclusion to the mailinglist **cdist-configuration-management at googlegroups.com**
or open a merge request at https://code.ungleich.ch/ungleich-public/cdist.
How to submit a new type
------------------------
For detailed information about types, see `cdist type <cdist-type.html>`_.
Submitting a type works as described above, with the additional requirement
that a corresponding manpage named man.rst in ReSTructured text format with
the manpage-name "cdist-type__NAME" is included in the type directory
AND the manpage builds (`make man`).
Warning: Submitting "exec" or "run" types that simply echo their parameter in
**gencode** will not be accepted, because they are of no use. Every type can output
code and thus such a type introduces redundant functionality that is given by
core cdist already.
Example git workflow
---------------------
The following workflow works fine for most developers
.. code-block:: sh
# get latest upstream master branch
git clone https://code.ungleich.ch/ungleich-public/cdist.git
# update if already existing
cd cdist; git fetch -v; git merge origin/master
# create a new branch for your feature/bugfix
cd cdist # if you haven't done before
git checkout -b documentation_cleanup
# *hack*
*hack*
# clone the cdist repository on code.ungleich.ch if you haven't done so
# configure your repo to know about your clone (only once)
git remote add ungleich git@code.ungleich.ch:YOURUSERNAME/cdist.git
# push the new branch to ungleich gitlab
git push ungleich documentation_cleanup
# (or everything)
git push --mirror ungleich
# create a merge request at ungleich gitlab (use a browser)
# *fixthingsbecausequalityassurancefoundissuesinourpatch*
*hack*
# push code to ungleich gitlab again
git push ... # like above
# add comment that everything should be green now (use a browser)
# go back to master branch
git checkout master
# update master branch that includes your changes now
git fetch -v origin
git diff master..origin/master
git merge origin/master
If at any point you want to go back to the original master branch, you can
use **git stash** to stash your changes away::
.. code-block:: sh
# assume you are on documentation_cleanup
git stash
# change to master and update to most recent upstream version
git checkout master
git fetch -v origin
git merge origin/master
Similarly when you want to develop another new feature, you go back
to the master branch and create another branch based on it::
.. code-block:: sh
# change to master and update to most recent upstream version
git checkout master
git fetch -v origin
git merge origin/master
git checkout -b another_feature
(you can repeat the code above for as many features as you want to develop
in parallel)

View file

@ -0,0 +1,182 @@
How to install cdist
====================
Requirements
-------------
Source Host
~~~~~~~~~~~
This is the machine from which you will configure target hosts.
* /bin/sh: A posix like shell (for instance bash, dash, zsh)
* Python >= 3.2
* SSH client
* sphinx (for building html docs and/or the man pages)
Target Hosts
~~~~~~~~~~~~
* /bin/sh: A posix like shell (for instance bash, dash, zsh)
* SSH server
Install cdist
-------------
From git
~~~~~~~~
Cloning cdist from git gives you the advantage of having
a version control in place for development of your own stuff
immediately.
To install cdist, execute the following commands:
.. code-block:: sh
git clone https://code.ungleich.ch/ungleich-public/cdist.git
cd cdist
export PATH=$PATH:$(pwd -P)/bin
From version 4.2.0 cdist tags and releases are signed.
You can get GPG public key used for signing `here <_static/pgp-key-EFD2AE4EC36B6901.asc>`_.
It is assumed that you are familiar with *git* ways of signing and verification.
You can also get cdist from `github mirror <https://github.com/ungleich/cdist>`_.
To install cdist with distutils from cloned repository, first you have to
create version.py:
.. code-block:: sh
./bin/build-helper version
Then you install it with:
.. code-block:: sh
make install
or with:
.. code-block:: sh
make install-user
to install it into user *site-packages* directory.
Or directly with distutils:
.. code-block:: sh
python setup.py install
Note that `bin/build-helper` script is intended for cdist maintainers.
Available versions in git
^^^^^^^^^^^^^^^^^^^^^^^^^
* The active development takes place in the **master** branch
* The released versions can be found in the tags
Other branches may be available for features or bugfixes, but they
may vanish at any point. To select a specific branch use
.. code-block:: sh
# Generic code
git checkout -b <localbranchname> origin/<branchname>
So for instance if you want to use and stay with version 4.1, you can use
.. code-block:: sh
git checkout -b 4.1 origin/4.1
Building and using documentation (man and html)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you want to build and use the documentation, run:
.. code-block:: sh
make docs
Documentation comes in two formats, man pages and full HTML
documentation. Documentation is built into distribution's
docs/dist directory. man pages are in docs/dist/man and
HTML documentation in docs/dist/html.
If you want to use man pages, run:
.. code-block:: sh
export MANPATH=$MANPATH:$(pwd -P)/docs/dist/man
Or you can move man pages from docs/dist/man directory to some
other directory and add it to MANPATH.
Full HTML documentation can be accessed at docs/dist/html/index.html.
You can also build only man pages or only html documentation, for
only man pages run:
.. code-block:: sh
make man
for only html documentation run:
.. code-block:: sh
make html
You can also build man pages for types in your ~/.cdist directory:
.. code-block:: sh
make dotman
Built man pages are now in docs/dist/man directory. If you have
some other custom .cdist directory, e.g. /opt/cdist then use:
.. code-block:: sh
make DOT_CDIST_PATH=/opt/cdist dotman
Note that `dotman`-target has to be built before a `make docs`-run, otherwise
the custom man-pages are not picked up.
Python package
~~~~~~~~~~~~~~
Cdist is available as a python package at
`PyPi <http://pypi.python.org/pypi/cdist/>`_. You can install it using
.. code-block:: sh
pip install cdist
Installing from source with signature verification
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you want to install cdist from signed source and verify it, first you need to
download cdist archive and its detached signature.
Get both, *cdist-x.y.z.tar.gz* and *cdist-x.y.z.tar.gz.asc* from release
notes of the desired tag *x.y.z* at
`cdist git repository <https://code.ungleich.ch/ungleich-public/cdist/-/tags>`_.
Get GPG public key used for signing `here <_static/pgp-key-EFD2AE4EC36B6901.asc>`_
and import it into GPG.
Now cdist source archive can be verified using `gpg`, e.g. to verify `cdist-6.2.0`:
.. code-block:: sh
$ gpg --verify cdist-6.2.0.tar.gz.asc cdist-6.2.0.targ.gz
gpg: Signature made Sat Nov 30 23:14:19 2019 CET
gpg: using RSA key 69767822F3ECC3C349C1EFFFEFD2AE4EC36B6901
gpg: Good signature from "ungleich GmbH (ungleich FOSS) <foss@ungleich.ch>" [ultimate]
Further steps are the same as for `installing from git <cdist-install.html#from-git>`_.

View file

@ -0,0 +1,47 @@
cdist integration / using cdist as library
==========================================
Description
-----------
cdist can be integrate with other applications by importing cdist and other
cdist modules and setting all by hand. There are also helper functions which
aim to ease this integration. Just import **cdist.integration** and use its
functions:
* :strong:`cdist.integration.configure_hosts_simple` for configuration
* :strong:`cdist.integration.install_hosts_simple` for installation.
Functions require `host` and `manifest` parameters.
`host` can be specified as a string representing host or as iterable
of hosts. `manifest` is a path to initial manifest. For other cdist
options default values will be used. `verbose` is a desired verbosity
level which defaults to VERBOSE_INFO. `cdist_path` parameter specifies
path to cdist executable, if it is `None` then functions will try to
find it first from local lib directory and then in PATH.
In case of cdist error :strong:`cdist.Error` exception is raised.
:strong:`WARNING`: cdist integration helper functions are not yet stable!
Examples
--------
.. code-block:: sh
# configure host from python interactive shell
>>> import cdist.integration
>>> cdist.integration.configure_hosts_simple('185.203.114.185',
... '~/.cdist/manifest/init')
# configure host from python interactive shell, specifying verbosity level
>>> import cdist.integration
>>> cdist.integration.configure_hosts_simple(
... '185.203.114.185', '~/.cdist/manifest/init',
... verbose=cdist.argparse.VERBOSE_TRACE)
# configure specified dns hosts from python interactive shell
>>> import cdist.integration
>>> hosts = ('dns1.ungleich.ch', 'dns2.ungleich.ch', 'dns3.ungleich.ch', )
>>> cdist.integration.configure_hosts_simple(hosts,
... '~/.cdist/manifest/init')

View file

@ -0,0 +1,211 @@
Inventory
=========
Introduction
------------
cdist comes with simple built-in tag based inventory. It is a simple inventory
with list of hosts and a host has a list of tags.
Inventory functionality is still in **beta** so it can be used only if beta
command line flag is specified (-b, --beta) or setting CDIST_BETA env var.
Description
-----------
The idea is to have simple tagging inventory. There is a list of hosts and for
each host there are tags. Inventory database is a set of files under inventory
database base directory. Filename equals hostname. Each file contains tags for
hostname with each tag on its own line.
Using inventory you can now configure hosts by selecting them by tags.
Tags have no values, as tags are just tags. Tag name-value would in this
context mean that host has two tags and it is selected by specifying that both
tags are present.
This inventory is **KISS** cdist built-in inventory database. You can maintain it
using cdist inventory interface or using standard UNIX tools.
cdist inventory interface
-------------------------
With cdist inventory interface you can list host(s) and tag(s), add host(s),
add tag(s), delete host(s) and delete tag(s).
Configuring hosts using inventory
---------------------------------
config command now has new options, **-t**, **-a** and **-A**.
**-A** means that all hosts in tag db is selected.
**-a** means that selected hosts must contain ALL specified tags.
**-t** means that host specifies tag - all hosts that have specified tags are
selected.
Examples
--------
.. code-block:: sh
# List inventory content
$ cdist inventory list -b
# List inventory for specified host localhost
$ cdist inventory list -b localhost
# List inventory for specified tag loadbalancer
$ cdist inventory list -b -t loadbalancer
# Add hosts to inventory
$ cdist inventory add-host -b web1 web2 web3
# Delete hosts from file old-hosts from inventory
$ cdist inventory del-host -b -f old-hosts
# Add tags to specified hosts
$ cdist inventory add-tag -b -t europe,croatia,web,static web1 web2
# Add tag to all hosts in inventory
$ cdist inventory add-tag -b -t vm
# Delete all tags from specified host
$ cdist inventory del-tag -b -a localhost
# Delete tags read from stdin from hosts specified by file hosts
$ cdist inventory del-tag -b -T - -f hosts
# Configure hosts from inventory with any of specified tags
$ cdist config -b -t web dynamic
# Configure hosts from inventory with all specified tags
$ cdist config -b -t -a web dynamic
# Configure all hosts from inventory db
$ cdist config -b -A
Example of manipulating database
--------------------------------
.. code-block:: sh
$ python3 scripts/cdist inventory list -b
$ python3 scripts/cdist inventory add-host -b localhost
$ python3 scripts/cdist inventory add-host -b test.mycloud.net
$ python3 scripts/cdist inventory list -b
localhost
test.mycloud.net
$ python3 scripts/cdist inventory add-host -b web1.mycloud.net web2.mycloud.net shell1.mycloud.net shell2.mycloud.net
$ python3 scripts/cdist inventory list -b
localhost
test.mycloud.net
web1.mycloud.net
web2.mycloud.net
shell1.mycloud.net
shell2.mycloud.net
$ python3 scripts/cdist inventory add-tag -b -t web web1.mycloud.net web2.mycloud.net
$ python3 scripts/cdist inventory add-tag -b -t shell shell1.mycloud.net shell2.mycloud.net
$ python3 scripts/cdist inventory add-tag -b -t cloud
$ python3 scripts/cdist inventory list -b
localhost cloud
test.mycloud.net cloud
web1.mycloud.net cloud,web
web2.mycloud.net cloud,web
shell1.mycloud.net cloud,shell
shell2.mycloud.net cloud,shell
$ python3 scripts/cdist inventory add-tag -b -t test,web,shell test.mycloud.net
$ python3 scripts/cdist inventory list -b
localhost cloud
test.mycloud.net cloud,shell,test,web
web1.mycloud.net cloud,web
web2.mycloud.net cloud,web
shell1.mycloud.net cloud,shell
shell2.mycloud.net cloud,shell
$ python3 scripts/cdist inventory del-tag -b -t shell test.mycloud.net
$ python3 scripts/cdist inventory list -b
localhost cloud
test.mycloud.net cloud,test,web
web1.mycloud.net cloud,web
web2.mycloud.net cloud,web
shell1.mycloud.net cloud,shell
shell2.mycloud.net cloud,shell
$ python3 scripts/cdist inventory add-tag -b -t all
$ python3 scripts/cdist inventory add-tag -b -t mistake
$ python3 scripts/cdist inventory list -b
localhost all,cloud,mistake
test.mycloud.net all,cloud,mistake,test,web
web1.mycloud.net all,cloud,mistake,web
web2.mycloud.net all,cloud,mistake,web
shell1.mycloud.net all,cloud,mistake,shell
shell2.mycloud.net all,cloud,mistake,shell
$ python3 scripts/cdist inventory del-tag -b -t mistake
$ python3 scripts/cdist inventory list -b
localhost all,cloud
test.mycloud.net all,cloud,test,web
web1.mycloud.net all,cloud,web
web2.mycloud.net all,cloud,web
shell1.mycloud.net all,cloud,shell
shell2.mycloud.net all,cloud,shell
$ python3 scripts/cdist inventory del-host -b localhost
$ python3 scripts/cdist inventory list -b
test.mycloud.net all,cloud,test,web
web1.mycloud.net all,cloud,web
web2.mycloud.net all,cloud,web
shell1.mycloud.net all,cloud,shell
shell2.mycloud.net all,cloud,shell
$ python3 scripts/cdist inventory list -b -t web
test.mycloud.net all,cloud,test,web
web1.mycloud.net all,cloud,web
web2.mycloud.net all,cloud,web
$ python3 scripts/cdist inventory list -b -t -a web test
test.mycloud.net all,cloud,test,web
$ python3 scripts/cdist inventory list -b -t -a web all
test.mycloud.net all,cloud,test,web
web1.mycloud.net all,cloud,web
web2.mycloud.net all,cloud,web
$ python3 scripts/cdist inventory list -b -t web all
test.mycloud.net all,cloud,test,web
web1.mycloud.net all,cloud,web
web2.mycloud.net all,cloud,web
shell1.mycloud.net all,cloud,shell
shell2.mycloud.net all,cloud,shell
$ cd cdist/inventory
$ ls -1
shell1.mycloud.net
shell2.mycloud.net
test.mycloud.net
web1.mycloud.net
web2.mycloud.net
$ ls -l
total 20
-rw-r--r-- 1 darko darko 16 Jun 24 12:43 shell1.mycloud.net
-rw-r--r-- 1 darko darko 16 Jun 24 12:43 shell2.mycloud.net
-rw-r--r-- 1 darko darko 19 Jun 24 12:43 test.mycloud.net
-rw-r--r-- 1 darko darko 14 Jun 24 12:43 web1.mycloud.net
-rw-r--r-- 1 darko darko 14 Jun 24 12:43 web2.mycloud.net
$ cat test.mycloud.net
test
all
web
cloud
$ cat web2.mycloud.net
all
web
cloud
For more info about inventory commands and options see `cdist <man1/cdist.html>`_\ (1).
Using external inventory
------------------------
cdist can be used with any external inventory where external inventory is
some storage or database from which you can get a list of hosts to configure.
cdist can then be fed with this list of hosts through stdin or file using
**-f** option. For example, if your host list is stored in sqlite3 database
hosts.db and you want to select hosts which purpose is **django** then you
can use it with cdist like:
.. code-block:: sh
$ sqlite3 hosts.db "select hostname from hosts where purpose = 'django';" | cdist config

View file

@ -0,0 +1,376 @@
Manifest
========
Description
-----------
Manifests are used to define which objects to create.
Objects are instances of **types**, like in object oriented programming languages.
An object is represented by the combination of
**type + slash + object name**: **\__file/etc/cdist-configured** is an
object of the type **__file** with the name **etc/cdist-configured**.
All available types can be found in the **cdist/conf/type/** directory,
use **ls cdist/conf/type** to get the list of available types. If you have
setup the MANPATH correctly, you can use **man cdist-reference** to access
the reference with pointers to the manpages.
Types in manifests are used like normal command line tools. Let's have a look
at an example::
# Create object of type __package with the parameter state = absent
__package apache2 --state absent
# Same with the __directory type
__directory /tmp/cdist --state present
These two lines create objects, which will later be used to realise the
configuration on the target host.
Manifests are executed locally as a shell script using **/bin/sh -e**.
The resulting objects are stored in an internal database.
The same object can be redefined in multiple different manifests as long as
the parameters are exactly the same.
In general, manifests are used to define which types are used depending
on given conditions.
Initial and type manifests
--------------------------
Cdist knows about two types of manifests: The initial manifest and type
manifests. The initial manifest is used to define, which configurations
to apply to which hosts. The type manifests are used to create objects
from types. More about manifests in types can be found in `cdist type <cdist-type.html>`_.
Define state in the initial manifest
------------------------------------
The **initial manifest** is the entry point for cdist to find out, which
**objects** to configure on the selected host.
Cdist expects the initial manifest at **cdist/conf/manifest/init**.
Within this initial manifest you define which objects should be
created on which host. To distinguish between hosts, you can use the
environment variable **__target_host** and/or **__target_hostname** and/or
**__target_fqdn**. Let's have a look at a simple example::
__cdistmarker
case "$__target_host" in
localhost)
__directory /home/services/kvm-vm --parents yes
;;
esac
This manifest says: Independent of the host, always use the type
**__cdistmarker**, which creates the file **/etc/cdist-configured**,
with the timestamp as content.
The directory **/home/services/kvm-vm**, including all parent directories,
is only created on the host **localhost**.
As you can see, there is no magic involved, the manifest is simple shell code that
utilises cdist types. Every available type can be executed like a normal
command.
Splitting up the initial manifest
---------------------------------
If you want to split up your initial manifest, you can create other shell
scripts in **cdist/conf/manifest/** and include them in **cdist/conf/manifest/init**.
Cdist provides the environment variable **__manifest** to reference
the directory containing the initial manifest (see `cdist reference <cdist-reference.html>`_).
The following example would include every file with a **.sh** suffix::
# Include *.sh
for manifest in $__manifest/*.sh; do
# And source scripts into our shell environment
. "$manifest"
done
Dependencies
------------
If you want to describe that something requires something else, just
setup the variable "require" to contain the requirements. Multiple
requirements can be added white space separated.
::
1 # No dependency
2 __file /etc/cdist-configured
3
4 # Require above object
5 require="__file/etc/cdist-configured" __link /tmp/cdist-testfile \
6 --source /etc/cdist-configured --type symbolic
7
8 # Require two objects
9 require="__file/etc/cdist-configured __link/tmp/cdist-testfile" \
10 __file /tmp/cdist-another-testfile
Above the "require" variable is only set for the command that is
immediately following it. Dependencies should always be declared that way.
On line 4 you can see that the instantiation of a type "\__link" object needs
the object "__file/etc/cdist-configured" to be present, before it can proceed.
This also means that the "\__link" command must make sure, that either
"\__file/etc/cdist-configured" already is present, or, if it's not, it needs
to be created. The task of cdist is to make sure, that the dependency will be
resolved appropriately and thus "\__file/etc/cdist-configured" be created
if necessary before "__link" proceeds (or to abort execution with an error).
If you really need to make all types depend on a common dependency, you can
export the "require" variable as well. But then, if you need to add extra
dependencies to a specific type, you have to make sure that you append these
to the globally already defined one.
::
# First of all, update the package index
__package_update_index
# Upgrade all the installed packages afterwards
require="__package_update_index" __package_upgrade_all
# Create a common dependency for all the next types so that they get to
# be executed only after the package upgrade has finished
export require="__package_upgrade_all"
# Ensure that lighttpd is installed after we have upgraded all the packages
__package lighttpd --state present
# Ensure that munin is installed after lighttpd is present and after all
# the packages are upgraded
require="$require __package/lighttpd" __package munin --state present
All objects that are created in a type manifest are automatically required
from the type that is calling them. This is called "autorequirement" in
cdist jargon.
You can find a more in depth description of the flow execution of manifests
in `cdist execution stages <cdist-stages.html>`_ and of how types work in `cdist type <cdist-type.html>`_.
Create dependencies from execution order
-----------------------------------------
You can tell cdist to execute all types in the order in which they are created
in the manifest by setting up the variable CDIST_ORDER_DEPENDENCY.
When cdist sees that this variable is setup, the current created object
automatically depends on the previously created object.
It essentially helps you to build up blocks of code that build upon each other
(like first creating the directory xyz than the file below the directory).
Read also about `notes on CDIST_ORDER_DEPENDENCY <cdist-best-practice.html#notes-on-cdist-order-dependency>`_.
In version 6.2.0 semantic CDIST_ORDER_DEPENDENCY is finally fixed and well defined.
CDIST_ORDER_DEPENDENCY defines type order dependency context. Order dependency context
starts when CDIST_ORDER_DEPENDENCY is set, and ends when it is unset. After each
manifest execution finishes, any existing order dependency context is automatically
unset. This ensures that CDIST_ORDER_DEPENDENCY is valid within the manifest where it
is used. When order dependency context is defined then cdist executes types in the
order in which they are created in the manifest inside order dependency context.
Sometimes the best way to see how something works is to see examples.
Suppose you have defined **initial manifest**:
.. code-block:: sh
__cycle1 cycle1
export CDIST_ORDER_DEPENDENCY=1
__cycle2 cycle2
__cycle3 cycle3
with types **__cycle1**:
.. code-block:: sh
export CDIST_ORDER_DEPENDENCY=1
__file /tmp/cycle11
__file /tmp/cycle12
__file /tmp/cycle13
**__cycle2**:
.. code-block:: sh
__file /tmp/cycle21
export CDIST_ORDER_DEPENDENCY=1
__file /tmp/cycle22
__file /tmp/cycle23
unset CDIST_ORDER_DEPENDENCY
__file /tmp/cycle24
**__cycle3**:
.. code-block:: sh
__file /tmp/cycle31
__file /tmp/cycle32
export CDIST_ORDER_DEPENDENCY=1
__file /tmp/cycle33
__file /tmp/cycle34
For the above config, cdist results in the following expected *dependency graph*
(type *__cycleX* is shown as *cX*, *__file/tmp/cycleXY* is shown as *fcXY*):
::
c1---->fc11
| /\
| |
+----->fc12
| /\
| |
+----->fc13
c2--+--->fc21
/\ |
| |
| +----->fc22
| | /\
| | |
| +----->fc23
| |
| |
| +----->fc24
|
|
c3---->fc31
|
|
+----->fc32
|
|
+----->fc33
| /\
| |
+----->fc34
Before version 6.2.0 the above configuration would result in cycle:
::
ERROR: 185.203.112.26: Cycle detected in object dependencies:
__file/tmp/cycle11 -> __cycle3/cycle3 -> __cycle2/cycle2 -> __cycle1/cycle1 -> __file/tmp/cycle11!
The following manifest shows an example for order dependency contexts:
.. code-block:: sh
__file /tmp/fileA
export CDIST_ORDER_DEPENDENCY=1
__file /tmp/fileB
__file /tmp/fileC
__file /tmp/fileD
unset CDIST_ORDER_DEPENDENCY
__file /tmp/fileE
__file /tmp/fileF
export CDIST_ORDER_DEPENDENCY=1
__file /tmp/fileG
__file /tmp/fileH
unset CDIST_ORDER_DEPENDENCY
__file /tmp/fileI
This means:
* C depends on B
* D depends on C
* H depends on G
and there are no other dependencies from this manifest.
Overrides
---------
In some special cases, you would like to create an already defined object
with different parameters. In normal situations this leads to an error in cdist.
If you wish, you can setup the environment variable CDIST_OVERRIDE
(any value or even empty is ok) to tell cdist, that this object override is
wanted and should be accepted.
ATTENTION: Only use this feature if you are 100% sure in which order
cdist encounters the affected objects, otherwise this results
in an undefined situation.
If CDIST_OVERRIDE and CDIST_ORDER_DEPENDENCY are set for an object,
CDIST_ORDER_DEPENDENCY will be ignored, because adding a dependency in case of
overrides would result in circular dependencies, which is an error.
Examples
--------
The initial manifest may for instance contain the following code:
.. code-block:: sh
# Always create this file, so other sysadmins know cdist is used.
__file /etc/cdist-configured
case "$__target_host" in
my.server.name)
__directory /root/bin/
__file /etc/issue.net --source "$__manifest/issue.net
;;
esac
The manifest of the type "nologin" may look like this:
.. code-block:: sh
__file /etc/nologin --source "$__type/files/default.nologin"
This example makes use of dependencies:
.. code-block:: sh
# Ensure that lighttpd is installed
__package lighttpd --state present
# Ensure that munin makes use of lighttpd instead of the default webserver
# package as decided by the package manager
require="__package/lighttpd" __package munin --state present
How to override objects:
.. code-block:: sh
# for example in the initial manifest
# create user account foobar with some hash for password
__user foobar --password 'some_fancy_hash' --home /home/foobarexample
# ... many statements and includes in the manifest later ...
# somewhere in a conditionally sourced manifest
# (e.g. for example only sourced if a special application is on the target host)
# this leads to an error ...
__user foobar --password 'some_other_hash'
# this tells cdist, that you know that this is an override and should be accepted
CDIST_OVERRIDE=yes __user foobar --password 'some_other_hash'
# it's only an override, means the parameter --home is not touched
# and stays at the original value of /home/foobarexample
Dependencies defined by execution order work as following:
.. code-block:: sh
# Tells cdist to execute all types in the order in which they are created ...
export CDIST_ORDER_DEPENDENCY=on
__sample_type 1
require="__some_type_somewhere/id" __sample_type 2
__example_type 23
# Now this types are executed in the creation order until the variable is unset
unset CDIST_ORDER_DEPENDENCY
# all now following types cdist makes the order ..
__not_in_order_type 42
# how it works :
# this lines above are translated to:
__sample_type 1
require="__some_type_somewhere/id __sample_type/1" __sample_type 2
require="__sample_type/2" __example_type 23
__not_in_order_type 42

View file

@ -0,0 +1,94 @@
Messaging
=========
Description
-----------
cdist has a simple but powerful way of allowing communication between
the initial manifest and types as well as types and types.
Whenever execution is passed from cdist to one of the
scripts described below, cdist generate 2 new temporary files
and exports the environment variables **__messages_in** and
**__messages_out** to point to them.
Before handing over the control, the content of the global message
file is copied into the file referenced by **$__messages_in**.
After cdist gained control back, the content of the file referenced
by **$__messages_out** is appended to the global message file.
This way overwriting any of the two files by accident does not
interfere with other types.
The order of execution is not defined unless you create dependencies
between the different objects (see `cdist manifest <cdist-manifest.html>`_) and thus you
can only react reliably on messages by objects that you depend on.
Availability
------------
Messaging is possible between all **local** scripts:
- initial manifest
- type/manifest
- type/gencode-local
- type/gencode-remote
Examples
--------
When you want to emit a message use:
.. code-block:: sh
echo "something" >> "$__messages_out"
When you want to react on a message use:
.. code-block:: sh
if grep -q "^__your_type/object/id:something" "$__messages_in"; then
echo "I do something else"
fi
Some real life examples:
.. code-block:: sh
# Reacting on changes from block for keepalive
if grep -q "^__block/keepalive-vrrp" "$__messages_in"; then
echo /etc/init.d/keepalived restart
fi
# Reacting on changes of configuration files
if grep -q "^__file/etc/one" $__messages_in; then
echo 'for init in /etc/init.d/opennebula*; do $init restart; done'
fi
Restart sshd on changes
.. code-block:: sh
os="$(cat "$__global/explorer/os")"
case "$os" in
centos|redhat|suse)
restart="/etc/init.d/sshd restart"
;;
debian|ubuntu)
restart="/etc/init.d/ssh restart"
;;
*)
cat << eof >&2
Unsupported os $os.
If you would like to have this type running on $os,
you can either develop the changes and send a pull
request or ask for a quote at www.ungleich.ch
eof
exit 1
;;
esac
if grep -q "^__key_value/PermitRootLogin" "$__messages_in"; then
echo $restart
fi

View file

@ -0,0 +1,19 @@
Supported operating systems
===========================
cdist was tested or is know to run on at least
* `Alpine Linux <https://alpinelinux.org>`_
* `Archlinux <http://www.archlinux.org>`_
* `CentOS <http://www.centos.org>`_