tom/cdist
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Merge branch 'master' into install_integration

Browse source
This commit is contained in:
Nico Schottelius 2014-06-13 13:34:31 +02:00
commit 9bf2d58a8f
131 changed files with 1636 additions and 272 deletions
Download patch file Download diff file Expand all files Collapse all files

65
docs/changelog
View file

@ -5,8 +5,71 @@ Changelog
* Exception: No braces means author == Nico Schottelius
3.0.8:
3.1.6:
* Type __package_yum: Support retrieving package via URL
* Type __hostname: Support SuSE and have CentOS use sysconfig value
* Type __locale: Support SuSE
* Type __timezone: Support SuSE
* Type __file: Support MacOS X (Manuel Hutter)
3.1.5: 2014-05-05
* Type __zypper_repo: Automatically import gpg keys (Daniel Heule)
* Type __zypper_service: Automatically import gpg keys (Daniel Heule)
3.1.4: 2014-05-04
* Core: Ensure all created files end in \n (Steven Armstrong)
* Documentation: Cleanup up, added HTML links (Tomas Pospisek)
* Explorer interfaces: Remove test output (Daniel Heule)
* Type __jail: Add messaging support (Jake Guffey)
3.1.3: 2014-04-29
* New Type: __yum_repo (Steven Armstrong)
* Type __hostname: Add support for CentOS
3.1.2: 2014-04-12
* Documentation: Add missing environment variables to reference
* Type __qemu_img: size is optional, if state is not present
* Type __key_value: Rewrite using awk (Daniel Heule)
* New Type: __dog_vdi
3.1.1: 2014-03-26
* Core: Make __object and __object_id available to code (Daniel Heule)
* New explorer: cpu_cores (Daniel Heule/Thomas Oettli)
* New explorer: cpu_sockets (Daniel Heule/Thomas Oettli)
* New explorer: machine_type (Daniel Heule/Thomas Oettli)
* New explorer: memory (Daniel Heule/Thomas Oettli)
* Type __jail: Fix parameter names in explorer (Jake Guffey)
* Type __line: Ensure permissions are kept (Steven Armstrong)
* Type __link: Do not create link in directory, if link exists (Steven Armstrong)
* Type __package_pkg_openbsd: Improve error handling (og)
3.1.0: 2014-03-19
* New Type: __rbenv
* Type __file: Enhance OpenBSD Support (og)
* Type __git: Pass onwer/group/mode values to __directory
* Type __iptable_rule: Fix example documentation (Antoine Catton)
* Type __key_value: Add messaging support
* Type __package_pkg_openbsd: Allow to change PKG_PATH (og)
* Type __ssh_authorized_keys: Allow managing existing keys (Steven Armstrong)
* Type __user: Enhance OpenBSD Support (og)
3.0.9: 2014-02-14
* Core: Ignore order dependencies if override is set (Daniel Heule)
* Core: Improve Mac OS X support for unit tests (Steven Armstrong)
* Type __locale: Error out in case of unsupported OS
* Type __jail: Use default parameters for state (Daniel Heule)
* Type __pf_ruleset: Use default parameters for state (Daniel Heule)
* Type __postgres_database: Use default parameters for state (Daniel Heule)
* Type __postgres_role: Use default parameters for state (Daniel Heule)
* Type __rvm: Use default parameters for state (Daniel Heule)
* Type __rvm_gem: Use default parameters for state (Daniel Heule)
* Type __rvm_gemset: Use default parameters for state (Daniel Heule)
* Type __rvm_ruby: Use default parameters for state (Daniel Heule)
3.0.8: 2014-02-11
* Core: Enhance object id verification (Daniel Heule)
* Core: Add unit tests for dependencies based on execution order (Daniel Heule)
* Core: Add unit tests for dry run (Daniel Heule)
3.0.7: 2014-02-08
* Core: Allow dependencies to be created based execution order (Daniel Heule)

10
docs/dev/fancy-ideas
View file

@ -4,4 +4,14 @@
implemented as a proof of concept at:
https://github.com/asteven/cdist/tree/type-namespaces
Execute all global explorers only when needed #286
My intention is to create a brunch of global explorer which are of use in some cases and makes cdist more userfriendly. But now, all global explorers are allways executed, even the return value of the explorers is never used.
I think a possible approach can be to replace the result files with pipes, and on first read of the pipe, the explorer is executed by the core, all following read calls from the pipe are answered from the core with the result of the first real execute of the explorer.
So cdist can have an unlimited number of global explorers and only used explorers are executed on the target host, all other explorers laying around are simply ignored.
Also a possible approach would be to create a new explorer type (dynamic explorers) which are sitting in a different directory to (for example dynexploer) and only this ones are executed with the conditional approach explained above. So the overhead to create pipes and monitor it is only in place on explorers which are not interesting for everyone ...

27
docs/dev/logs/2013-01-03.dependency-issue Normal file
View file

@ -0,0 +1,27 @@
Problem shown by using __rbenv:
__rbenv/nicotest
__git /home/nico/.rbenv
__package git
__directory /home/nico/.rbenv
require="__git/home/nico/.rbenv"
__git /home/nico/.rbenv/plugins/ruby-build
__package git
__directory /home/nico/.rbenv/plugins/ruby-build
1) if children do NOT automatically depend on their parents requiremnts
__directory /home/nico/.rbenv/plugins/ruby-build fails:
because __directory /home/nico/.rbenv/plugins is created by
__git /home/nico/.rbenv, but __directory /home/nico/.rbenv/plugins/ruby-build
does not depend on __git /home/nico/.rbenv
2) if children DO automatically depend on their parents requiremnts
__package git from __git /home/nico/.rbenv/plugins/ruby-build depends on __git /home/nico/.rbenv.
__git /home/nico/.rbenv depends on __package git (via autorequire)
=> circular dependency, they depend on each other

86
docs/dev/logs/2014-02-13.discussion Normal file
View file

@ -0,0 +1,86 @@
With Steven
t marker .cdist breaks
- use random marker that starts with .cdist-
- has fixed number of following characters (like 6 or 10)
- write marker name to $__global/marker
- export $__global/marker path as $__marker
- document variable in cdist-reference
- also document the pattern how the marker is built
so that other people may be able to dig into the structure
from outside
t save method
- in $__global/method
- values
- config
- install
- document path and description in cdist-reference
t save whole runtime in cache
- missing items
- initial manifest may be specified on commandline
- always save the initial manifest to $__global/initial-manifest
- currently it is a lost tempfile
- remote exec / remote copy
- save to $__global/remote_exec
- save to $__global/remote_copy
- stdout and stderr of everything
- need to implement Steven's patch of stderr/stdout capturing
- exit code of cdist
- if it is complete, we can use it for replay / reconfigure
- new idea: replay / reconfig / reinstall
- --from-cache?
t stderr/stdout
- capture all messages
- prefix with target_host
- implementation exists in one of Steven's branches
- ping steven for updated pull request
x on error dump all information about the failing object
- where created
- stderr
- stdout
- parameter (+values)
- everything known [tm]
t multiple versions of cache
- see #298
t absolute path of types, explorer
- resolve instead of using the temporary link name
- #305
t report command
- from cache?
- #306
t add session to "run directories"
- instead of /var/lib/cdist (remote)
- instead of static dir in cache
- same id remote and local
- maybe timestamp
- in or excluding the pid of cdist?
- cache
- also save when cdist fails
- save exit code
- be able to restore config
- new command: cdist clean-cache
- --since
- --keep-versions --keep-lala $num
- cdist 4.0.0pre2
- cleanup in preos
- logging for types
cdist log ...?
- cdist logserver
- $__global/log.socket
- fifo?
echo into logpipe?

18
docs/dev/logs/2014-02-18.unauthenticated-packages Normal file
View file

@ -0,0 +1,18 @@
- we cannot install packages, which are not authenticated:
INFO: voicerepublic-staging.sky.ungleich.ch: Executing code for __package_apt/deb-multimedia-keyring
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
deb-multimedia-keyring
0 upgraded, 1 newly installed, 0 to remove and 13 not upgraded.
Need to get 14.4 kB of archives.
After this operation, 46.1 kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
deb-multimedia-keyring
E: There are problems and -y was used without --force-yes
ERROR: voicerepublic-staging.sky.ungleich.ch: Command failed: ssh -o User=root -q voicerepublic-staging.sky.ungleich.ch /bin/sh -e /var/lib/cdist/object/__package_apt/deb-multimedia-keyring/.cdist/code-remote
INFO: cdist: Total processing time for 1 host(s): 72.07943892478943
ERROR: cdist: Failed to configure the following hosts: voicerepublic-staging.sky.ungleich.ch

15
docs/man/cdist-reference.text.sh
View file

@ -1,6 +1,7 @@
#!/bin/sh
#
# 2010-2013 Nico Schottelius (nico-cdist at schottelius.org)
# 2010-2014 Nico Schottelius (nico-cdist at schottelius.org)
# 2014 Daniel Heule (hda at sfs.biz)
#
# This file is part of cdist.
#
@ -209,10 +210,10 @@ __messages_out::
Available for: initial manifest, type manifest, type gencode
__object::
Directory that contains the current object.
Available for: type manifest, type explorer, type gencode
Available for: type manifest, type explorer, type gencode and code scripts
__object_id::
The type unique object id.
Available for: type manifest, type explorer, type gencode
Available for: type manifest, type explorer, type gencode and code scripts
Note: The leading and the trailing "/" will always be stripped (caused by
the filesystem database and ensured by the core).
Note: Double slashes ("//") will not be fixed and result in an error.
@ -236,7 +237,13 @@ The following environment variables influence the behaviour of cdist:
require::
Setup dependencies between objects (see cdist-manifest(7))
CDIST_ALLOW_OVERRIDE::
CDIST_LOCAL_SHELL::
Use this shell locally instead of /bin/sh to execute scripts
CDIST_REMOTE_SHELL::
Use this shell remotely instead of /bin/sh to execute scripts
CDIST_OVERRIDE::
Allow overwriting type parameters (see cdist-manifest(7))
CDIST_ORDER_DEPENDENCY::

8
docs/man/man1/cdist.text
View file

@ -26,7 +26,7 @@ cdist supports different subcommands as explained below.
GENERAL
-------
All commands except the following options:
All commands accept the following options:
-d, --debug::
Set log level to debug
@ -34,7 +34,7 @@ All commands except the following options:
-h, --help::
Show the help screen
-v, --verbose:
-v, --verbose::
Set log level to info, be more verbose
-V, --version::
@ -72,10 +72,10 @@ Configure one or more hosts
-s, --sequential::
Operate on multiple hosts sequentially
--remote-copy REMOTE_COPY:
--remote-copy REMOTE_COPY::
Command to use for remote copy (should behave like scp)
--remote-exec REMOTE_EXEC:
--remote-exec REMOTE_EXEC::
Command to use for remote execution (should behave like ssh)
SHELL

2
docs/man/man7/cdist-bootstrap.text
View file

@ -25,7 +25,7 @@ location.
For starters, having cdist (which includes the configuration database) on
your notebook should be fine.
Additionally an external copy of the git repository the configuration
relies in is recommended, for use as backup as well to allow easy collaboration
relies on is recommended, for use as backup as well as to allow easy collaboration
with others.
For more sophisticated setups developing cdist configurations with multiple

57
docs/man/man7/cdist-manifest.text
View file

@ -13,7 +13,7 @@ DESCRIPTION
Manifests are used to define which objects to create.
Objects are instances of **types**, like in object oriented programming languages.
An object is represented by the combination of
**type + slash + object name**: **__file/etc/cdist-configured** is an
**type + slash + object name**: **\__file/etc/cdist-configured** is an
object of the type ***__file*** with the name ***etc/cdist-configured***.
All available types can be found in the **cdist/conf/type/** directory,
@ -29,7 +29,7 @@ at an example:
__package apache2 --state absent
# Same with the __directory type
__directory /tmp/cdist --state present
__directory /tmp/cdist --state present
--------------------------------------------------------------------------------
These two lines create objects, which will later be used to realise the
@ -89,7 +89,7 @@ SPLITTING UP THE INITIAL MANIFEST
---------------------------------
If you want to split up your initial manifest, you can create other shell
scripts in **cdist/conf/manifest/** and include them in **cdist/conf/manifest/init**.
Cdist provides the environment variable ***__manifest*** to reference to
Cdist provides the environment variable ***__manifest*** to reference
the directory containing the initial manifest (see cdist-reference(7)).
The following example would include every file with a **.sh** suffix:
@ -110,24 +110,39 @@ setup the variable "require" to contain the requirements. Multiple
requirements can be added white space separated.
--------------------------------------------------------------------------------
# No dependency
__file /etc/cdist-configured
# Require above object
require="__file/etc/cdist-configured" __link /tmp/cdist-testfile \
--source /etc/cdist-configured --type symbolic
# Require two objects
require="__file/etc/cdist-configured __link/tmp/cdist-testfile" \
__file /tmp/cdist-another-testfile
1 # No dependency
2 __file /etc/cdist-configured
3
4 # Require above object
5 require="__file/etc/cdist-configured" __link /tmp/cdist-testfile \
6 --source /etc/cdist-configured --type symbolic
7
8 # Require two objects
9 require="__file/etc/cdist-configured __link/tmp/cdist-testfile" \
10 __file /tmp/cdist-another-testfile
--------------------------------------------------------------------------------
Above the "require" variable is only set for the command that is
immediately following it. Dependencies should allways be declared that way.
On line 4 you can see that the instantion of a type "__link" object needs
the object "__file/etc/cdist-configured" to be present, before it can proceed.
This also means that the "__link" command must make sure, that either
"__file/etc/cdist-configured" allready is present, or, if it's not, it needs
to be created. The task of cdist is to make sure, that the dependency will be
resolved appropriately and thus "__file/etc/cdist-configured" be created
if necessary before "__link" proceeds (or to abort execution with an error).
All objects that are created in a type manifest are automatically required
from the type that is calling them. This is called "autorequirement" in
cdist jargon.
You can find an more in depth description of the flow execution of manifests
in cdist-stages(7) and of how types work in cdist-type(7).
CREATE DEPENDENCIES FROM EXECUTION ORDER
-----------------------------------------
You can tell cdist to execute all types in the order in which they are created
@ -138,8 +153,6 @@ automatically depends on the previously created object.
It essentially helps you to build up blocks of code that build upon each other
(like first creating the directory xyz than the file below the directory).
THIS IS A BETA FEATURE AND MAY BE REMOVED OR CHANGED AT ANY TIME.
OVERRIDES
---------
@ -149,10 +162,12 @@ If you whish, you can setup the environment variable CDIST_OVERRIDE
(any value or even empty is ok) to tell cdist, that this object override is
wanted and should be accepted.
ATTENTION: Only use this feature if you are 100% sure in which order
cdist encounter the affected objects, otherwhise this results
into an undefined situation.
cdist encounters the affected objects, otherwhise this results
in an undefined situation.
THIS IS A BETA FEATURE AND MAY BE REMOVED OR CHANGED AT ANY TIME.
If CDIST_OVERRIDE and CDIST_ORDER_DEPENDENCY are set for an object,
CDIST_ORDER_DEPENDENCY will be ignored, because adding a dependency in case of
overrides would result in circular dependencies, which is an error.
@ -194,7 +209,7 @@ How to override objects:
--------------------------------------------------------------------------------
# for example in the inital manifest
# reate user account foobar with some hash for password
# create user account foobar with some hash for password
__user foobar --password 'some_fancy_hash' --home /home/foobarexample
# ... many statements and includes in the manifest later ...
@ -206,8 +221,8 @@ __user foobar --password 'some_other_hash'
# this tells cdist, that you know that this is an override and should be accepted
CDIST_OVERRIDE=yes __user foobar --password 'some_other_hash'
# its only an override, means the parameter --home is not touched
# and stay at the original value of /home/foobarexample
# it's only an override, means the parameter --home is not touched
# and stays at the original value of /home/foobarexample
--------------------------------------------------------------------------------
Dependencies defined by execution order work as following:

23
docs/man/man7/cdist-type.text
View file

@ -25,7 +25,7 @@ to use.
HOW TO USE A TYPE
-----------------
You can use types from the initial manifest or the type manifest like a
normal command:
normal shell command:
--------------------------------------------------------------------------------
# Creates empty file /etc/cdist-configured
@ -252,6 +252,27 @@ echo "touch /etc/cdist-configured"
--------------------------------------------------------------------------------
VARIABLE ACCESS FROM THE GENERATED SCRIPTS
------------------------------------------
In the generated scripts, you have access to the following cdist variables
- __object
- __object_id
but only for read operations, means there is no back copy of this
files after the script execution.
So when you generate a script with the following content, it will work:
--------------------------------------------------------------------------------
if [ -f "$__object/parameter/name" ]; then
name="$(cat "$__object/parameter/name")"
else
name="$__object_id"
fi
--------------------------------------------------------------------------------
HINTS FOR TYPEWRITERS
----------------------
It must be assumed that the target is pretty dumb and thus does not have high

BIN
docs/speeches/2014-05-08_linuxtag_berlin.odp Normal file
View file

Binary file not shown.

BIN
docs/speeches/2014-05-08_linuxtag_berlin.pdf Normal file
View file

Binary file not shown.

BIN
docs/speeches/2014-05-19_cdi.st-zkb_linux_erfa.odp Normal file
View file

Binary file not shown.

BIN
docs/speeches/2014-05-19_cdi.st-zkb_linux_erfa.pdf Normal file
View file

Binary file not shown.

5
docs/web/cdist/update.mdwn
View file

@ -55,6 +55,11 @@ To upgrade to the lastet version do
## General Update Instructions
### Updating from 3.0 to 3.1
The type **\_\_ssh_authorized_keys** now also manages existing keys,
not only the ones added by cdist.
### Updating from 2.3 to 3.0
The **changed** attribute of objects has been removed.