stephan/cdist
1
0
Fork 0
forked from ungleich-public/cdist
Code Issues Pull requests Projects Releases Wiki Activity

Merge branch 'master' into notifications

Browse source
This commit is contained in:
Nico Schottelius 2013-11-25 22:54:36 +01:00
commit d00947711d
250 changed files with 4447 additions and 1653 deletions
Download patch file Download diff file Expand all files Collapse all files

79
docs/changelog
View file

@ -4,9 +4,84 @@ Changelog
* Changes are always commented with their author in (braces)
* Exception: No braces means author == Nico Schottelius
next:
* Type __jail: State absent should implies stopped (Jake Guffey)
2.3.6: 2013-11-25
* New Type: __locale
* Type __line: Ensure special characters are not interpreted
2.3.5: 2013-10-10
* Core: Unit test fix for remote_copy (Steven Armstrong)
* Documentation: Updated manpages of __package and __file (Alex Greif)
* Documentation: Add more examples to cdist-manifest (Dan Levin)
* Type __package_apt: Do not install recommends by default
2.3.4: 2013-10-03
* Core: Add missing bits to support dry run (Steven Armstrong)
* Core: Make unit test remote copy more compatible with scp (Steven Armstrong)
* New Type: __postfix (Steven Armstrong)
* New Type: __postfix_master (Steven Armstrong)
* New Type: __postfix_postconf (Steven Armstrong)
* New Type: __postfix_postmap (Steven Armstrong)
* New Type: __postfix_reload (Steven Armstrong)
* Type __line: Ensure regex does not contain /
* Type __ssh_authorized_keys: Bugfix: Preserve ownership (Steven Armstrong)
2.3.3: 2013-09-09
* Core: Add support for default values of optional parameters (Steven Armstrong)
* Type __start_on_boot: Bugfix for systemd (Steven Armstrong)
2.3.2: 2013-09-05
* Build: Ensure tests don't change attributes of non-test files
* Core: Fix typo in argument parser
* Core: Code cleanup: Remove old install code (Steven Armstrong)
* Core: Improve error message when using non-existing type in requirement
* New Type: __iptables_rule
* New Type: __iptables_apply
* Type __cdist: Also create home directory
* Type __cdist: Add support for --shell parameter
* Type __motd: Regenerate motd on Debian and Ubuntu
2.3.1: 2013-08-28
* Core: Support relative paths for configuration directories
* Core: Code cleanup (removed context class, added log class)
* Documentation: Add more best practises
* Documentation: Add troubleshooting chapter
* Type __key_value: Fix quoting problem (Steven Armstrong)
2.3.0: 2013-08-12
* Core: Added support for cdist shell
* Documentation: Improved some manpages
2.2.0: 2013-07-12
* Build: Cleanup the Makefile
* Type __package_opkg: Use shortcut version
* Core: Remove old pseudo object id "singleton" (Steven Armstrong)
2.1.2: 2013-07-09
* Build: Change clean-dist target to "distclean"
* Build: Moved a lot of build logic into Makefile for dependency resolution
* Core: Make global explorers available to initial manifest (Arkaitz Jimenez)
* Core: Change execution order to run object as one unit
* Documentation: Improved documentation (Tomáš Pospíšek)
* New Remote Example: Add support for sudo operations (Chase James)
* New Type: __update_alternatives
* New Type: __cdist
* Type __apt_ppa: Fix comparison operator (Tyler Akins)
* Type __start_on_boot: Archlinux changed to use systemd - adapt type
* Type __git: Missing quotes added (Chase James)
* Type __postgres_database: Make state parameter optional (Chase James)
* Type __postgres_role: Make state parameter optional, fix password bug (Chase James)
* Type __process: Make state parameter optional
* Type __cron: Simplyfied and syntax change
2.1.1: 2013-04-08
* Core: Use dynamic dependency resolver to allow indirect self dependencies
* Core: Remove umask call - protect /var/lib/cdist only (Arkaitz Jimenez)
* Explorer os: Added Slackware support (Eivind Uggedal)
* Type __git: Support mode and fix owner/group settings (contradict)
* Type __jail: State absent should implies stopped (Jake Guffey)
* Type __directory: Make stat call compatible with FreeBSD (Jake Guffey)
* Type __cron: Allow crontab without entries (Arkaitz Jimenez)
* Type __user: Add support for creating user home (Arkaitz Jimenez)
2.1.0: 2012-12-09
* Core: Ensure global explorers are executable

10
docs/changelog.future Normal file
View file

@ -0,0 +1,10 @@
Changelog
---------
* Changes are always commented with their author in (braces)
* Exception: No braces means author == Nico Schottelius
future (maybe 3.x?):
* Type __cron: Dropped support for old internal format
Using this version prior to running cdist 2.1.2 will
break add the cron entries twice.

BIN
docs/dev/logs/2012-09-03.dep-ideas.xoj Normal file
View file

Binary file not shown.

BIN
docs/dev/logs/2012-11-02.cdist-vs-centralised-development.xoj Normal file
View file

Binary file not shown.

282
docs/dev/logs/2012-11-15.cdist-sexy-interaction.svg Normal file
View file

@ -0,0 +1,282 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->
<svg
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
width="1052.3622"
height="744.09448"
id="svg2"
version="1.1"
inkscape:version="0.48.3.1 r9886"
sodipodi:docname="cdist-sexy-actions.svg"
inkscape:export-filename="/home/users/nico/cdist-sexy-actions.png"
inkscape:export-xdpi="90"
inkscape:export-ydpi="90">
<defs
id="defs4" />
<sodipodi:namedview
id="base"
pagecolor="#ffffff"
bordercolor="#666666"
borderopacity="1.0"
inkscape:pageopacity="0.0"
inkscape:pageshadow="2"
inkscape:zoom="0.62488877"
inkscape:cx="526.18109"
inkscape:cy="410.90353"
inkscape:document-units="px"
inkscape:current-layer="layer1"
showgrid="true"
objecttolerance="20"
guidetolerance="20"
inkscape:window-width="1436"
inkscape:window-height="861"
inkscape:window-x="0"
inkscape:window-y="18"
inkscape:window-maximized="0"
gridtolerance="10" />
<metadata
id="metadata7">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
<dc:title></dc:title>
</cc:Work>
</rdf:RDF>
</metadata>
<g
inkscape:label="Layer 1"
inkscape:groupmode="layer"
id="layer1"
transform="translate(0,-308.2677)">
<g
id="g3791"
transform="translate(-65.448375,393.5891)">
<rect
y="167.46855"
x="222.23357"
height="88.893425"
width="173.74623"
id="rect2985"
style="fill:#cdff13;fill-opacity:0.90416715;stroke:none" />
<text
sodipodi:linespacing="125%"
id="text3755"
y="228.0777"
x="260.61935"
style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Bitstream Vera Sans"
xml:space="preserve"><tspan
y="228.0777"
x="260.61935"
id="tspan3757"
sodipodi:role="line">cdist</tspan></text>
</g>
<g
id="g3802"
transform="translate(-88.702304,-97.993841)">
<rect
y="519.00165"
x="109.09647"
height="90.913727"
width="452.54834"
id="rect3796"
style="fill:#008000;fill-opacity:0.90416715;stroke:none" />
<text
sodipodi:linespacing="125%"
id="text3798"
y="575.57019"
x="171.72594"
style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Bitstream Vera Sans"
xml:space="preserve"><tspan
y="575.57019"
x="171.72594"
id="tspan3800"
sodipodi:role="line">configures hosts</tspan></text>
</g>
<g
id="g3791-5"
transform="translate(258.6201,492.81494)">
<g
id="g3834"
transform="translate(204.05081,-98.994949)">
<rect
style="fill:#cdff13;fill-opacity:0.90416715;stroke:none"
id="rect2985-2"
width="173.74623"
height="88.893425"
x="222.23357"
y="167.46855" />
<text
xml:space="preserve"
style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Bitstream Vera Sans"
x="264.65994"
y="224.03709"
id="text3755-9"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3757-2"
x="264.65994"
y="224.03709">sexy</tspan></text>
</g>
</g>
<g
id="g3802-3"
transform="translate(436.48671,-101.85286)">
<g
id="g3866">
<rect
style="fill:#008000;fill-opacity:0.90416715;stroke:none"
id="rect3796-9"
width="452.54834"
height="90.913727"
x="109.09647"
y="519.00165" />
<text
xml:space="preserve"
style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Bitstream Vera Sans"
x="143.44167"
y="573.54987"
id="text3798-6"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3800-5"
x="143.44167"
y="573.54987">manages inventory</tspan></text>
</g>
</g>
<g
id="g3802-9"
transform="matrix(0.96624748,0,0,1,-134.02038,188.43537)">
<g
id="g3980"
transform="translate(112.6206,22.403987)">
<rect
style="fill:#822a0e;fill-opacity:1;stroke:none"
id="rect3796-1"
width="319.27777"
height="146.92369"
x="115.72122"
y="536.6048" />
<text
xml:space="preserve"
style="font-size:40px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Bitstream Vera Sans"
x="269.50381"
y="592.71771"
id="text3798-3"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3800-6"
x="269.50381"
y="592.71771">installs hosts</tspan><tspan
sodipodi:role="line"
x="275.871"
y="642.71771"
id="tspan3968">(missing) </tspan></text>
</g>
</g>
<path
style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
d="m 330.53142,605.54234 354.37306,0.15493"
id="path3924"
inkscape:connector-type="polyline"
inkscape:connector-curvature="0"
inkscape:connection-start="#g3791"
inkscape:connection-start-point="d4"
inkscape:connection-end="#g3791-5"
inkscape:connection-end-point="d4" />
<text
xml:space="preserve"
style="font-size:27.59350204px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Bitstream Vera Sans"
x="447.33084"
y="637.85706"
id="text3926"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3928"
x="447.33084"
y="637.85706">interact</tspan></text>
<path
style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
d="m 244.62052,561.05765 1.06374,-49.13612"
id="path3930"
inkscape:connector-type="polyline"
inkscape:connector-curvature="0"
inkscape:connection-start="#g3791"
inkscape:connection-start-point="d4"
inkscape:connection-end="#g3802"
inkscape:connection-end-point="d4" />
<path
style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;display:inline"
d="m 243.08193,649.95108 -1.26428,97.49307"
id="path3932"
inkscape:connector-type="polyline"
inkscape:connector-curvature="0"
inkscape:connection-start="#g3791"
inkscape:connection-start-point="d4"
inkscape:connection-end="#g3802-9"
inkscape:connection-end-point="d4" />
<path
style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
d="m 771.80236,561.28854 0.0297,-53.22603"
id="path3934"
inkscape:connector-type="polyline"
inkscape:connector-curvature="0"
inkscape:connection-start="#g3791-5"
inkscape:connection-start-point="d4"
inkscape:connection-end="#g3802-3"
inkscape:connection-end-point="d4" />
<g
id="g3802-9-7"
transform="matrix(0.96624748,0,0,1,323.63894,252.66181)">
<g
id="g3900-3"
transform="matrix(1.0748862,0,0,0.96932859,-4.8574514,97.533037)">
<g
id="g3970"
transform="translate(58.55042,-102.35709)">
<rect
y="519.00165"
x="158.40208"
height="144.96896"
width="431.08368"
id="rect3796-1-1"
style="fill:#822a0e;fill-opacity:1;stroke:none" />
<text
sodipodi:linespacing="125%"
id="text3798-3-7"
y="576.61359"
x="374.46384"
style="font-size:40px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Bitstream Vera Sans"
xml:space="preserve"><tspan
y="576.61359"
x="374.46384"
id="tspan3800-6-3"
sodipodi:role="line">visualises inventory</tspan><tspan
id="tspan3966"
y="626.61359"
x="380.83102"
sodipodi:role="line">(missing) </tspan></text>
</g>
</g>
</g>
<path
style="fill:none;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
d="m 771.03726,650.18197 -1.54888,92.98943"
id="path3986"
inkscape:connector-type="polyline"
inkscape:connector-curvature="0"
inkscape:connection-start="#g3791-5"
inkscape:connection-start-point="d4"
inkscape:connection-end="#g3802-9-7"
inkscape:connection-end-point="d4" />
</g>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 10 KiB

BIN
docs/dev/logs/2013-01-03.dependencies-visualised.xoj Normal file
View file

Binary file not shown.

34
docs/dev/logs/2013-02-05.debugging-wrong-singleton-type-parameter Normal file
View file

@ -0,0 +1,34 @@
Traceback (most recent call last):
File "/home/users/nico/p/cdist/cdist/bin/../scripts/cdist", line 230, in <module>
commandline()
File "/home/users/nico/p/cdist/cdist/bin/../scripts/cdist", line 104, in commandline
args.func(args)
File "/home/users/nico/p/cdist/cdist/bin/../scripts/cdist", line 107, in config
configinstall(args, mode=cdist.config.Config)
File "/home/users/nico/p/cdist/cdist/bin/../scripts/cdist", line 143, in configinstall
configinstall_onehost(host, args, mode, parallel=False)
File "/home/users/nico/p/cdist/cdist/bin/../scripts/cdist", line 180, in configinstall_onehost
c.deploy_and_cleanup()
File "/home/users/nico/oeffentlich/rechner/projekte/cdist/cdist/cdist/config_install.py", line 74, in deploy_and_cleanup
self.deploy_to()
File "/home/users/nico/oeffentlich/rechner/projekte/cdist/cdist/cdist/config_install.py", line 68, in deploy_to
self.stage_prepare()
File "/home/users/nico/oeffentlich/rechner/projekte/cdist/cdist/cdist/config_install.py", line 91, in stage_prepare
self.context.local.type_path):
File "/home/users/nico/oeffentlich/rechner/projekte/cdist/cdist/cdist/core/cdist_object.py", line 80, in list_objects
yield cls(cdist.core.CdistType(type_base_path, type_name), object_base_path, object_id=object_id)
File "/home/users/nico/oeffentlich/rechner/projekte/cdist/cdist/cdist/core/cdist_object.py", line 65, in __init__
self.validate_object_id()
File "/home/users/nico/oeffentlich/rechner/projekte/cdist/cdist/cdist/core/cdist_object.py", line 130, in validate_object_id
(self.cdist_type.name, self.parameters))
File "/home/users/nico/oeffentlich/rechner/projekte/cdist/cdist/cdist/util/fsproperty.py", line 210, in __get__
return self._get_attribute(instance, owner)
File "/home/users/nico/oeffentlich/rechner/projekte/cdist/cdist/cdist/util/fsproperty.py", line 202, in _get_attribute
path = self._get_path(instance)
File "/home/users/nico/oeffentlich/rechner/projekte/cdist/cdist/cdist/util/fsproperty.py", line 190, in _get_path
path = path(instance)
File "/home/users/nico/oeffentlich/rechner/projekte/cdist/cdist/cdist/core/cdist_object.py", line 192, in <lambda>
parameters = fsproperty.DirectoryDictProperty(lambda obj: os.path.join(obj.base_path, obj.parameter_path))
AttributeError: 'CdistObject' object has no attribute 'parameter_path'
[22:37] brief:~%

15
docs/dev/logs/2013-02-05.weird-notsingleton-type-error Normal file
View file

@ -0,0 +1,15 @@
Hard to find the source bug/problem:
DEBUG: solr.petspremium.de: (emulator) __file//etc/solr/solr.xml: Finished __file/etc/solr/solr.xml/.cdist {'mode': '0644', 'source': '/home/users/nico/.tmp/tmpn27s24/out/conf/type/__petspremium_solr/files/solr/solr.xml'}
+ for file in '$(find . -type f | sed '\''s,^./,,'\'')'
+ dfile=/etc/solr/web.xml
+ reqdir=/etc/solr
+ require=__directory/etc/solr
+ __file /etc/solr/web.xml --source /home/users/nico/.tmp/tmpn27s24/out/conf/type/__petspremium_solr/files/solr/web.xml --mode 0644
DEBUG: solr.petspremium.de: (emulator): /home/users/nico/.tmp/tmpn27s24/out/bin/__file: Namespace(mode='0644', object_id=['/etc/solr/web.xml'], source='/home/users/nico/.tmp/tmpn27s24/out/conf/type/__petspremium_solr/files/solr/web.xml')
DEBUG: solr.petspremium.de: (emulator) __file//etc/solr/web.xml: Recording requirement: __directory/etc/solr
DEBUG: solr.petspremium.de: (emulator) __file//etc/solr/web.xml: Finished __file/etc/solr/web.xml/.cdist {'source': '/home/users/nico/.tmp/tmpn27s24/out/conf/type/__petspremium_solr/files/solr/web.xml', 'mode': '0644'}
ERROR: solr.petspremium.de: Type __directory requires object id (is not a singleton type)
INFO: Total processing time for 1 host(s): 9.756716251373291
ERROR: Failed to deploy to the following hosts: solr.petspremium.de

30
docs/dev/logs/2013-04-03.dependency-discussion Normal file
View file

@ -0,0 +1,30 @@
Steven, Nico
Discussion raised due to proposal from Arkaitz Jimenez
--------------------------------------------------------------------------------
Proposal changes back to cdist behaviour as of 2011 (see commit 61b7b68).
Change would introduce:
- no direct stage based running
- stages only in object (not globally)
- cannot build full dependency list before beginning
- Thus wildcard requirements (require="__file/*") don't work anymore
Accepting this or similar approaches means:
- Drop wildcard requirements (is undocumented anyway)
- Type execution is closed (again)
Furthermore/other points:
- Change cdist to continue run as long as possible
- Don't stop if an object fails
- Record failure, print at the end (and exit non zero)
- Logging
- Catch output of manifest, gencode, code, do not display directly
- Print at the end
- Prefix with hostname as usual!

BIN
docs/dev/logs/2013-04-08.execution-graph.xoj Normal file
View file

Binary file not shown.

77
docs/dev/logs/2013-04-10.discussion Normal file
View file

@ -0,0 +1,77 @@
Steven, Nico (ETH office)
- Try out patch for dependency resolver changing from [nico]
- Add tests
- Cleanup code:
- remove all old resolver parts (including tests!)
- remve wildcard matching pattern code
- Cache: [nobody]
- Should cache be usable by types?
- Should all run outputs be stored?
- Different caches for install and config
- Replace fsproperties with cconfig [steven]
- Maybe support "rerun from previous version (cache)"? [nobody]
- need to include initial manifest(s!)
- copy/link types
- save remote-{exec,copy} parameters (copy or save argument list)
- cdist replay / oldconfig ?
- Support diffing two configurations [nobody]
- cdist diff ?
- Nested Types [both]
- Motivation:
- Put everything related into one directory
- Have a look at it when Arkaitz pushes out pull request
- Implementations:
1) Arkaitz
Folder structure Call Object
__package/ __package abc __package/abc
__package/type/pkg __package.pkg abc __package.pkg/abc
__package/type/pkg/type/green __package.pkg.green abc __package.pkg.green/abc
...
__package.pkg __package.pkg abc __package.pkg/abc
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
=> Need to forbid types with "." in the name!
2) Steven (earlier version)
Folder structure Call Object
__package/.type __package abc __package/abc
__package/pkg/.type __package.pkg abc __package.pkg/abc
__package/pkg/green/.type __package.pkg.green abc __package.pkg.green/abc
- Clashes:
- if __<type>.<name> and __<type> and subtype <name> exist both (in both implementations)
- Install [nobody]
- Merge into master?
- Needs some cleanups
- PreOS [nobody]
- cdist preos / preos-generate
--output=<dir-or-filename>
--arch=[i386|amd64|arm??]
--type=[usb, cdrom/iso, floppy, pxe]
--other-params (?)
- Maybe implement using cdist config indirectly and a type __preos
- Can be:
- Internally only (devs)
- Usable by end users
- Requirements:
- git
- buildchain
- toolchain for target arch
- ...

44
docs/dev/logs/2013-04-12.execution-order Normal file
View file

@ -0,0 +1,44 @@
Old:
- global explores (all)
- initial manifest
- for each object
execute type explorers
execute manifest
continue until all objects (including newly created)
have their type explorers/manifests run
- build dependency tree
- for each object
execute gencode-*
execute code-*
New:
- run all global explorers
- run initial manifest
creates zero or more cdist_objects
- for each cdist_object
if not cdist_object.has_unfullfilled_requirements:
execute type explorers
execute manifest
may create new objects, resulting in autorequirements
# Gained requirements during manifest run
if object.has_auto_requirements():
continue
cdist_object.execute gencode-*
cdist_object.execute code-*
Requirements / Test cases for requirments / resolver:
- omnipotence
-
--------------------------------------------------------------------------------
ERROR: localhost: The following objects could not be resolved: __cdistmarker/singleton requires autorequires ; __directory/etc/sudoers.d requires autorequires ; __file/etc/sudoers.d/nico requires __directory/etc/sudoers.d autorequires ; __file/etc/motd requires autorequires ; __package_pacman/atop requires autorequires ; __package_pacman/screen requires autorequires ; __package_pacman/strace requires autorequires ; __package_pacman/vim requires autorequires ; __package_pacman/zsh requires autorequires ; __package_pacman/lftp requires autorequires ; __package_pacman/nmap requires autorequires ; __package_pacman/ntp requires autorequires ; __package_pacman/rsync requires autorequires ; __package_pacman/rtorrent requires autorequires ; __package_pacman/wget requires autorequires ; __package_pacman/nload requires autorequires ; __package_pacman/iftop requires autorequires ; __package_pacman/mosh requires autorequires ; __package_pacman/git requires autorequires ; __package_pacman/mercurial requires autorequires ; __package_pacman/netcat requires autorequires ; __package_pacman/python-virtualenv requires autorequires ; __package_pacman/wireshark-cli requires autorequires ; __package_pacman/sudo requires autorequires
INFO: Total processing time for 1 host(s): 32.30426597595215
ERROR: Failed to deploy to the following hosts: localhost

340
docs/dev/logs/2013-05-04.ssh Normal file
View file

@ -0,0 +1,340 @@
- analysis of ssh connections for callback
SSH_CLIENT='::1 38502 22'
SSH_CONNECTION='::1 38502 ::1 22'
-> callback possible to source host
[ target host ] <--------------|
| |
| |
| |
| trigger | configuration
| |
v |
[ configuration host ] ----|
- dynamic port allocation for tunneling
[1:37] bento:~% ssh -R 0:localhost:22 localhost
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Allocated port 53161 for remote forward to localhost:22
SSH_AUTH_SOCK=/tmp/ssh-zDCWbUVcUK/agent.30749
SSH_CLIENT='::1 38587 22'
SSH_CONNECTION='::1 38587 ::1 22'
SSH_TTY=/dev/pts/21
- ssh_config:
DynamicForward
LocalForward
RemoteForward
- testing
[1:52] bento:cdist% netstat -anp | grep 56844
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 127.0.0.1:56844 0.0.0.0:* LISTEN -
tcp6 0 0 ::1:56844 :::* LISTEN -
[1:53] bento:cdist%
[1:48] bento:~% ssh -R 0:localhost:22 localhost
Allocated port 56844 for remote forward to localhost:22
...
- chatting
01:42 -!- Irssi: Join to #openssh was synced in 0 secs
01:42 < telmich> good evening
01:43 < telmich> I am trying to make use of remote port forwarding using dynamic port
allocation (port=0) -- I am wondering if there is an easy way to
access the port number on the remote side easily?
01:44 < telmich> background for this question is: I'd like to allow various clients to
login to a configuration server, which then configures the clients by
using the tunnel the client provides for the server to ssh back into
02:07 < BasketCase> telmich: afaik you need to use a tool like ss/netstat/lsof to see what port it has open
- ssh debug
[11:37] bento:~% ssh -R 0:localhost:22 localhost
Allocated port 33562 for remote forward to localhost:22
.. . .x+=:. s
dF @88> z` ^% :8
'88bu. %8P . <k .88
. '*88888bu . .@8Ned8" :888ooo
.udR88N ^"*8888N .@88u .@^%8888" -*8888888
<888'888k beWE "888L ''888E` x88: `)8b. 8888
9888 'Y" 888E 888E 888E 8888N=*8888 8888
9888 888E 888E 888E %8" R88 8888
9888 888E 888F 888E @8Wou 9% .8888Lu=
?8888u../ .888N..888 888& .888888P` ^%888*
"8888P' `"888*"" R888" ` ^"F 'Y"
"P' "" ""
Welcome to a cdist automated system!
Last login: Sat May 4 01:52:46 2013 from localhost.localdomain
debug1: PAM: reinitializing credentials
debug1: permanently_set_uid: 0/0
Environment:
USER=root
LOGNAME=root
HOME=/root
PATH=/usr/bin:/bin:/usr/sbin:/sbin
MAIL=/var/spool/mail/root
SHELL=/bin/bash
SSH_CLIENT=::1 57848 22
SSH_CONNECTION=::1 57848 ::1 22
SSH_TTY=/dev/pts/32
TERM=rxvt-unicode
XDG_SESSION_ID=1
XDG_RUNTIME_DIR=/run/user/1000
XDG_SEAT=seat0
XDG_VTNR=1
SSH_AUTH_SOCK=/tmp/ssh-6j0elukLHA/agent.17260
[root@bento ~]#
[root@bento nico]# /usr/sbin/sshd -D -d
debug1: sshd version OpenSSH_6.2, OpenSSL 1.0.1e 11 Feb 2013
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: read PEM private key done: type ECDSA
debug1: private host key: #2 type 3 ECDSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-D'
debug1: rexec_argv[2]='-d'
Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from ::1 port 57848
debug1: Client protocol version 2.0; client software version OpenSSH_6.2
debug1: match: OpenSSH_6.2 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: permanently_set_uid: 99/99 [preauth]
debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com zlib@openssh.com [preauth]
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com zlib@openssh.com [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user root service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "root"
debug1: PAM: setting PAM_RHOST to "localhost.localdomain"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user root service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug1: test whether pkalg/pkblob are acceptable [preauth]
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: matching key found: file /root/.ssh/authorized_keys, line 2
Found matching RSA key: 2e:1b:3f:10:01:1d:21:6c:6c:1e:3d:a9:33:ba:3c:f7
debug1: restore_uid: 0/0
Postponed publickey for root from ::1 port 57848 ssh2 [preauth]
debug1: userauth-request for user root service ssh-connection method publickey [preauth]
debug1: attempt 2 failures 0 [preauth]
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: matching key found: file /root/.ssh/authorized_keys, line 2
Found matching RSA key: 2e:1b:3f:10:01:1d:21:6c:6c:1e:3d:a9:33:ba:3c:f7
debug1: restore_uid: 0/0
debug1: ssh_rsa_verify: signature correct
debug1: do_pam_account: called
Accepted publickey for root from ::1 port 57848 ssh2
debug1: monitor_child_preauth: root has been authenticated by privileged process
debug1: Enabling compression at level 6. [preauth]
debug1: monitor_read_log: child log fd closed
debug1: PAM: establishing credentials
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch_20
debug1: server_input_global_request: rtype tcpip-forward want_reply 1
debug1: server_input_global_request: tcpip-forward listen localhost port 0
debug1: Local forwarding listening on ::1 port 0.
debug1: Allocated listen port 33562
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 33562.
debug1: channel 1: new [port listener]
debug1: server_input_channel_open: ctype session rchan 0 win 1048576 max 16384
debug1: input_session_request
debug1: channel 2: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 2
debug1: session_open: session 0: link with channel 2
debug1: server_input_channel_open: confirm session
debug1: server_input_global_request: rtype no-more-sessions@openssh.com want_reply 0
debug1: server_input_channel_req: channel 2 request auth-agent-req@openssh.com reply 0
debug1: session_by_channel: session 0 channel 2
debug1: session_input_channel_req: session 0 req auth-agent-req@openssh.com
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: restore_uid: 0/0
debug1: channel 3: new [auth socket]
debug1: server_input_channel_req: channel 2 request pty-req reply 1
debug1: session_by_channel: session 0 channel 2
debug1: session_input_channel_req: session 0 req pty-req
debug1: Allocating pty.
debug1: session_pty_req: session 0 alloc /dev/pts/32
debug1: server_input_channel_req: channel 2 request shell reply 1
debug1: session_by_channel: session 0 channel 2
debug1: session_input_channel_req: session 0 req shell
debug1: Setting controlling tty using TIOCSCTTY.
--------------------------------------------------------------------------------
debug1: server_input_global_request: rtype tcpip-forward want_reply 1
debug1: server_input_global_request: tcpip-forward listen localhost port 0
debug1: Local forwarding listening on ::1 port 0.
debug1: Allocated listen port 33562
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 33562.
[11:49] bento:openssh-6.2p1% grep "Allocated listen port" -r .
./channels.c: debug("Allocated listen port %d",
[11:49] bento:openssh-6.2p1%
--------------------------------------------------------------------------------
[11:54] bento:~% ssh -R 0:localhost:22 -R 0:192.168.1.1:33 localhost
Allocated port 48392 for remote forward to localhost:22
Allocated port 37515 for remote forward to 192.168.1.1:33
debug1: server_input_global_request: rtype tcpip-forward want_reply 1
debug1: server_input_global_request: tcpip-forward listen localhost port 0
debug1: Local forwarding listening on ::1 port 0.
debug1: Allocated listen port 48392
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 48392.
debug1: channel 1: new [port listener]
debug1: server_input_global_request: rtype tcpip-forward want_reply 1
debug1: server_input_global_request: tcpip-forward listen localhost port 0
debug1: Local forwarding listening on ::1 port 0.
debug1: Allocated listen port 37515
debug1: channel 2: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 37515.
debug1: channel 3: new [port listener]
debug1: server_input_channel_open: ctype session rchan 0 win 1048576 max 16384
debug1: input_session_request
debug1: channel 4: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 4
debug1: session_open: session 0: link with channel 4
debug1: Local forwarding listening on ::1 port 5555.
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 5555.
debug1: channel 1: new [port listener]
debug1: server_input_global_request: rtype tcpip-forward want_reply 1
debug1: server_input_global_request: tcpip-forward listen localhost port 4444
debug1: Local forwarding listening on ::1 port 4444.
debug1: channel 2: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 4444.
debug1: channel 3: new [port listener]
debug1: server_input_channel_open: ctype session rchan 0 win 1048576 max 16384
debug1: input_session_request
debug1: channel 4: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 4
--------------------------------------------------------------------------------
[12:06] bento:openssh-6.2p1% grep SSH_CONNECTION -r *
audit-bsm.c: case SSH_CONNECTION_CLOSE:
audit.c: {SSH_CONNECTION_CLOSE, "CONNECTION_CLOSE"},
audit.c: {SSH_CONNECTION_ABANDON, "CONNECTION_ABANDON"},
audit.h: SSH_CONNECTION_CLOSE, /* closed after attempting auth or session */
audit.h: SSH_CONNECTION_ABANDON, /* closed without completing auth */
audit-linux.c: case SSH_CONNECTION_CLOSE:
monitor.c: case SSH_CONNECTION_CLOSE:
regress/proxy-connect.sh: SSH_CONNECTION=`${SSH} -$p -F $OBJ/ssh_proxy 999.999.999.999 'echo $SSH_CONNECTION'`
regress/proxy-connect.sh: if [ "$SSH_CONNECTION" != "UNKNOWN 65535 UNKNOWN 65535" ]; then
regress/proxy-connect.sh: fail "bad SSH_CONNECTION"
session.c: child_set_env(&env, &envsize, "SSH_CONNECTION", buf);
sftp-server.c: if ((cp = getenv("SSH_CONNECTION")) != NULL) {
sftp-server.c: error("Malformed SSH_CONNECTION variable: \"%s\"",
sftp-server.c: getenv("SSH_CONNECTION"));
ssh.0: SSH_CONNECTION Identifies the client and server ends of the
ssh.1:.It Ev SSH_CONNECTION
sshd.c: PRIVSEP(audit_event(SSH_CONNECTION_CLOSE));
sshd.c: audit_event(SSH_CONNECTION_ABANDON);
[12:06] bento:openssh-6.2p1%
--------------------------------------------------------------------------------
debug1: Remote connections from LOCALHOST:5555 forwarded to local address localhost:22
--------------------------------------------------------------------------------
[12:42] bento:openssh-6.2p1% grep tcpip-forward *
channels.c: packet_put_cstring("tcpip-forward");
channels.c: packet_put_cstring("cancel-tcpip-forward");
Binary file channels.o matches
grep: contrib: Is a directory
Binary file libssh.a matches
grep: openbsd-compat: Is a directory
grep: regress: Is a directory
grep: scard: Is a directory
serverloop.c: if (strcmp(rtype, "tcpip-forward") == 0) {
serverloop.c: debug("server_input_global_request: tcpip-forward listen %s port %d",
serverloop.c: } else if (strcmp(rtype, "cancel-tcpip-forward") == 0) {
serverloop.c: debug("%s: cancel-tcpip-forward addr %s port %d", __func__,
Binary file serverloop.o matches
Binary file ssh matches
Binary file sshd matches
Binary file ssh-keyscan matches
Binary file ssh-keysign matches
[12:42] bento:openssh-6.2p1%
--------------------------------------------------------------------------------
Channel information for (remote) forwarding:
c = channel_new("port listener", type, sock, sock, -1,
CHAN_TCP_WINDOW_DEFAULT, CHAN_TCP_PACKET_DEFAULT,
0, "port listener", 1);
c->path = xstrdup(host);
c->host_port = port_to_connect;
c->listening_addr = addr == NULL ? NULL : xstrdup(addr);
if (listen_port == 0 && allocated_listen_port != NULL &&
!(datafellows & SSH_BUG_DYNAMIC_RPORT))
c->listening_port = *allocated_listen_port;
else
c->listening_port = listen_port;
--------------------------------------------------------------------------------
Code handling remote forwarding in the client:
- ssh_init_forwarding
- channel_request_remote_forwarding
Sends hostname + port for ssh1 only - not send in ssh2
Code handling forwarding / listening in the server:
- channel_new: creates channels, 2 per listener (ipv4/ipv6)
- channels_alloc contains number of channels
- server_input_global_request
Reads only listen port, not hostname/port to connect to
- channel_setup_remote_fwd_listener
- channel_setup_remote_fwd_listener
Code handling environment variables:
- child_set_env
1236 child_set_env(&env, &envsize, "SSH_CONNECTION", buf);

40
docs/dev/logs/2013-05-17.ssh-callback-socat Normal file
View file

@ -0,0 +1,40 @@
start ssh
to controlhost,
bind other side to
localhost:22
targethost ------> ssh ------> controlhost
|
|
socat: connect stdin/stdout to ?
start cdist with port information
added
Use
socat
--------------------------------------------------------------------------------
TCP:<host>:<port>
Connects to <port> [TCP service] on <host> [IP address] using TCP/IP version 4 or 6 depending on address specifi‐
cation, name resolution, or option pf.
Option groups: FD,SOCKET,IP4,IP6,TCP,RETRY
Useful options: crnl, bind, pf, connect-timeout, tos, mtudiscover, mss, nodelay, nonblock, sourceport, retry,
readbytes
See also: TCP4, TCP6, TCP-LISTEN, UDP, SCTP-CONNECT, UNIX-CONNECT
forever
--------------------------------------------------------------------------------
[root@nico-dev-vm-snr01 yum.repos.d]# ps aux | grep socat
nico 25035 0.0 0.0 41640 1524 ? Ss 13:27 0:00 socat - TCP-LISTEN:1234
root 25037 0.0 0.0 103240 836 pts/1 S+ 13:27 0:00 grep socat
[root@nico-dev-vm-snr01 yum.repos.d]#
--------------------------------------------------------------------------------

38
docs/dev/logs/2013-07-12.release Normal file
View file

@ -0,0 +1,38 @@
- setup release date in docs/changelog to today manually
- checkout master branch
[
x check if date is correct in docs/changelog
x ensure all unittests work
- requires (wrong/outdated) versionfile!
x compile manpages
x compile speeches
]
[
x add manpages to website repo
x add speeches to website repo
x rsync cdist docs to website repo & add to website repo
x create blog entry & add to website repo
]
x upload website
x fix latest link for manpages
x send mail to mailinglist -> also requires git tag & git release
x should also require web-release including blog!
- create PKGBUILD for archlinux release
x create git tag / read description
t if necessary create version branch
x change to version branch and merge tag!
x update git repos
x update website from repo
x create release on freecode
x create versionfile
x make pypi release
x make archlinux release
manual last steps:
- announce on linkedin
- announce on twitter

56
docs/dev/logs/2013-07-25.source-error-does-not-stop-cdist Normal file
View file

@ -0,0 +1,56 @@
Symptom:
running something in a manifest and that fails does not exist
the cdist run
Analysis:
Find out what the shell does:
[23:56] bento:testshell% cat a.sh
# source something that fails
. b.sh
[23:57] bento:testshell% cat b.sh
nosuchcommand
[23:57] bento:testshell% sh -e a.sh
a.sh: 2: .: b.sh: not found
[23:57] bento:testshell% echo $?
2
-> exit 2 -> looks good
Find out what the python does:
[23:57] bento:testshell% python3
Python 3.3.2 (default, May 21 2013, 15:40:45)
[GCC 4.8.0 20130502 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_call(["/bin/sh", "-e", "a.sh"])
a.sh: 2: .: b.sh: not found
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.3/subprocess.py", line 544, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/bin/sh', '-e', 'a.sh']' returned non-zero exit status 2
>>>
Conclusion:
Manifests that execute (!) other shell scripts does
not necessarily give the -e flag to the other script
-> called script can have failures, but exit 0
if something the last thing executed does exit 0!
Solution:
Instead of doing stuff like
"$__manifest/special"
use
sh -e "$__manifest/special"
or source the script:
. "$__manifest/special"
(runs the script in the same namespace/process as everything in the
calling script)

2
docs/dev/logs/2013-08-07.shell Normal file
View file

@ -0,0 +1,2 @@
What about having a cdist shell to have a shell with all available types?
Let's give it a try!

28
docs/dev/logs/2013-08-12.release Normal file
View file

@ -0,0 +1,28 @@
- already on 2.3.0-1 during release
- user bug: there should be no changes / commits during a release process
hard linking docs/man/man7/cdist-type__user.7 -> cdist-2.3.0-1-g8192c2c/docs/man/man7
hard linking docs/man/man7/cdist-type__user.html -> cdist-2.3.0-1-g8192c2c/docs/man/man7
hard linking docs/man/man7/cdist-type__user_groups.7 -> cdist-2.3.0-1-g8192c2c/docs/man/man7
hard linking docs/man/man7/cdist-type__user_groups.html -> cdist-2.3.0-1-g8192c2c/docs/man/man7
hard linking scripts/cdist -> cdist-2.3.0-1-g8192c2c/scripts
creating dist
Creating tar archive
removing 'cdist-2.3.0-1-g8192c2c' (and everything under it)
running upload
Submitting dist/cdist-2.3.0-1-g8192c2c.tar.gz to http://pypi.python.org/pypi
Server response (200): OK
touch .lock-pypi
./PKGBUILD.in 2.3.0
==> Retrieving sources...
-> Downloading cdist-2.3.0.tar.gz...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 404 Not Found
==> ERROR: Failure while downloading cdist-2.3.0.tar.gz
Aborting...
make: *** [PKGBUILD] Error 1
[12:38] bento:cdist%

17
docs/dev/logs/2013-08-18.cache-enhancement Normal file
View file

@ -0,0 +1,17 @@
- always save cache = outdir
- even if run aborts (for debugging)
- add a state flag
- save cache in a date based directory
- also add support for a per-host pidfile
- allow user to specify cache dir - to give
full flexibility
- drop context - it is a very small unecessary wrapper
- maye introduce cdist.log instead!
- replace out_path with out_base
- directory under which all the subdirectories are
created
-> by default ~/.cdist/run
-> out_base_path
- drop support for deprecated environment variables
__cdist_out_dir
__cdist_remote_out_dir

5
docs/dev/logs/2013-08-28.release Normal file
View file

@ -0,0 +1,5 @@
- release process releases pypi from something
that is git describe based, not changelog based...
- git describe should equal changelog, but may be
inconsistent due to branch merging!

34
docs/dev/logs/2013-09-05.test-cp Normal file
View file

@ -0,0 +1,34 @@
Test copy copys symlinks - making real files would be better
Test how to use cp:
[12:54] bento:~% cd test
[12:54] bento:test% ln -s /etc/passwd
[12:54] bento:test% cd ..
[12:54] bento:~% cp -r test test2
[12:54] bento:~% ls -lh test2/
total 4.0K
lrwxrwxrwx 1 nico nico 11 Sep 5 12:54 passwd -> /etc/passwd
[12:54] bento:~% rm -rf test2/
--------------------------------------------------------------------------------
[12:54] bento:~% ls -lh test2/
total 4.0K
lrwxrwxrwx 1 nico nico 11 Sep 5 12:54 passwd -> /etc/passwd
[12:54] bento:~% rm -rf test2/
[12:54] bento:~% cp -r --dereference test test2
[12:56] bento:~% ls -l test2/
total 4
-rw------- 1 nico nico 960 Sep 5 12:56 passwd
[12:56] bento:~%
--------------------------------------------------------------------------------
[13:04] bento:cdist% git describe
2.3.2
[13:09] bento:cdist% vi MANIFEST.in
[13:09] bento:cdist% vi MANIFEST
[13:09] bento:cdist% vi setup.py
[13:09] bento:cdist% cat cdist/version.py
VERSION = "2.3.1-34-g7acf041"
[13:10] bento:cdist%

13
docs/dev/logs/2013-10-03.ossawards/infos Normal file
View file

@ -0,0 +1,13 @@
Required for the ossawards until 2013-10-06:
- all source code
- licenses GPLv3
- installation instructions,
- On Linux do the following:
- pip install
-
- necessary documents and
- a demo video onto our web hard.
- installation
- cdist via cdist
- presentation
- build from existing ones (?)

6
docs/dev/logs/2013-10-29.__line Normal file
View file

@ -0,0 +1,6 @@
- fix handling of fixed strings
- ensure special characters are not interpreted
[12:18] bento:~% cat /etc/bash.bashrc
cat: /etc/bash.bashrc: Permission denied
[12:19] bento:~%

BIN
docs/gfx/cdist-and-sexy-white.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 5.5 KiB

BIN
docs/gfx/label-cdist-ngcm.odt Normal file
View file

Binary file not shown.

17
docs/man/cdist-reference.text.sh
View file

@ -116,8 +116,13 @@ confdir/type/<name>/parameter/required::
confdir/type/<name>/parameter/optional::
Parameters optionally accepted by type, \n seperated list.
confdir/type/<name>/parameter/default/*::
Default values for optional parameters.
Assuming an optional parameter name of 'foo', it's default value would
be read from the file confdir/type/<name>/parameter/default/foo.
confdir/type/<name>/parameter/boolean::
Boolean parameters accepted by type, \n seperated list.
Boolean parameters accepted by type, \n seperated list.
confdir/type/<name>/explorer::
Location of the type specific explorers.
@ -179,13 +184,13 @@ ENVIRONMENT VARIABLES
---------------------
__explorer::
Directory that contains all global explorers.
Available for: explorer, type explorer
Available for: initial manifest, explorer, type explorer, shell
__manifest::
Directory that contains the initial manifest.
Available for: initial manifest, type manifest
Available for: initial manifest, type manifest, shell
__global::
Directory that contains generic output like explorer.
Available for: initial manifest, type manifest, type gencode
Available for: initial manifest, type manifest, type gencode, shell
__object::
Directory that contains the current object.
Available for: type manifest, type explorer, type gencode
@ -200,7 +205,7 @@ __object_name::
Available for: type manifest, type explorer, type gencode
__target_host::
The host we are deploying to.
Available for: explorer, initial manifest, type explorer, type manifest, type gencode
Available for: explorer, initial manifest, type explorer, type manifest, type gencode, shell
__type::
Path to the current type.
Available for: type manifest, type gencode
@ -216,6 +221,6 @@ SEE ALSO
COPYING
-------
Copyright \(C) 2011-2012 Nico Schottelius. Free use of this software is
Copyright \(C) 2011-2013 Nico Schottelius. Free use of this software is
granted under the terms of the GNU General Public License version 3 (GPLv3).
eof

60
docs/man/man1/cdist.text
View file

@ -5,40 +5,51 @@ Nico Schottelius <nico-cdist--@--schottelius.org>
NAME
----
cdist - Configuration management
cdist - Usable Configuration Management
SYNOPSIS
--------
cdist [-h] [-V]
cdist [-h] [-d] [-v] [-V] {banner,config,shell} ...
cdist banner
cdist banner [-h] [-d] [-v]
cdist config [-h] [-d] [-V] [-c CONF_DIR] [-i MANIFEST] [-p] [-s] host [host ...]
cdist shell [-h] [-d] [-v] [-s SHELL]
DESCRIPTION
-----------
cdist is the frontend executable to the cdist configuration management.
cdist supports different as explained below. The options to the main
program are:
cdist supports different subcommands as explained below.
GENERAL
-------
All commands except the following options:
-d, --debug::
Set log level to debug
-h, --help::
Show the help screen
-v, --verbose:
Set log level to info, be more verbose
-V, --version::
Show version and exit
BANNER
-------
Displays the cdist banner.
------
Displays the cdist banner. Useful for printing
cdist posters - a must have for every office.
CONFIG
------
Configure a system
Configure one or more hosts
-h, --help::
Show the help screen
@ -52,9 +63,6 @@ Configure a system
--conf-dir argument have higher precedence over those set through the
environment variable.
-d, --debug::
Enable debug output
-i MANIFEST, --initial-manifest MANIFEST::
Path to a cdist manifest or - to read from stdin
@ -70,20 +78,30 @@ Configure a system
--remote-exec REMOTE_EXEC:
Command to use for remote execution (should behave like ssh)
SHELL
-----
This command allows you to spawn a shell that enables access
to the types as commands. It can be thought as an
"interactive manifest" environment. See below for example
usage. Its primary use is for debugging type parameters.
-s/--shell::
Select shell to use, defaults to current shell
EXAMPLES
--------
--------------------------------------------------------------------------------
# Configure ikq05.ethz.ch with debug enabled
cdist config -d ikq05.ethz.ch
% cdist config -d ikq05.ethz.ch
# Configure hosts in parallel and use a different configuration directory
cdist config -c ~/p/cdist-nutzung \
% cdist config -c ~/p/cdist-nutzung \
-p ikq02.ethz.ch ikq03.ethz.ch ikq04.ethz.ch
# Use custom remote exec / copy commands
cdist config --remote-exec /path/to/my/remote/exec \
% cdist config --remote-exec /path/to/my/remote/exec \
--remote-copy /path/to/my/remote/copy \
-p ikq02.ethz.ch ikq03.ethz.ch ikq04.ethz.ch
@ -91,10 +109,18 @@ cdist config --remote-exec /path/to/my/remote/exec \
cdist banner
# Show help
cdist --help
% cdist --help
# Show Version
cdist --version
% cdist --version
# Enter a shell that has access to emulated types
% cdist shell
% __git
usage: __git --source SOURCE [--state STATE] [--branch BRANCH]
[--group GROUP] [--owner OWNER] [--mode MODE] object_id
--------------------------------------------------------------------------------
@ -125,5 +151,5 @@ SEE ALSO
COPYING
-------
Copyright \(C) 2011-2012 Nico Schottelius. Free use of this software is
Copyright \(C) 2011-2013 Nico Schottelius. Free use of this software is
granted under the terms of the GNU General Public License version 3 (GPLv3).

55
docs/man/man7/cdist-best-practice.text
View file

@ -118,7 +118,7 @@ The following **.git/config** is taken from a a real world scenario:
url = git://git.schottelius.org/cdist
fetch = +refs/heads/*:refs/remotes/upstream/*
# Same as upstream, but works when being offline
# Same as upstream, but works when being offline
[remote "local"]
fetch = +refs/heads/*:refs/remotes/local/*
url = /home/users/nico/p/cdist
@ -167,7 +167,7 @@ TEMPLATING
* create directory templates/ in your type (convention)
* create the template as an executable file like templates/basic.conf.sh, it will output text using shell variables for the values
--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------
#!/bin/sh
# in the template, use cat << eof (here document) to output the text
# and use standard shell variables in the template
@ -182,19 +182,58 @@ server {
error_log /var/log/nginx/$SERVERNAME_error.log
}
EOF
--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------
* in the manifest, export the relevant variables and add the following lines in your manifest:
--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------
# export variables needed for the template
export SERVERNAME='test"
export ROOT='/var/www/test'
# render the template
mkdir -p "$__object/files"
"$__type/templates/basic.conf.sh" > "$__object/files/basic.conf"
# send the rendered template
__file /etc/nginx/sites-available/test.conf --state present --source "$__object/files/basic.conf"
--------------------------------------------------------------------------------------
# send the rendered template
__file /etc/nginx/sites-available/test.conf \
--state present
--source "$__object/files/basic.conf"
--------------------------------------------------------------------------------
TESTING A NEW TYPE
------------------
If you want to test a new type on a node, you can tell cdist to only use an
object of this type: Use the '--initial-manifest' parameter
with - (stdin) as argument and feed object into stdin
of cdist:
--------------------------------------------------------------------------------
# Singleton type without parameter
echo __ungleich_munin_server | cdist --initial-manifest - munin.panter.ch
# Singleton type with parameter
echo __ungleich_munin_node --allow 1.2.3.4 | \
cdist --initial-manifest - rails-19.panter.ch
# Normal type
echo __file /tmp/stdintest --mode 0644 | \
cdist --initial-manifest - cdist-dev-01.ungleich.ch
--------------------------------------------------------------------------------
OTHER CONTENT IN CDIST REPOSITORY
---------------------------------
Usually the cdist repository contains all configuration
items. Sometimes you may have additional resources that
you would like to store in your central configuration
repositiory (like password files from KeepassX,
Libreoffice diagrams, etc.).
It is recommended to use a subfolder named "non-cdist"
in the repository for such content: It allows you to
easily distinguish what is used by cdist and what not
and also to store all important files in one
repository.
SEE ALSO
--------
@ -204,5 +243,5 @@ SEE ALSO
COPYING
-------
Copyright \(C) 2011-2012 Nico Schottelius. Free use of this software is
Copyright \(C) 2011-2013 Nico Schottelius. Free use of this software is
granted under the terms of the GNU General Public License version 3 (GPLv3).

6
docs/man/man7/cdist-bootstrap.text
View file

@ -64,11 +64,11 @@ So **2.0** is the latest version branch in this example.
All versions (2.0.x) within one version branch (2.0) are compatible to each
other and won't break your configuration when updating.
It's up to you decide on which branch you want to base your own work:
It's up to you to decide which branch you want to base your own work on:
master contains more recent changes, newer types, but may also break.
The versions branches are stable, but thus may miss the latest features.
The version branches are stable, but may lack the latest features.
Your decision can be changed later on, but may result in merge conflicts,
which you'd have to solve.
which you will need to solve.
Let's assume you want latest stuff and select the master branch as base for
your own work. Now it's time to create your branch, which contains your

2
docs/man/man7/cdist-hacker.text
View file

@ -51,7 +51,7 @@ work nor kill the authors brain:
- On a merge request, always name the branch I should pull from
- Always ensure **all** manpages build. Use **./build man** to test.
- If you developed more than **one** feature, consider submitting them in
seperate branches. This way one feature can already be included, even if
separate branches. This way one feature can already be included, even if
the other needs to be improved.
As soon as your work meets these requirements, write a mail

31
docs/man/man7/cdist-manifest.text
View file

@ -11,7 +11,7 @@ cdist-manifest - (Re-)Use types
DESCRIPTION
-----------
Manifests are used to define which objects to create.
Objects are instances of **types**, like in object orientated programming languages.
Objects are instances of **types**, like in object oriented programming languages.
An object is represented by the combination of
**type + slash + object name**: **__file/etc/cdist-configured** is an
object of the type ***__file*** with the name ***etc/cdist-configured***.
@ -25,8 +25,8 @@ the reference with pointers to the manpages.
Types in manifests are used like normal command line tools. Let's have a look
at an example:
--------------------------------------------------------------------------------
# Create object of type __package with the parameter state = removed
__package apache2 --state removed
# Create object of type __package with the parameter state = absent
__package apache2 --state absent
# Same with the __directory type
__directory /tmp/cdist --state present
@ -57,9 +57,9 @@ DEFINE STATE IN THE INITIAL MANIFEST
------------------------------------
The **initial manifest** is the entry point for cdist to find out, which
**objects** to configure on the selected host.
Cdist searches for the initial manifest at **cdist/conf/manifest/init**.
Cdist expects the initial manifest at **cdist/conf/manifest/init**.
Within this initial manifest, you define, which objects should be
Within this initial manifest you define, which objects should be
created on which host. To distinguish between hosts, you can use the
environment variable **__target_host**. Let's have a look at a simple
example:
@ -107,7 +107,7 @@ DEPENDENCIES
------------
If you want to describe that something requires something else, just
setup the variable "require" to contain the requirements. Multiple
requirements can be added white space seperated.
requirements can be added white space separated.
--------------------------------------------------------------------------------
# No dependency
@ -135,12 +135,12 @@ The initial manifest may for instance contain the following code:
--------------------------------------------------------------------------------
# Always create this file, so other sysadmins know cdist is used.
__file /etc/cdist-configured --type file
__file /etc/cdist-configured
case "$__target_host" in
my.server.name)
__file /root/bin/ --type directory
__file /etc/issue.net --type file --source "$__manifest/issue.net
__directory /root/bin/
__file /etc/issue.net --source "$__manifest/issue.net
;;
esac
--------------------------------------------------------------------------------
@ -148,9 +148,20 @@ esac
The manifest of the type "nologin" may look like this:
--------------------------------------------------------------------------------
__file /etc/nologin --type file --source "$__type/files/default.nologin"
__file /etc/nologin --source "$__type/files/default.nologin"
--------------------------------------------------------------------------------
This example makes use of dependencies:
--------------------------------------------------------------------------------
# Ensure that lighttpd is installed
__package lighttpd --state present
# Ensure that munin makes use of lighttpd instead of the default webserver
# package as decided by the package manager
require="__package/lighttpd" __package munin --state present
--------------------------------------------------------------------------------
SEE ALSO
--------

4
docs/man/man7/cdist-quickstart.text
View file

@ -72,7 +72,9 @@ As soon as you are able to login without password to localhost,
we can use cdist to configure it. You can copy and paste the following
code into your shell to get started and configure localhost:
--------------------------------------------------------------------------------
# Get cdist
# Get cdist
# Mirrors can be found on
# http://www.nico.schottelius.org/software/cdist/install/#index2h4
git clone git://git.schottelius.org/cdist
# Create manifest (maps configuration to host(s)

2
docs/man/man7/cdist-stages.text
View file

@ -33,7 +33,7 @@ be created, if it has different parameters.
STAGE 3: OBJECT INFORMATION RETRIEVAL
-------------------------------------
Every object is checked whether its type has explorers and if so, these are
executed on the target host. The results are transfered back
executed on the target host. The results are transferred back
and can be used in the following stages to decide what changes need to be made
on the target to implement the desired state.

63
docs/man/man7/cdist-troubleshooting.text Normal file
View file

@ -0,0 +1,63 @@
cdist-troubleshooting(7)
========================
Nico Schottelius <nico-cdist--@--schottelius.org>
NAME
----
cdist-troubleshooting - common problems and their solutions
ERROR IN MANIFEST IS NOT CONSIDERED AN ERROR BY CDIST
-----------------------------------------------------
Situation: You are executing other scripts from a manifest.
This script fails, but cdist does not recognise the error.
An example script would be something like this:
--------------------------------------------------------------------------------
% cat ~/.cdist/manifest/init
"$__manifest/special"
% cat ~/.cdist/manifest/special
#!/bin/sh
echo "Here is an unclean exiting script"
somecommandthatdoesnotexist
echo "I continue here although previous command failed"
--------------------------------------------------------------------------------
We can clearly see that **somecommandthatdoesnotexist**
will fail in ~/.cdist/manifest/special. But as the custom
script is not called with the -e flag (exit on failure) of shell,
it does not lead to an error. And thus cdist sees the exit 0
code of the last echo line instead of the failing command.
All scripts executed by cdist carry the -e flag.
To prevent the above from happening, there are three solutions available,
two of which can be used in the calling script:
--------------------------------------------------------------------------------
# Execute as before, but abort on failure
sh -e "$__manifest/special"
# Source the script in our namespace, runs in a set -e environment:
. "$__manifest/special"
--------------------------------------------------------------------------------
The third solution is to include a shebang header in every script
you write to use the -e flag:
--------------------------------------------------------------------------------
% cat ~/.cdist/manifest/special
#!/bin/sh -e
...
--------------------------------------------------------------------------------
SEE ALSO
--------
- cdist(1)
- cdist-tutorial(7)
COPYING
-------
Copyright \(C) 2013 Nico Schottelius. Free use of this software is
granted under the terms of the GNU General Public License version 3 (GPLv3).

22
docs/man/man7/cdist-type.text
View file

@ -67,25 +67,31 @@ A type consists of
Types are stored below cdist/conf/type/. Their name should always be prefixed with
two underscores (__) to prevent collisions with other executables in $PATH.
To begin a new type, just create the directory **cdist/conf/type/__NAME**.
To implement a new type, create the directory **cdist/conf/type/__NAME**.
DEFINING PARAMETERS
-------------------
Every type consists of required, optional and boolean parameters, which must
be created in a newline seperated file in ***parameter/required***,
each be declared in a newline separated file in ***parameter/required***,
***parameter/required_multiple***, ***parameter/optional***,
***parameter/optional_multiple*** and ***parameter/boolean***.
Parameters which are allowed multiple times should be listed in
required_multiple or optional_multiple respectively. For all other parameters
the standard unix behaviour of the last given wins is applied.
required_multiple or optional_multiple respectively. All other parameters
follow the standard unix behaviour "the last given wins".
If either is missing, the type will have no required, no optional, no boolean
or no parameters at all.
Default values for optional parameters can be predefined in
***parameter/default/<name>***.
Example:
--------------------------------------------------------------------------------
echo servername >> cdist/conf/type/__nginx_vhost/parameter/required
echo logdirectory >> cdist/conf/type/__nginx_vhost/parameter/optional
echo loglevel >> cdist/conf/type/__nginx_vhost/parameter/optional
mkdir cdist/conf/type/__nginx_vhost/parameter/default
echo warning > cdist/conf/type/__nginx_vhost/parameter/default/loglevel
echo server_alias >> cdist/conf/type/__nginx_vhost/parameter/optional_multiple
echo use_ssl >> cdist/conf/type/__nginx_vhost/parameter/boolean
--------------------------------------------------------------------------------
@ -108,6 +114,9 @@ if [ -f "$__object/parameter/logdirectory" ]; then
logdirectory="$(cat "$__object/parameter/logdirectory")"
fi
# optional parameter with predefined default
loglevel="$(cat "$__object/parameter/loglevel")"
# boolean parameter
if [ -f "$__object/parameter/use_ssl" ]; then
# file exists -> True
@ -125,7 +134,7 @@ fi
INPUT FROM STDIN
-----------------
----------------
Every type can access what has been written on stdin when it has been called.
The result is saved into the ***stdin*** file in the object directory.
@ -141,6 +150,7 @@ If you have not seen this syntax (<< eof) before, it may help you to read
about "here documents".
In the __file type, stdin is used as source for the file, if - is used for source:
--------------------------------------------------------------------------------
if [ -f "$__object/parameter/source" ]; then
source="$(cat "$__object/parameter/source")"
@ -229,7 +239,7 @@ the output of gencode-remote is executed on the target.
The gencode scripts can make use of the parameters, the global explorers
and the type specific explorers.
If the gencode scripts encounter an error, it should print diagnostic
If the gencode scripts encounters an error, it should print diagnostic
messages to stderr and exit non-zero. If you need to debug the gencode
script, you can write to stderr:

BIN
docs/speeches/2012-12-11_lisa.odp Normal file
View file

Binary file not shown.

10
docs/speeches/2013-01-23_panter.notes Normal file
View file

@ -0,0 +1,10 @@
sexy & sexy: ein glückliches Paar
inhalt vom vortrag
ziele von sexy und cdist
systemadministration hochgradig zu automatisieren
effizientes (tägliches) arbeiten

BIN
docs/speeches/2013-01-23_panter.odp Normal file
View file

Binary file not shown.

BIN
docs/speeches/2013-03-25_ad_novum.odp Normal file
View file

Binary file not shown.

BIN
docs/speeches/2013-10-05_ossawards.odp Normal file
View file

Binary file not shown.

BIN
docs/speeches/2013-10-05_ossawards/ossawards-2013-1sys.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 2.3 KiB

BIN
docs/speeches/2013-10-05_ossawards/ossawards-2013-1sys1comp.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 3.2 KiB

BIN
docs/speeches/2013-10-05_ossawards/ossawards-2013-1sys4comp.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 4 KiB

BIN
docs/speeches/2013-10-05_ossawards/ossawards-2013-4comp.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 2 KiB

BIN
docs/speeches/2013-10-05_ossawards/ossawards-2013-4sys.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 7.4 KiB

BIN
docs/speeches/2013-10-05_ossawards/ossawards-2013-sys-cdist.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 14 KiB

BIN
docs/speeches/2013-10-05_ossawards/ossawards-2013-sys-chairman.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 9.5 KiB

BIN
docs/speeches/2013-10-05_ossawards/ossawards-2013.xoj Normal file
View file

Binary file not shown.

BIN
docs/speeches/2013-11-22_eth_linux_erfa.odp Normal file
View file

Binary file not shown.

BIN
docs/speeches/2013-11-22_eth_linux_erfa.pdf Normal file
View file

Binary file not shown.

2
docs/web/cdist/documentation.mdwn
View file

@ -4,4 +4,6 @@ You can browse the latest
[latest version of the manpages](/software/cdist/man/latest) or
have a look at [all versions](/software/cdist/man).
You can also view [speeches about cdist](/software/cdist/speeches).
[[!tag cdist unix]]

18
docs/web/cdist/install.mdwn
View file

@ -10,7 +10,7 @@ This is the machine you use to configure the target hosts.
* /bin/sh: A posix like shell (for instance bash, dash, zsh)
* Python >= 3.2
* SSH client
* Asciidoc (for building the manpages)
* Asciidoc and xsltproc (for building the manpages)
### Target Hosts
@ -45,21 +45,25 @@ For Debian **wheezy** or newer:
On **squeeze** you can add following line in **/etc/apt/sources.list**
deb http://ftp.debian.org/debian wheezy main
deb http://ftp.debian.org/debian wheezy main
And add pinning entry in **/etc/apt/preferences.d/wheezy**:
Package: *
Pin: release n=wheezy
Pin-Priority: 1
Package: *
Pin: release n=wheezy
Pin-Priority: 1
Please be aware that both **openssh-server** and **openssh-client** might be
removed on **python3.2** installation. You surely want to reinstall them:
apt-get install -t wheezy openssh-server openssh-client
apt-get install -t wheezy openssh-server openssh-client
For older Debian versions, installing python 3.2 from source is required.
If you want to build the cdist manpages:
aptitude install --without-recommends asciidoc xsltproc
### Fedora
Fedora 15 and newer includes a recent python.
@ -139,7 +143,7 @@ To install cdist, execute the following commands:
If you want to build and use the manpages, run:
./build man
make man
export MANPATH=$MANPATH:$(pwd -P)/doc/man
#### Available versions in git

57
docs/web/cdist/update.mdwn
View file

@ -14,13 +14,67 @@ If you stay on a version branche (i.e. 1.0, 1.1., ...), nothing should break.
The master branch on the other hand is the development branch and may not be
working, break your setup or eat the tree in your garden.
### Safely upgrading to new versions
To upgrade to **any** further cdist version, you can take the
following procedure to do a safe upgrade:
# Create new branch to try out the update
git checkout -b upgrade_cdist
# Get latest cdist version in git database
git fetch -v
# see what will happen on merge - replace
# master with the branch you plan to merge
git diff upgrade_cdist..origin/master
# Merge the new version
git merge origin/master
Now you can ensure all custom types work with the new version.
Assume that you need to go back to an older version during
the migration/update, you can do so as follows:
# commit changes
git commit -m ...
# go back to original branch
git checkout master
After that, you can go back and continue the upgrade:
# git checkout upgrade_cdist
## Update The Python Package
To upgrade to the lastet version do
pip install --upgrade cdist
## Update Instructions
## General Update Instructions
### Updating from 2.2 to 2.3
No incompatiblities.
### Updating from 2.1 to 2.2
Starting with 2.2, the syntax for requiring a singleton type changed:
Old format:
require="__singleton_type/singleton" ...
New format:
require="__singleton_type" ...
Internally the "singleton" object id was dropped to make life more easy.
You can probably fix your configuration by running the following code
snippet (currently untested, please report back if it works for you):
find ~/.cdist/* -type f -exec sed -i 's,/singleton,,' {} \;
### Updating from 2.0 to 2.1
@ -46,7 +100,6 @@ Have a look at the update guide for [[2.0 to 2.1|2.0-to-2.1]].
* Type **\_\_user**: Parameter --groups removed (use the new \_\_user_groups type)
* Type **\_\_ssh_authorized_key** has been replaced by more flexible type
**\_\_ssh_authorized_keys**
* require="" is deprecated: Use --after and --before as parameters instead
### Updating from 1.7 to 2.0

3
docs/web/cdist/why.mdwn
View file

@ -42,7 +42,8 @@ in almost all cases all dependencies are usually fulfilled.
Cdist does not require an agent or a high level programming
languages on the target host: it will run on any host that
has a **ssh server running** and a posix compatible shell
(**/bin/sh**).
(**/bin/sh**). Compared to other configuration management systems,
it does not require to open up an additional port.
## Push based distribution