Merge branch 'master' into preos

docs/changelog

@@ -5,6 +5,32 @@ Changelog
 	* Exception: No braces means author == Nico Schottelius
 
 
+3.1.10: 2015-01-06
+	* Core: Fix too many open files bug (#343)
+	* Type __ssh_authorized_keys: Remove unneeded explorer (Steven Armstrong)
+	* Type __ssh_authorized_keys: Fix empty output bug of entry explorer (Steven Armstrong)
+	* Type __package_apt: Add support for --target-release
+	* Type __locale: Add support for Ubuntu
+	* Documentation: Fix typo in maintainer file (Stephan Kulla)
+
+3.1.9: 2014-10-17
+	* Type __package_emerge: Fix handling of slotted packages (Daniel Heule)
+	* Type __package_apt: Use --force-confdef (Ricardo Catalinas Jiménez)
+	* Type __package_update_index: Decrease verbosity (Ricardo Catalinas Jiménez)
+	* Type __package_upgrade_all: Decrease verbosity (Ricardo Catalinas Jiménez)
+
+3.1.8: 2014-10-01
+	* New Type: __package_update_index (Ricardo Catalinas Jiménez)
+	* New Type: __package_upgrade_all  (Ricardo Catalinas Jiménez)
+
+3.1.7: 2014-09-29
+	* Type __cdistmarker: Fix typo (Ricardo Catalinas Jiménez)
+	* Core: Bugfix: Export messaging to manifests (Ricardo Catalinas Jiménez)
+	* Explorer cpu_cores, cpu_sockets, memory: Add Mac OS X support (Manuel Hutter)
+	* Type __ssh_authorized_keys: Ensure keys are correctly added (Steven Armstrong)
+	* New Type: __ssh_authorized_key (Steven Armstrong)
+	* New Type: __package_pkgng_freebsd (Jake Guffey)
+
 3.1.6: 2014-08-18
 	* New Type: __ssh_dot_ssh
 	* Type __package_yum: Support retrieving package via URL

docs/dev/logs/2014-09-10.preos-keys

@@ -0,0 +1,18 @@
+
+cdist preos keyramfs --keyfile --keyfile
+
+[17:51] freiheit:vcs% mkdir preos-keys
+[17:51] freiheit:vcs% mkdir -p preos-keys/root/.ssh 
+[17:56] freiheit:vcs% 
+
+chown root:root -R preos-keys/
+chmod 0600 preos-keys/root/.ssh/authorized_keys 
+
+chmod 0700 preos-keys/root/
+chmod 0700 preos-keys/root/.ssh/
+
+[18:20:17] freiheit:/home/users/nico/.ungleich/ungleich/vcs/preos-keys# find . | cpio -H newc -o | gzip -9 >  ../initramfs.cpio.gz
+4 blocks
+[18:21:08] freiheit:/home/users/nico/.ungleich/ungleich/vcs/preos-keys# 
+
+

docs/dev/logs/2014-11-11.discussion

@@ -0,0 +1,104 @@
+- logging/cache destination
+    local:
+        ~/.cdist/log/by-host/$__target/host/config/YYYY/MM/DD/hhmmss/
+        ~/.cdist/log/by-session/YYYY/MM/DD/hhmmss/$__target_host/
+            config/
+            install/
+            export/
+    remote:
+        /var/lib/cdist/YYYY-MM-DD-hhmmss-$sourcehost.$pid
+
+    rm old directories on remote side
+
+- support for tags
+    - for partial configuration
+    - supports also install
+    - on object definition, define
+        - a) don't care (i.e. no tags)
+        - b) require tag (only if this tag is setup)
+        - c) require not tag (only if this tag is *not* setup)
+        - d) what if both given (conflicting)
+    - names for parameters:
+        - cdist config / cdist type
+            --if-tag / --not-if-tag / --require-tag
+            --not-if-tag
+
+- logging
+    - command line
+    - stdin of cdist
+    - stdout/stderr/stdin of types
+        - new: stdout/stderr
+    - initial manifest
+        - if coming from stdin
+
+- logging configurable
+    - to be discussed
+
+- sudo remote
+    - cp to tmp & mv
+    - umask issue?
+
+- install
+    - via tagged types
+    - 
+
+- export
+    - one /export script per type
+    - exports of type running after object's code is done
+    - global export should also exist after everything
+    - PR & merge
+    - change DONE status to CODE_DONE
+    - introduce EXPORT_DONE
+
+
+- preos
+    - merge with debian support only
+    - we are open to support --os-type later
+
+- stackable remotes
+    - change API for remote_exec and remote_copy
+    - new minor version
+    - PR & merge
+
+- locking
+    - optional
+    - remote lock
+    - based on $(ls /var/lib/cdist/) > 0
+
+- ideas for parallelisation
+    - run explorer in parallel
+        - type
+        - object
+    - objects without dependencies can be run in parallel
+
+- connection test
+    - just implement
+
+- multi user environment
+    - not really needed [at the moment]
+    - can be implemented by 
+        - git branches
+        - setting the output dir
+
+- python2 support with __future__
+    - steven votes against it
+    - nico does not care too much to object
+
+
+- pull based
+    - sshd / stdin + stdout
+    - use Use ProxyCommand with stdin/stdout
+        - http://www.nico.schottelius.org/blog/openssh-6.2-add-callback-functionality-using-dynamic-remote-port-forwarding/
+
+    - cdist grant-pull-access <targethost>
+        - generate user
+        - ssh pubkeypair
+    - call wraper script on targethost
+        - it is shell!
+        - ssh cdistuser@controlhost 
+
+- config replay/redo/undo
+    - not now
+
+
+- have a new discussion about handling uris

docs/dev/logs/dependency-tree-with-fruits

@@ -0,0 +1,29 @@
+1. Manifests need type explorers to be run before
+    Only this way the manifest can use the values
+
+2. Type explorers can depend on the (successful) execution of other objects
+    A type explorer may lookup values (ownership, size, processes, etc.)
+    that are depending on the code of other objects.
+
+3. The 'require="a" b' dependency specifies only that execution of b needs to wait until a is finished
+    Other requirements given at another position may not be
+    executed before b, as cdist interprets require="a" as 
+    "a is everything that is needed to realise b"
+ 
+4. The 'require' state is thus *not* suitable to trigger reloads
+    A type b, which uses the require="" statement to wait for all
+    require="" dependencies, may be executed in an incomplete
+    status.
+
+5. Depending on multiple dependencies defined at different locations requires all objects to be present
+    Because otherwise the dependent object cannot be sure that the
+    list of objects it waits for is complete.
+
+6. If a statement requires all objects to be present, all manifests must have run already.
+    If all manifests have run already, also all type explorers have run already (#1).
+    If all type explorers have run already, some objects may have been finished completly already (#2).
+
+7. Because of #4 and #6, we need to introduce another dependency that allows reloading after
+
+8. Because the object could be finished already due to a 'require' dependency, we need another script
+    Maybe "gen-post-run-local", "gen-post-run-remote"

docs/man/cdist-reference.text.sh

@@ -203,10 +203,10 @@ __global::
     Directory that contains generic output like explorer.
     Available for: initial manifest, type manifest, type gencode, shell
 __messages_in::
-    File to read messages from
+    File to read messages from.
     Available for: initial manifest, type manifest, type gencode
 __messages_out::
-    File to write messages
+    File to write messages.
     Available for: initial manifest, type manifest, type gencode
 __object::
     Directory that contains the current object.

docs/man/man7/cdist-explorer.text

@@ -50,7 +50,7 @@ else
    name="$__object_id"
 fi
 
-# Except dpkg failing, if package is not known / installed
+# Expect dpkg failing, if package is not known / installed
 dpkg -s "$name" 2>/dev/null || exit 0
 --------------------------------------------------------------------------------
 
@@ -64,5 +64,5 @@ SEE ALSO
 
 COPYING
 -------
-Copyright \(C) 2010-2012 Nico Schottelius. Free use of this software is
+Copyright \(C) 2010-2014 Nico Schottelius. Free use of this software is
 granted under the terms of the GNU General Public License version 3 (GPLv3).

docs/man/man7/cdist-manifest.text

@@ -110,6 +110,7 @@ setup the variable "require" to contain the requirements. Multiple
 requirements can be added white space separated.
 
 --------------------------------------------------------------------------------
+
  1 # No dependency
  2 __file /etc/cdist-configured
  3 
@@ -121,21 +122,43 @@ requirements can be added white space separated.
  9 require="__file/etc/cdist-configured __link/tmp/cdist-testfile" \
 10    __file /tmp/cdist-another-testfile
 
-
 --------------------------------------------------------------------------------
 
 Above the "require" variable is only set for the command that is 
-immediately following it. Dependencies should allways be declared that way.
+immediately following it. Dependencies should always be declared that way.
 
-On line 4 you can see that the instantion of a type "__link" object needs
+On line 4 you can see that the instantion of a type "\__link" object needs
 the object "__file/etc/cdist-configured" to be present, before it can proceed.
 
-This also means that the "__link" command must make sure, that either
-"__file/etc/cdist-configured" allready is present, or, if it's not, it needs
+This also means that the "\__link" command must make sure, that either
+"\__file/etc/cdist-configured" allready is present, or, if it's not, it needs
 to be created. The task of cdist is to make sure, that the dependency will be
-resolved appropriately and thus "__file/etc/cdist-configured" be created
+resolved appropriately and thus "\__file/etc/cdist-configured" be created
 if necessary before "__link" proceeds (or to abort execution with an error).
 
+If you really need to make all types depend on a common dependency, you can
+export the "require" variable as well. But then, if you need to add extra
+dependencies to a specific type, you have to make sure that you append these
+to the globally already defined one.
+
+--------------------------------------------------------------------------------
+
+# First of all, update the package index
+__package_update_index
+# Upgrade all the installed packages afterwards
+require="__package_update_index" __package_upgrade_all
+# Create a common dependency for all the next types so that they get to
+# be executed only after the package upgrade has finished
+export require="__package_upgrade_all"
+
+# Ensure that lighttpd is installed after we have upgraded all the packages
+__package lighttpd --state present
+# Ensure that munin is installed after lighttpd is present and after all
+# the packages are upgraded
+require="$require __package/lighttpd" __package munin --state present
+
+--------------------------------------------------------------------------------
+
 All objects that are created in a type manifest are automatically required
 from the type that is calling them. This is called "autorequirement" in
 cdist jargon.

docs/man/man7/cdist-type.text

@@ -100,7 +100,7 @@ echo use_ssl >> cdist/conf/type/__nginx_vhost/parameter/boolean
 USING PARAMETERS
 ----------------
 The parameters given to a type can be accessed and used in all type scripts
-(e.g manifest, gencode-*, explorer/*). Note that boolean parameters are
+(e.g manifest, gencode, explorer). Note that boolean parameters are
 represented by file existence. File exists -> True,
 file does not exist -> False
 
@@ -281,7 +281,7 @@ on the target, there must be another type that provides this tool and the first
 type should create an object of the specific type.
 
 If your type wants to save temporary data, that may be used by other types
-later on (for instance __file), you can save them in the subdirectory
+later on (for instance \__file), you can save them in the subdirectory
 "files" below $__object (but you must create it yourself).
 cdist will not touch this directory.
 

docs/web/cdist/install.mdwn

@@ -137,7 +137,7 @@ immediately.
 
 To install cdist, execute the following commands:
 
-    git clone git://git.schottelius.org/cdist
+    git clone https://github.com/telmich/cdist.git
     cd cdist
     export PATH=$PATH:$(pwd -P)/bin