From 765c5bb50cd6cc13993e1f5a761eb31f851920b7 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sat, 8 Dec 2012 13:43:18 -0800 Subject: [PATCH 01/37] insert date for 2.1.0 Signed-off-by: Nico Schottelius --- docs/changelog | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelog b/docs/changelog index 4e0a5d99..d61af31e 100644 --- a/docs/changelog +++ b/docs/changelog @@ -4,7 +4,7 @@ Changelog * Changes are always commented with their author in (braces) * Exception: No braces means author == Nico Schottelius -2.1.0: +2.1.0: 2012-12-08 * Core: Ensure global explorers are executable * Core: Ensure type explorers are executable (Steven Armstrong) * New Type: __git From f64aaf74b7f17ea862ecf2787ebd61a5b8c4aec4 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sat, 8 Dec 2012 13:44:02 -0800 Subject: [PATCH 02/37] +quotes Signed-off-by: Nico Schottelius --- build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build b/build index d6c9a172..572fb768 100755 --- a/build +++ b/build @@ -341,7 +341,7 @@ eof web) set -e - $0 web-doc + "$0" web-doc # Fix ikiwiki, which does not like symlinks for pseudo security ssh tee.schottelius.org \ "cd /home/services/www/nico/www.nico.schottelius.org/www/software/cdist/man && From d12645f23041cc28bdc4cc53cbec5e945d1c6bb8 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sat, 8 Dec 2012 13:53:11 -0800 Subject: [PATCH 03/37] not an Signed-off-by: Nico Schottelius --- docs/web/cdist/why.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/web/cdist/why.mdwn b/docs/web/cdist/why.mdwn index 0fcdf5c5..6dcfd441 100644 --- a/docs/web/cdist/why.mdwn +++ b/docs/web/cdist/why.mdwn @@ -41,7 +41,7 @@ Cdist requires very litte on a target system. Even better, in almost all cases all dependencies are usually fulfilled. Cdist does not require an agent or a high level programming languages on the target host: it will run on any host that -has an **ssh server running** and a posix compatible shell +has a **ssh server running** and a posix compatible shell (**/bin/sh**). ## Push based distribution From dc41c7a9e6437dc71a88ea456311b8658e7c89a0 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sat, 8 Dec 2012 14:08:01 -0800 Subject: [PATCH 04/37] reference the upgrade guide for 2.0 to 2.1 Signed-off-by: Nico Schottelius --- docs/web/cdist/update.mdwn | 2 + docs/web/cdist/update/2.0-to-2.1.mdwn | 118 ++++++++++++++++++++++++++ 2 files changed, 120 insertions(+) create mode 100644 docs/web/cdist/update/2.0-to-2.1.mdwn diff --git a/docs/web/cdist/update.mdwn b/docs/web/cdist/update.mdwn index a50e7224..a1120db0 100644 --- a/docs/web/cdist/update.mdwn +++ b/docs/web/cdist/update.mdwn @@ -24,6 +24,8 @@ To upgrade to the lastet version do ### Updating from 2.0 to 2.1 +Have a look at the upgrade guide for [[2.0 to 2.1|2.0-to-2.1]]. + * Type **\_\_package* and \_\_process** use --state **present** or **absent**. The states **removed/installed** and **stopped/running** have been removed. Support for the new states is already present in 2.0. diff --git a/docs/web/cdist/update/2.0-to-2.1.mdwn b/docs/web/cdist/update/2.0-to-2.1.mdwn new file mode 100644 index 00000000..6fb072b9 --- /dev/null +++ b/docs/web/cdist/update/2.0-to-2.1.mdwn @@ -0,0 +1,118 @@ +[[!meta title="Update Guide for 2.0 to 2.1"]] + +## Introduction + +When changing your installation from 2.0 to 2.1, there are +a lot of changes coming up. 2.1 is mainly a cleanup release, +which removes long time deprecated behaviour, but also makes +a lot of things more consistent and allows you to split off your types, +explorers and manifest to custom directories. + +This document will guide you to a successful update. + +## Preperation + +As for every software and system you use in production, you should first of +all make a backup of your data. To prevent any breakage, it is +recommended to create a new git branch to do the update on: + + % git checkout -b update_to_2.1 + +This also ensure that whenever you need to do a change in your +2.0 based tree, you can simply go back to that branch, apply the change +and configure your systems - independently of your update progress! + +Next fetch the latest upstream changes, I assume that +origin refers to one of the upstream mirrors (change origin if you use +another remote name for upstream cdist): + + % git fetch -v origin + +## Merge the changes + +Now try to merge upstream into the new branch. + + % git merge origin/2.1 + +Fix any conflicts that may have been occurred due to local changes +and then **git add** and *git commit** those changes. This should seldomly +occur and if, it's mostly for people hacking on the cdist core. + +## Move "conf" directory + +One of the biggest changes in cdist 2.1 is that you can have multiple +**conf** directories: Indeed, the new default behaviour of cdist is to +search for conf directories + + * below the python module (cdist/conf in the source tree or in the installed location) + * at ~/.cdist/ (on conf suffix there) + +So you can now choose, where to store your types. + +### Integrate your conf/ back into the tree + +If you choose to store your types together with the upstream types, +you can just move all your stuff below **cdist/conf**: + + % git mv conf/type/* cdist/conf/type + % git mv conf/manifest/* cdist/conf/manifest + % git mv conf/explorer/* cdist/conf/explorer + % git commit -m "Re-Integrate my conf directory into cdist 2.1 tree" + +### Move your conf/ directory to ~/.cdist + +If you want to store your site specific +configuration outside of the cdist tree, you +can move your conf/ directory to your homedirectory ($HOME) under ~/.cdist: + + % mv conf ~/.cdist + % git rm -r conf + % git commit -m "Move my conf directory to ~/.cdist" + +It it still recommended to use a version control system like git in it: + + % cd ~/.cdist + % git init + % git add . + % git commit -m "Create new git repository containing my cdist configuration" + +## Test the migration + +Some of the types shipped with upstream were changed, so you may want to test +the result by running cdist on one of your staging target hosts: + + % ./bin/cdist config -v staging-host + +All incompatibilities are listed on the [[cdist update page|software/cdist/update]], +so you can browse through the list and update your configuration. + +## Final Cleanups + +When everything is tested, there are some cleanups to be done to finalise the update. + +### When continuing to keep conf/ in the tree + +You can then merge back your changes into the master tree and continue to work +as normal. + +### When using ~/.cdist + +If you decided to move your site specific code to ~/.cdist, you can now switch your +**master** branch or version branch to upstream directly. Assumnig you are in the +cdist directory, having your previous branch checked out, you can create a clean +state using the following commands: + + % upstream_branch=2.1 + % current_branch=$(git rev-parse --abbrev-ref HEAD) + % git checkout -b archive_my_own_tree + % git branch -D "$current_branch" + % git checkout -b "$current_branch" "origin/$upstream_branch" + +Afther these commands, your previous main branch is accessible at +**archive_my_own_tree** and your branch is now tracking upstream. + +## Questions? Critics? Hint?? + +If you think this manual helped or misses some information, do not +hesitate to contact us on any of the usual ways (irc, mailinglist, +github issue tracker, ...). From 98e86ae345a65fbb8519aa2ca082818d8fd6ccac Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sat, 8 Dec 2012 14:15:03 -0800 Subject: [PATCH 05/37] update, not grade Signed-off-by: Nico Schottelius --- docs/web/cdist/update.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/web/cdist/update.mdwn b/docs/web/cdist/update.mdwn index a1120db0..e486dff9 100644 --- a/docs/web/cdist/update.mdwn +++ b/docs/web/cdist/update.mdwn @@ -24,7 +24,7 @@ To upgrade to the lastet version do ### Updating from 2.0 to 2.1 -Have a look at the upgrade guide for [[2.0 to 2.1|2.0-to-2.1]]. +Have a look at the update guide for [[2.0 to 2.1|2.0-to-2.1]]. * Type **\_\_package* and \_\_process** use --state **present** or **absent**. The states **removed/installed** and **stopped/running** have been removed. From dc277b10c7602b8f3460776708af49d59a930aae Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sat, 8 Dec 2012 14:15:28 -0800 Subject: [PATCH 06/37] -typo Signed-off-by: Nico Schottelius --- docs/web/cdist/update/2.0-to-2.1.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/web/cdist/update/2.0-to-2.1.mdwn b/docs/web/cdist/update/2.0-to-2.1.mdwn index 6fb072b9..1d0037ab 100644 --- a/docs/web/cdist/update/2.0-to-2.1.mdwn +++ b/docs/web/cdist/update/2.0-to-2.1.mdwn @@ -111,7 +111,7 @@ state using the following commands: Afther these commands, your previous main branch is accessible at **archive_my_own_tree** and your branch is now tracking upstream. -## Questions? Critics? Hint?? +## Questions? Critics? Hints? If you think this manual helped or misses some information, do not hesitate to contact us on any of the usual ways (irc, mailinglist, From 55aad252001064f534d83aa4bb7e8674e16d5a61 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 9 Dec 2012 08:34:29 -0800 Subject: [PATCH 07/37] almost the eight Signed-off-by: Nico Schottelius --- docs/changelog | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelog b/docs/changelog index d61af31e..e7f77870 100644 --- a/docs/changelog +++ b/docs/changelog @@ -4,7 +4,7 @@ Changelog * Changes are always commented with their author in (braces) * Exception: No braces means author == Nico Schottelius -2.1.0: 2012-12-08 +2.1.0: 2012-12-09 * Core: Ensure global explorers are executable * Core: Ensure type explorers are executable (Steven Armstrong) * New Type: __git From 67c7351bec864fb65d294883698f2beb567fb02e Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 9 Dec 2012 11:49:26 -0800 Subject: [PATCH 08/37] install the right package, not pyro... Signed-off-by: Nico Schottelius --- cdist/conf/type/__package_pip/gencode-remote | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cdist/conf/type/__package_pip/gencode-remote b/cdist/conf/type/__package_pip/gencode-remote index 3456ced2..ec1c89f8 100644 --- a/cdist/conf/type/__package_pip/gencode-remote +++ b/cdist/conf/type/__package_pip/gencode-remote @@ -46,10 +46,10 @@ fi case "$state_should" in present) - echo $pip install -q pyro + echo $pip install -q "$name" ;; absent) - echo $pip uninstall -q -y pyro + echo $pip uninstall -q -y "$name" ;; *) echo "Unknown state: $state_should" >&2 From e4c830ebca34555aa3e11b3fd1a32013458fb3e5 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 9 Dec 2012 11:50:03 -0800 Subject: [PATCH 09/37] update changes Signed-off-by: Nico Schottelius --- docs/changelog | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/changelog b/docs/changelog index e7f77870..a255f42b 100644 --- a/docs/changelog +++ b/docs/changelog @@ -16,6 +16,7 @@ Changelog * Type __jail: Change optional parameter "started" to boolean "stopped" parameter, change optional parameter "devfs-enable" to boolean "devfs-disable" parameter and change optional parameter "onboot" to boolean. + * Type __package_pip: Bugfix: Installeded the package, not pyro * Remove Type __ssh_authorized_key: Superseeded by __ssh_authorized_keys * Support for CDIST_PATH (Steven Armstrong) From 224157610597d657e769f1d65c08f3fa53d4f0bd Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 9 Dec 2012 14:26:11 -0800 Subject: [PATCH 10/37] document $HOME/.cdist Signed-off-by: Nico Schottelius --- docs/man/cdist-reference.text.sh | 48 +++++++++++++++++++++----------- 1 file changed, 32 insertions(+), 16 deletions(-) diff --git a/docs/man/cdist-reference.text.sh b/docs/man/cdist-reference.text.sh index 8e3f49a2..8d230028 100755 --- a/docs/man/cdist-reference.text.sh +++ b/docs/man/cdist-reference.text.sh @@ -59,62 +59,78 @@ cat << eof PATHS ----- -If not specified otherwise, all paths are relative to the checkout directory. +$HOME/.cdist:: + The standard cdist configuration directory relative to your home directory + This is usually the place you want to store your site specific configuration -conf/:: - Contains the (static) configuration like manifests, types and explorers. +cdist/conf/:: + The distribution configuration directory + This contains types and explorers to be used -conf/manifest/init:: +confdir:: + Cdist will use all available configuration directories and create + a temporary confdir containing links to the real configuration directories. + This way it is possible to merge configuration directories. + + By default it consists of everything in $HOME/.cdist and cdist/conf/. + + For more details see cdist(1) + +confdir/manifest/init:: This is the central entry point. It is an executable (+x bit set) shell script that can use values from the explorers to decide which configuration to create for the specified target host. Its intent is to used to define mapping from configurations to hosts. -conf/manifest/*:: +confdir/manifest/*:: All other files in this directory are not directly used by cdist, but you can seperate configuration mappings, if you have a lot of code in the conf/manifest/init file. This may also be helpful to have different admins maintain different groups of hosts. -conf/explorer/:: +confdir/explorer/:: Contains explorers to be run on the target hosts, see cdist-explorer(7). -conf/type/:: +confdir/type/:: Contains all available types, which are used to provide some kind of functionality. See cdist-type(7). -conf/type//:: +confdir/type//:: Home of the type . This directory is referenced by the variable __type (see below). -conf/type//man.text:: +confdir/type//man.text:: Manpage in Asciidoc format (required for inclusion into upstream) -conf/type//manifest:: +confdir/type//manifest:: Used to generate additional objects from a type. -conf/type//gencode-local:: +confdir/type//gencode-local:: Used to generate code to be executed on the source host -conf/type//gencode-remote:: +confdir/type//gencode-remote:: Used to generate code to be executed on the target host -conf/type//parameter/required:: +confdir/type//parameter/required:: Parameters required by type, \n seperated list. -conf/type//parameter/optional:: +confdir/type//parameter/optional:: Parameters optionally accepted by type, \n seperated list. -conf/type//parameter/boolean:: +confdir/type//parameter/boolean:: Boolean parameters accepted by type, \n seperated list. -conf/type//explorer:: +confdir/type//explorer:: Location of the type specific explorers. This directory is referenced by the variable __type_explorer (see below). See cdist-explorer(7). +confdir/type//files:: + This directory is reserved for user data and will not be used + by cdist at any time + out/:: This directory contains output of cdist and is usually located in a temporary directory and thus will be removed after the run. From dd9083327d8dfb298c3ad507190e70e15e19371c Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 9 Dec 2012 14:41:50 -0800 Subject: [PATCH 11/37] update confdir paths Signed-off-by: Nico Schottelius --- docs/man/cdist-reference.text.sh | 4 ++-- docs/man/man1/cdist.text | 2 +- docs/man/man7/cdist-hacker.text | 2 +- docs/man/man7/cdist-manifest.text | 8 ++++---- docs/man/man7/cdist-type.text | 18 +++++++++--------- 5 files changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/man/cdist-reference.text.sh b/docs/man/cdist-reference.text.sh index 8d230028..d8c23a79 100755 --- a/docs/man/cdist-reference.text.sh +++ b/docs/man/cdist-reference.text.sh @@ -59,7 +59,7 @@ cat << eof PATHS ----- -$HOME/.cdist:: +\$HOME/.cdist:: The standard cdist configuration directory relative to your home directory This is usually the place you want to store your site specific configuration @@ -72,7 +72,7 @@ confdir:: a temporary confdir containing links to the real configuration directories. This way it is possible to merge configuration directories. - By default it consists of everything in $HOME/.cdist and cdist/conf/. + By default it consists of everything in \$HOME/.cdist and cdist/conf/. For more details see cdist(1) diff --git a/docs/man/man1/cdist.text b/docs/man/man1/cdist.text index c52e3696..113454a7 100644 --- a/docs/man/man1/cdist.text +++ b/docs/man/man1/cdist.text @@ -78,7 +78,7 @@ EXAMPLES # Configure ikq05.ethz.ch with debug enabled cdist config -d ikq05.ethz.ch -# Configure hosts in parallel and use a different home directory +# Configure hosts in parallel and use a different configuration directory cdist config -c ~/p/cdist-nutzung \ -p ikq02.ethz.ch ikq03.ethz.ch ikq04.ethz.ch diff --git a/docs/man/man7/cdist-hacker.text b/docs/man/man7/cdist-hacker.text index ee88ca29..d0f9a399 100644 --- a/docs/man/man7/cdist-hacker.text +++ b/docs/man/man7/cdist-hacker.text @@ -44,7 +44,7 @@ work nor kill the authors brain: - All files should contain the usual header (Author, Copying, etc.) - Code submission must be done via git -- Do not add conf/manifest/init - This file should only be touched in your +- Do not add cdist/conf/manifest/init - This file should only be touched in your private branch! - Code to be included should be branched of the upstream "master" branch - Exception: Bugfixes to a version branch diff --git a/docs/man/man7/cdist-manifest.text b/docs/man/man7/cdist-manifest.text index b9dfe655..19f6053e 100644 --- a/docs/man/man7/cdist-manifest.text +++ b/docs/man/man7/cdist-manifest.text @@ -16,8 +16,8 @@ An object is represented by the combination of **type + slash + object name**: **__file/etc/cdist-configured** is an object of the type ***__file*** with the name ***etc/cdist-configured***. -All available types can be found in the **conf/type/** directory, -use **ls conf/type** to get the list of available types. If you have +All available types can be found in the **cdist/conf/type/** directory, +use **ls cdist/conf/type** to get the list of available types. If you have setup the MANPATH correctly, you can use **man cdist-reference** to access the reference with pointers to the manpages. @@ -57,7 +57,7 @@ DEFINE STATE IN THE INITIAL MANIFEST ------------------------------------ The **initial manifest** is the entry point for cdist to find out, which **objects** to configure on the selected host. -Cdist searches for the initial manifest at **conf/manifest/init**. +Cdist searches for the initial manifest at **cdist/conf/manifest/init**. Within this initial manifest, you define, which objects should be created on which host. To distinguish between hosts, you can use the @@ -88,7 +88,7 @@ command. SPLITTING UP THE INITIAL MANIFEST --------------------------------- If you want to split up your initial manifest, you can create other shell -scripts in **conf/manifest/** and include them in **conf/manifest/init**. +scripts in **cdist/conf/manifest/** and include them in **cdist/conf/manifest/init**. Cdist provides the environment variable ***__manifest*** to reference to the directory containing the initial manifest (see cdist-reference(7)). diff --git a/docs/man/man7/cdist-type.text b/docs/man/man7/cdist-type.text index a5064f91..54b67be5 100644 --- a/docs/man/man7/cdist-type.text +++ b/docs/man/man7/cdist-type.text @@ -64,10 +64,10 @@ A type consists of - explorer (optional) - gencode (optional) -Types are stored below conf/type/. Their name should always be prefixed with +Types are stored below cdist/conf/type/. Their name should always be prefixed with two underscores (__) to prevent collisions with other executables in $PATH. -To begin a new type, just create the directory **conf/type/__NAME**. +To begin a new type, just create the directory **cdist/conf/type/__NAME**. DEFINING PARAMETERS @@ -84,10 +84,10 @@ or no parameters at all. Example: -------------------------------------------------------------------------------- -echo servername >> conf/type/__nginx_vhost/parameter/required -echo logdirectory >> conf/type/__nginx_vhost/parameter/optional -echo server_alias >> conf/type/__nginx_vhost/parameter/optional_multiple -echo use_ssl >> conf/type/__nginx_vhost/parameter/boolean +echo servername >> cdist/conf/type/__nginx_vhost/parameter/required +echo logdirectory >> cdist/conf/type/__nginx_vhost/parameter/optional +echo server_alias >> cdist/conf/type/__nginx_vhost/parameter/optional_multiple +echo use_ssl >> cdist/conf/type/__nginx_vhost/parameter/boolean -------------------------------------------------------------------------------- @@ -98,7 +98,7 @@ The parameters given to a type can be accessed and used in all type scripts represented by file existence. File exists -> True, file does not exist -> False -Example: (e.g. in conf/type/__nginx_vhost/manifest) +Example: (e.g. in cdist/conf/type/__nginx_vhost/manifest) -------------------------------------------------------------------------------- # required parameter servername="$(cat "$__object/parameter/servername")" @@ -129,7 +129,7 @@ INPUT FROM STDIN Every type can access what has been written on stdin when it has been called. The result is saved into the ***stdin*** file in the object directory. -Example use of a type: (e.g. in conf/type/__archlinux_hostname) +Example use of a type: (e.g. in cdist/conf/type/__archlinux_hostname) -------------------------------------------------------------------------------- __file /etc/rc.conf --source - << eof ... @@ -186,7 +186,7 @@ mark it as a singleton: Just create the (empty) file "singleton" in your type directory: -------------------------------------------------------------------------------- -touch conf/type/__NAME/singleton +touch cdist/conf/type/__NAME/singleton -------------------------------------------------------------------------------- This will also change the way your type must be called: From d515f3dbe22d98bd9d41a735dc2067c9d2b06b11 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 9 Dec 2012 14:49:09 -0800 Subject: [PATCH 12/37] fix asciidoc formatting Signed-off-by: Nico Schottelius --- docs/man/cdist-reference.text.sh | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/man/cdist-reference.text.sh b/docs/man/cdist-reference.text.sh index d8c23a79..006e5138 100755 --- a/docs/man/cdist-reference.text.sh +++ b/docs/man/cdist-reference.text.sh @@ -195,10 +195,8 @@ __object:: __object_id:: The type unique object id. Available for: type manifest, type explorer, type gencode - Note: The leading and the trailing "/" will always be stripped (caused by the filesystem database and ensured by the core). - Note: Double slashes ("//") will not be fixed and result in an error. __object_name:: The full qualified name of the current object. From 7abcd09fdd903c606fe2416931eaa74d1338e76e Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 9 Dec 2012 14:55:38 -0800 Subject: [PATCH 13/37] -- lines Signed-off-by: Nico Schottelius --- docs/man/cdist-reference.text.sh | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/man/cdist-reference.text.sh b/docs/man/cdist-reference.text.sh index 006e5138..37b9d41b 100755 --- a/docs/man/cdist-reference.text.sh +++ b/docs/man/cdist-reference.text.sh @@ -71,9 +71,7 @@ confdir:: Cdist will use all available configuration directories and create a temporary confdir containing links to the real configuration directories. This way it is possible to merge configuration directories. - By default it consists of everything in \$HOME/.cdist and cdist/conf/. - For more details see cdist(1) confdir/manifest/init:: From 18a90eddeeedf163432886f349f4e08d4ca3b56d Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 9 Dec 2012 14:56:25 -0800 Subject: [PATCH 14/37] and --more line Signed-off-by: Nico Schottelius --- docs/man/cdist-reference.text.sh | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/man/cdist-reference.text.sh b/docs/man/cdist-reference.text.sh index 37b9d41b..225d647f 100755 --- a/docs/man/cdist-reference.text.sh +++ b/docs/man/cdist-reference.text.sh @@ -96,7 +96,6 @@ confdir/type/:: confdir/type//:: Home of the type . - This directory is referenced by the variable __type (see below). confdir/type//man.text:: From 7d7ad2630c65a4bd0469807b688490fe57898174 Mon Sep 17 00:00:00 2001 From: Jake Guffey Date: Tue, 11 Dec 2012 11:15:21 -0500 Subject: [PATCH 15/37] Fix issue #147 --state absent should imply --stopped. --- cdist/conf/type/__jail/gencode-remote | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/cdist/conf/type/__jail/gencode-remote b/cdist/conf/type/__jail/gencode-remote index 1f326bf1..7491754c 100755 --- a/cdist/conf/type/__jail/gencode-remote +++ b/cdist/conf/type/__jail/gencode-remote @@ -35,7 +35,8 @@ fi state="$(cat "$__object/parameter/state")" started="true" -[ -f "$__object/parameter/stopped" ] && started="false" +# If the user wants the jail gone, it implies it shouldn't be started. +[ -f "$__object/parameter/stopped" -o "$state" = "absent" ] && started="false" if [ -f "$__object/parameter/ip" ]; then ip="$(cat "$__object/parameter/ip")" @@ -92,14 +93,6 @@ fi present="$(cat "$__object/explorer/present")" status="$(cat "$__object/explorer/status")" -# Defining a jail as absent and started at the same time -# makes no sense. Treat this as an error. -if [ "$started" = "true" -a "$state" = "absent" ]; then - exec >&2 - echo "Can't have --state absent and --started true together\!" - exit 1 -fi - stopJail() { # Check $status before issuing command if [ "$status" = "STARTED" ]; then From 2e083db90ad0884392dbdce04442ff95022cf21d Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Tue, 11 Dec 2012 09:26:53 -0800 Subject: [PATCH 16/37] ++changes Signed-off-by: Nico Schottelius --- docs/changelog | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/changelog b/docs/changelog index a255f42b..319f7a38 100644 --- a/docs/changelog +++ b/docs/changelog @@ -4,6 +4,9 @@ Changelog * Changes are always commented with their author in (braces) * Exception: No braces means author == Nico Schottelius +next: + * Type __jail: State absent should implies stopped (Jake Guffey) + 2.1.0: 2012-12-09 * Core: Ensure global explorers are executable * Core: Ensure type explorers are executable (Steven Armstrong) From d289cc2d3e63ecd692f2f1e786bd9c58168df82e Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Tue, 11 Dec 2012 23:13:46 -0800 Subject: [PATCH 17/37] +solver that accepts __git behaviour :) Signed-off-by: Nico Schottelius --- docs/dev/logs/2012-12-11.dependencies | 72 +++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 docs/dev/logs/2012-12-11.dependencies diff --git a/docs/dev/logs/2012-12-11.dependencies b/docs/dev/logs/2012-12-11.dependencies new file mode 100644 index 00000000..e5506564 --- /dev/null +++ b/docs/dev/logs/2012-12-11.dependencies @@ -0,0 +1,72 @@ +2.1.0 behaviour: + + +__git foo + __package git --state present + +__git bar + __package git --state present + + +require="__git/foo" git bar: + + __git bar + __git foo + __package git --state present + __package git --state present + __git foo + __package git --state present + + -> detects circular dependency + + +-------------------------------------------------------------------------------- + +__package abc + __package_apt abc + +__sometype def + __package abc + __package_apt abc + + +-------------------------------------------------------------------------------- + +Change proposal: + +Each object only depends on the objects it directly requires, tree build to +ensure correct running behaviour: + + +__git bar + __git foo + __package git --state present + +__git foo + __package git --state present + +Order: + +1) + __package/git (leaf node!) + +2) + __git/foo (new leaf node!) + +3) + __git/bar (new leaf node!) + + +For __package: + +__sometype def + __package abc + +__package abc + __package_apt abc + +1) __package_apt/abc (leaf node) + +2) __package/abc (new leaf node) + +3) __sometype/def (new leaf node) From 2e81379c18a431aba1148e0e0e667e55ded6b03c Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Thu, 13 Dec 2012 06:41:33 +0100 Subject: [PATCH 18/37] add very simple solver for dependencies Signed-off-by: Nico Schottelius --- cdist/config_install.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/cdist/config_install.py b/cdist/config_install.py index 2c1edc44..8c99f8d9 100644 --- a/cdist/config_install.py +++ b/cdist/config_install.py @@ -139,6 +139,18 @@ class ConfigInstall(object): self.local.object_path, self.local.type_path) + all_resolved = False + while not all_resolved: + all_resolved = True + for cdist_object in objects: + if not cdist_object.state == cdist_object.STATE_DONE: + all_resolved = False + if cdist_object.satisfied_requirements: + self.object_run(cdist_object) + + + return + dependency_resolver = resolver.DependencyResolver(objects) self.log.debug(pprint.pformat(dependency_resolver.dependencies)) From 6b6037d9bbb68dad10d780e6a7d699734828d339 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Thu, 13 Dec 2012 06:44:55 +0100 Subject: [PATCH 19/37] todos for a simple resolver Signed-off-by: Nico Schottelius --- cdist/config_install.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/cdist/config_install.py b/cdist/config_install.py index 8c99f8d9..cab09b91 100644 --- a/cdist/config_install.py +++ b/cdist/config_install.py @@ -139,6 +139,9 @@ class ConfigInstall(object): self.local.object_path, self.local.type_path) + # FIXME: + # - think about parallel execution (same for stage_prepare) + # - catch unresolvable trees all_resolved = False while not all_resolved: all_resolved = True From 16340b7d346b75ebde5feb9379c64553ae557d4e Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Fri, 14 Dec 2012 08:41:17 +0100 Subject: [PATCH 20/37] report on broken requirements, begin to satisfy requirements Signed-off-by: Nico Schottelius --- cdist/config_install.py | 18 ++++++++++++++++++ cdist/core/cdist_object.py | 4 ++++ 2 files changed, 22 insertions(+) diff --git a/cdist/config_install.py b/cdist/config_install.py index cab09b91..6a0d5161 100644 --- a/cdist/config_install.py +++ b/cdist/config_install.py @@ -143,6 +143,7 @@ class ConfigInstall(object): # - think about parallel execution (same for stage_prepare) # - catch unresolvable trees all_resolved = False + objects_changed = False while not all_resolved: all_resolved = True for cdist_object in objects: @@ -150,6 +151,23 @@ class ConfigInstall(object): all_resolved = False if cdist_object.satisfied_requirements: self.object_run(cdist_object) + objects_changed = True + + # Reran, but no object was solved + if not objects_changed: + # Create list of unfinished objects + their requirements for print + + evil_objects = [] + for cdist_object in objects: + if not cdist_object.state == cdist_object.STATE_DONE: +# evil_objects.append({ name: cdist_object.name, +# requirements: cdist_object.requirements, +# autorequire: cdist_object.autorequire }) + evil_objects.append("%s (required: %s, autorequired: %s" % + (cdist_object.name, cdist_object.requirements, cdist_object.autorequire)) + + raise cdist.Error("Cannot solve requirements for the following objects: %s" % + (",".join(evil_objects))) return diff --git a/cdist/core/cdist_object.py b/cdist/core/cdist_object.py index 90a21e59..cb34f521 100644 --- a/cdist/core/cdist_object.py +++ b/cdist/core/cdist_object.py @@ -209,3 +209,7 @@ class CdistObject(object): os.makedirs(absolute_parameter_path, exist_ok=False) except EnvironmentError as error: raise cdist.Error('Error creating directories for cdist object: %s: %s' % (self, error)) + + def satisfied_requirements(self): + + From 81c92e5cc43e551b6d22871aa2e6763b4e3904b2 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Fri, 14 Dec 2012 08:41:56 +0100 Subject: [PATCH 21/37] merge all resolver code into cdist_object (to gain the requirement searching code Signed-off-by: Nico Schottelius --- cdist/core/cdist_object.py | 169 +++++++++++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) diff --git a/cdist/core/cdist_object.py b/cdist/core/cdist_object.py index cb34f521..1844aabf 100644 --- a/cdist/core/cdist_object.py +++ b/cdist/core/cdist_object.py @@ -213,3 +213,172 @@ class CdistObject(object): def satisfied_requirements(self): + +# -*- coding: utf-8 -*- +# +# 2011 Steven Armstrong (steven-cdist at armstrong.cc) +# +# This file is part of cdist. +# +# cdist is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# cdist is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with cdist. If not, see . +# +# + +import logging +import os +import itertools +import fnmatch + +import cdist + +log = logging.getLogger(__name__) + + +class CircularReferenceError(cdist.Error): + def __init__(self, cdist_object, required_object): + self.cdist_object = cdist_object + self.required_object = required_object + + def __str__(self): + return 'Circular reference detected: %s -> %s' % (self.cdist_object.name, self.required_object.name) + + +class RequirementNotFoundError(cdist.Error): + def __init__(self, requirement): + self.requirement = requirement + + def __str__(self): + return 'Requirement could not be found: %s' % self.requirement + + +class DependencyResolver(object): + """Cdist's dependency resolver. + + Usage: + >> resolver = DependencyResolver(list_of_objects) + # Easy access to the objects we are working with + >> resolver.objects['__some_type/object_id'] + + # Easy access to a specific objects dependencies + >> resolver.dependencies['__some_type/object_id'] + [, ] + # Pretty print the dependency graph + >> from pprint import pprint + >> pprint(resolver.dependencies) + # Iterate over all existing objects in the correct order + >> for cdist_object in resolver: + >> do_something_with(cdist_object) + """ + def __init__(self, objects, logger=None): + self.objects = dict((o.name, o) for o in objects) + self._dependencies = None + self.log = logger or log + + @property + def dependencies(self): + """Build the dependency graph. + + Returns a dict where the keys are the object names and the values are + lists of all dependencies including the key object itself. + """ + if self._dependencies is None: + log.info("Resolving dependencies...") + self._dependencies = d = {} + self._preprocess_requirements() + for name,cdist_object in self.objects.items(): + resolved = [] + unresolved = [] + self._resolve_object_dependencies(cdist_object, resolved, unresolved) + d[name] = resolved + return self._dependencies + + def find_requirements_by_name(self, requirements): + """Takes a list of requirement patterns and returns a list of matching object instances. + + Patterns are expected to be Unix shell-style wildcards for use with fnmatch.filter. + + find_requirements_by_name(['__type/object_id', '__other_type/*']) -> + [, , ] + """ + object_names = self.objects.keys() + for pattern in requirements: + found = False + for requirement in fnmatch.filter(object_names, pattern): + found = True + yield self.objects[requirement] + if not found: + # FIXME: get rid of the singleton object_id, it should be invisible to the code -> hide it in Object + singleton = os.path.join(pattern, 'singleton') + if singleton in self.objects: + yield self.objects[singleton] + else: + raise RequirementNotFoundError(pattern) + + def _preprocess_requirements(self): + """Find all autorequire dependencies and merge them to be just requirements + for further processing. + """ + for cdist_object in self.objects.values(): + if cdist_object.autorequire: + # The objects (children) that this cdist_object (parent) defined + # in it's type manifest shall inherit all explicit requirements + # that the parent has so that user defined requirements are + # fullfilled and processed in the expected order. + for auto_requirement in self.find_requirements_by_name(cdist_object.autorequire): + for requirement in cdist_object.requirements: + if requirement not in auto_requirement.requirements: + auto_requirement.requirements.append(requirement) + # On the other hand the parent shall depend on all the children + # it created so that the user can setup dependencies on it as a + # whole without having to know anything about the parents + # internals. + cdist_object.requirements.extend(cdist_object.autorequire) + # As we changed the object on disc, we have to ensure it is not + # preprocessed again if someone would call us multiple times. + cdist_object.autorequire = [] + + def _resolve_object_dependencies(self, cdist_object, resolved, unresolved): + """Resolve all dependencies for the given cdist_object and store them + in the list which is passed as the 'resolved' arguments. + + e.g. + resolved = [] + unresolved = [] + resolve_object_dependencies(some_object, resolved, unresolved) + print("Dependencies for %s: %s" % (some_object, resolved)) + """ + self.log.debug('Resolving dependencies for: %s' % cdist_object.name) + try: + unresolved.append(cdist_object) + for required_object in self.find_requirements_by_name(cdist_object.requirements): + self.log.debug("Object %s requires %s", cdist_object, required_object) + if required_object not in resolved: + if required_object in unresolved: + raise CircularReferenceError(cdist_object, required_object) + self._resolve_object_dependencies(required_object, resolved, unresolved) + resolved.append(cdist_object) + unresolved.remove(cdist_object) + except RequirementNotFoundError as e: + raise cdist.CdistObjectError(cdist_object, "requires non-existing " + e.requirement) + + def __iter__(self): + """Iterate over all unique objects and yield them in the correct order. + """ + iterable = itertools.chain(*self.dependencies.values()) + # Keep record of objects that have already been seen + seen = set() + seen_add = seen.add + for cdist_object in itertools.filterfalse(seen.__contains__, iterable): + seen_add(cdist_object) + yield cdist_object From 6a8778e59568947324a04b7449d2a4cb84178ef2 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Fri, 14 Dec 2012 08:43:52 +0100 Subject: [PATCH 22/37] remove most of the resolver boilerplate, keep the interesting functions Signed-off-by: Nico Schottelius --- cdist/core/cdist_object.py | 121 ++++++------------------------------- 1 file changed, 17 insertions(+), 104 deletions(-) diff --git a/cdist/core/cdist_object.py b/cdist/core/cdist_object.py index 1844aabf..fa358937 100644 --- a/cdist/core/cdist_object.py +++ b/cdist/core/cdist_object.py @@ -213,78 +213,6 @@ class CdistObject(object): def satisfied_requirements(self): - -# -*- coding: utf-8 -*- -# -# 2011 Steven Armstrong (steven-cdist at armstrong.cc) -# -# This file is part of cdist. -# -# cdist is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# cdist is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with cdist. If not, see . -# -# - -import logging -import os -import itertools -import fnmatch - -import cdist - -log = logging.getLogger(__name__) - - -class CircularReferenceError(cdist.Error): - def __init__(self, cdist_object, required_object): - self.cdist_object = cdist_object - self.required_object = required_object - - def __str__(self): - return 'Circular reference detected: %s -> %s' % (self.cdist_object.name, self.required_object.name) - - -class RequirementNotFoundError(cdist.Error): - def __init__(self, requirement): - self.requirement = requirement - - def __str__(self): - return 'Requirement could not be found: %s' % self.requirement - - -class DependencyResolver(object): - """Cdist's dependency resolver. - - Usage: - >> resolver = DependencyResolver(list_of_objects) - # Easy access to the objects we are working with - >> resolver.objects['__some_type/object_id'] - - # Easy access to a specific objects dependencies - >> resolver.dependencies['__some_type/object_id'] - [, ] - # Pretty print the dependency graph - >> from pprint import pprint - >> pprint(resolver.dependencies) - # Iterate over all existing objects in the correct order - >> for cdist_object in resolver: - >> do_something_with(cdist_object) - """ - def __init__(self, objects, logger=None): - self.objects = dict((o.name, o) for o in objects) - self._dependencies = None - self.log = logger or log - @property def dependencies(self): """Build the dependency graph. @@ -348,37 +276,22 @@ class DependencyResolver(object): # preprocessed again if someone would call us multiple times. cdist_object.autorequire = [] - def _resolve_object_dependencies(self, cdist_object, resolved, unresolved): - """Resolve all dependencies for the given cdist_object and store them - in the list which is passed as the 'resolved' arguments. - e.g. - resolved = [] - unresolved = [] - resolve_object_dependencies(some_object, resolved, unresolved) - print("Dependencies for %s: %s" % (some_object, resolved)) - """ - self.log.debug('Resolving dependencies for: %s' % cdist_object.name) - try: - unresolved.append(cdist_object) - for required_object in self.find_requirements_by_name(cdist_object.requirements): - self.log.debug("Object %s requires %s", cdist_object, required_object) - if required_object not in resolved: - if required_object in unresolved: - raise CircularReferenceError(cdist_object, required_object) - self._resolve_object_dependencies(required_object, resolved, unresolved) - resolved.append(cdist_object) - unresolved.remove(cdist_object) - except RequirementNotFoundError as e: - raise cdist.CdistObjectError(cdist_object, "requires non-existing " + e.requirement) +class CircularReferenceError(cdist.Error): + def __init__(self, cdist_object, required_object): + self.cdist_object = cdist_object + self.required_object = required_object + + def __str__(self): + return 'Circular reference detected: %s -> %s' % (self.cdist_object.name, self.required_object.name) + + +class RequirementNotFoundError(cdist.Error): + def __init__(self, requirement): + self.requirement = requirement + + def __str__(self): + return 'Requirement could not be found: %s' % self.requirement + + - def __iter__(self): - """Iterate over all unique objects and yield them in the correct order. - """ - iterable = itertools.chain(*self.dependencies.values()) - # Keep record of objects that have already been seen - seen = set() - seen_add = seen.add - for cdist_object in itertools.filterfalse(seen.__contains__, iterable): - seen_add(cdist_object) - yield cdist_object From cd8b3cf68d3523bf4e732dc761481fcd110d077c Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 16 Dec 2012 12:47:13 +0100 Subject: [PATCH 23/37] finish rewrite of dependency resolver Signed-off-by: Nico Schottelius --- cdist/config_install.py | 3 -- cdist/core/cdist_object.py | 58 +++++++++++++------------------------- 2 files changed, 20 insertions(+), 41 deletions(-) diff --git a/cdist/config_install.py b/cdist/config_install.py index 6a0d5161..4d6c10f2 100644 --- a/cdist/config_install.py +++ b/cdist/config_install.py @@ -160,9 +160,6 @@ class ConfigInstall(object): evil_objects = [] for cdist_object in objects: if not cdist_object.state == cdist_object.STATE_DONE: -# evil_objects.append({ name: cdist_object.name, -# requirements: cdist_object.requirements, -# autorequire: cdist_object.autorequire }) evil_objects.append("%s (required: %s, autorequired: %s" % (cdist_object.name, cdist_object.requirements, cdist_object.autorequire)) diff --git a/cdist/core/cdist_object.py b/cdist/core/cdist_object.py index fa358937..501c0fb3 100644 --- a/cdist/core/cdist_object.py +++ b/cdist/core/cdist_object.py @@ -211,25 +211,17 @@ class CdistObject(object): raise cdist.Error('Error creating directories for cdist object: %s: %s' % (self, error)) def satisfied_requirements(self): + """Return state whether all of our dependencies have been resolved already""" + satisfied = True - @property - def dependencies(self): - """Build the dependency graph. + for requirement in self.all_requirements(): + if not requirement.state == self.STATE_DONE: + satisfied = False + break + + return satisfied - Returns a dict where the keys are the object names and the values are - lists of all dependencies including the key object itself. - """ - if self._dependencies is None: - log.info("Resolving dependencies...") - self._dependencies = d = {} - self._preprocess_requirements() - for name,cdist_object in self.objects.items(): - resolved = [] - unresolved = [] - self._resolve_object_dependencies(cdist_object, resolved, unresolved) - d[name] = resolved - return self._dependencies def find_requirements_by_name(self, requirements): """Takes a list of requirement patterns and returns a list of matching object instances. @@ -239,7 +231,8 @@ class CdistObject(object): find_requirements_by_name(['__type/object_id', '__other_type/*']) -> [, , ] """ - object_names = self.objects.keys() + + object_names = self.list_object_namess(self.base_path) for pattern in requirements: found = False for requirement in fnmatch.filter(object_names, pattern): @@ -253,28 +246,17 @@ class CdistObject(object): else: raise RequirementNotFoundError(pattern) - def _preprocess_requirements(self): - """Find all autorequire dependencies and merge them to be just requirements - for further processing. + def all_requirements(self): """ - for cdist_object in self.objects.values(): - if cdist_object.autorequire: - # The objects (children) that this cdist_object (parent) defined - # in it's type manifest shall inherit all explicit requirements - # that the parent has so that user defined requirements are - # fullfilled and processed in the expected order. - for auto_requirement in self.find_requirements_by_name(cdist_object.autorequire): - for requirement in cdist_object.requirements: - if requirement not in auto_requirement.requirements: - auto_requirement.requirements.append(requirement) - # On the other hand the parent shall depend on all the children - # it created so that the user can setup dependencies on it as a - # whole without having to know anything about the parents - # internals. - cdist_object.requirements.extend(cdist_object.autorequire) - # As we changed the object on disc, we have to ensure it is not - # preprocessed again if someone would call us multiple times. - cdist_object.autorequire = [] + Return resolved autorequirements and requirements so that + a complete list of requirements is returned + """ + + all_requirements = [] + all_requirements.extend(self.find_requirements_by_name(self.requirements)) + all_requirements.extend(self.find_requirements_by_name(self.autorequire)) + + return unique(all_requirements) class CircularReferenceError(cdist.Error): From 442dc767dd245631c0a09da9da775c232b378c1e Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 16 Dec 2012 12:55:36 +0100 Subject: [PATCH 24/37] merge resolver test into object test - because object requirement solving is implemented Signed-off-by: Nico Schottelius --- cdist/test/object/__init__.py | 91 +++++++++++++++++++++++++++++++++ cdist/test/resolver/__init__.py | 88 ------------------------------- 2 files changed, 91 insertions(+), 88 deletions(-) delete mode 100644 cdist/test/resolver/__init__.py diff --git a/cdist/test/object/__init__.py b/cdist/test/object/__init__.py index 3a91f709..2612f060 100644 --- a/cdist/test/object/__init__.py +++ b/cdist/test/object/__init__.py @@ -200,3 +200,94 @@ class ObjectTestCase(test.CdistTestCase): self.assertTrue(isinstance(other_object, core.CdistObject)) self.assertEqual(other_object.cdist_type.name, '__first') self.assertEqual(other_object.object_id, 'man') + + +# -*- coding: utf-8 -*- +# +# 2010-2011 Steven Armstrong (steven-cdist at armstrong.cc) +# +# This file is part of cdist. +# +# cdist is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# cdist is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with cdist. If not, see . +# +# + +import os +import shutil + +import cdist +from cdist import test +from cdist import core + +import os.path as op +my_dir = op.abspath(op.dirname(__file__)) +fixtures = op.join(my_dir, 'fixtures') +object_base_path = op.join(fixtures, 'object') +type_base_path = op.join(fixtures, 'type') + + +class ResolverTestCase(test.CdistTestCase): + + def setUp(self): + self.objects = list(core.CdistObject.list_objects(object_base_path, type_base_path)) + self.object_index = dict((o.name, o) for o in self.objects) + + # Create a test object with requirements + self.test_object = core.CdistObject(cdist_type, base_path, object_id=None) + + def tearDown(self): + for o in self.objects: + o.requirements = [] + + def test_find_requirements_by_name_string(self): + requirements = ['__first/man', '__second/on-the', '__third/moon'] + required_objects = [self.object_index[name] for name in requirements] + self.assertEqual(sorted(list(self.dependency_resolver.find_requirements_by_name(requirements))), + sorted(required_objects)) + + def test_find_requirements_by_name_pattern(self): + requirements = ['__first/*', '__second/*-the', '__third/moon'] + requirements_expanded = [ + '__first/child', '__first/dog', '__first/man', '__first/woman', + '__second/on-the', '__second/under-the', + '__third/moon' + ] + required_objects = [self.object_index[name] for name in requirements_expanded] + self.assertEqual(sorted(list(self.dependency_resolver.find_requirements_by_name(requirements))), + sorted(required_objects)) + + def test_dependency_resolution(self): + first_man = self.object_index['__first/man'] + second_on_the = self.object_index['__second/on-the'] + third_moon = self.object_index['__third/moon'] + first_man.requirements = [second_on_the.name] + second_on_the.requirements = [third_moon.name] + self.assertEqual( + self.dependency_resolver.dependencies['__first/man'], + [third_moon, second_on_the, first_man] + ) + + def test_circular_reference(self): + first_man = self.object_index['__first/man'] + first_woman = self.object_index['__first/woman'] + first_man.requirements = [first_woman.name] + first_woman.requirements = [first_man.name] + with self.assertRaises(resolver.CircularReferenceError): + self.dependency_resolver.dependencies + + def test_requirement_not_found(self): + first_man = self.object_index['__first/man'] + first_man.requirements = ['__does/not/exist'] + with self.assertRaises(cdist.Error): + self.dependency_resolver.dependencies diff --git a/cdist/test/resolver/__init__.py b/cdist/test/resolver/__init__.py deleted file mode 100644 index baae26de..00000000 --- a/cdist/test/resolver/__init__.py +++ /dev/null @@ -1,88 +0,0 @@ -# -*- coding: utf-8 -*- -# -# 2010-2011 Steven Armstrong (steven-cdist at armstrong.cc) -# -# This file is part of cdist. -# -# cdist is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# cdist is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with cdist. If not, see . -# -# - -import os -import shutil - -import cdist -from cdist import test -from cdist import core -from cdist import resolver - -import os.path as op -my_dir = op.abspath(op.dirname(__file__)) -fixtures = op.join(my_dir, 'fixtures') -object_base_path = op.join(fixtures, 'object') -type_base_path = op.join(fixtures, 'type') - - -class ResolverTestCase(test.CdistTestCase): - - def setUp(self): - self.objects = list(core.CdistObject.list_objects(object_base_path, type_base_path)) - self.object_index = dict((o.name, o) for o in self.objects) - self.dependency_resolver = resolver.DependencyResolver(self.objects) - - def tearDown(self): - for o in self.objects: - o.requirements = [] - - def test_find_requirements_by_name_string(self): - requirements = ['__first/man', '__second/on-the', '__third/moon'] - required_objects = [self.object_index[name] for name in requirements] - self.assertEqual(sorted(list(self.dependency_resolver.find_requirements_by_name(requirements))), - sorted(required_objects)) - - def test_find_requirements_by_name_pattern(self): - requirements = ['__first/*', '__second/*-the', '__third/moon'] - requirements_expanded = [ - '__first/child', '__first/dog', '__first/man', '__first/woman', - '__second/on-the', '__second/under-the', - '__third/moon' - ] - required_objects = [self.object_index[name] for name in requirements_expanded] - self.assertEqual(sorted(list(self.dependency_resolver.find_requirements_by_name(requirements))), - sorted(required_objects)) - - def test_dependency_resolution(self): - first_man = self.object_index['__first/man'] - second_on_the = self.object_index['__second/on-the'] - third_moon = self.object_index['__third/moon'] - first_man.requirements = [second_on_the.name] - second_on_the.requirements = [third_moon.name] - self.assertEqual( - self.dependency_resolver.dependencies['__first/man'], - [third_moon, second_on_the, first_man] - ) - - def test_circular_reference(self): - first_man = self.object_index['__first/man'] - first_woman = self.object_index['__first/woman'] - first_man.requirements = [first_woman.name] - first_woman.requirements = [first_man.name] - with self.assertRaises(resolver.CircularReferenceError): - self.dependency_resolver.dependencies - - def test_requirement_not_found(self): - first_man = self.object_index['__first/man'] - first_man.requirements = ['__does/not/exist'] - with self.assertRaises(cdist.Error): - self.dependency_resolver.dependencies From 958d2d336f4fb17e3033e06473ca67ec847e3728 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Tue, 18 Dec 2012 17:16:26 +0100 Subject: [PATCH 25/37] various cleanups Signed-off-by: Nico Schottelius --- cdist/core/cdist_object.py | 35 ++++++++++------------ cdist/core/cdist_type.py | 40 ++++++++++++------------- cdist/test/object/__init__.py | 56 ++++++----------------------------- 3 files changed, 45 insertions(+), 86 deletions(-) diff --git a/cdist/core/cdist_object.py b/cdist/core/cdist_object.py index 501c0fb3..7d757e67 100644 --- a/cdist/core/cdist_object.py +++ b/cdist/core/cdist_object.py @@ -56,6 +56,21 @@ class CdistObject(object): STATE_RUNNING = "running" STATE_DONE = "done" + def __init__(self, cdist_type, base_path, object_id=None): + self.cdist_type = cdist_type # instance of Type + self.base_path = base_path + self.object_id = object_id + + self.validate_object_id() + self.sanitise_object_id() + + self.name = self.join_name(self.cdist_type.name, self.object_id) + self.path = os.path.join(self.cdist_type.path, self.object_id, OBJECT_MARKER) + self.absolute_path = os.path.join(self.base_path, self.path) + self.code_local_path = os.path.join(self.path, "code-local") + self.code_remote_path = os.path.join(self.path, "code-remote") + self.parameter_path = os.path.join(self.path, "parameter") + @classmethod def list_objects(cls, object_base_path, type_base_path): """Return a list of object instances""" @@ -112,21 +127,6 @@ class CdistObject(object): raise IllegalObjectIdError(self.object_id, "Missing object_id and type is not a singleton.") - def __init__(self, cdist_type, base_path, object_id=None): - self.cdist_type = cdist_type # instance of Type - self.base_path = base_path - self.object_id = object_id - - self.validate_object_id() - self.sanitise_object_id() - - self.name = self.join_name(self.cdist_type.name, self.object_id) - self.path = os.path.join(self.cdist_type.path, self.object_id, OBJECT_MARKER) - self.absolute_path = os.path.join(self.base_path, self.path) - self.code_local_path = os.path.join(self.path, "code-local") - self.code_remote_path = os.path.join(self.path, "code-remote") - self.parameter_path = os.path.join(self.path, "parameter") - def object_from_name(self, object_name): """Convenience method for creating an object instance from an object name. @@ -232,7 +232,7 @@ class CdistObject(object): [, , ] """ - object_names = self.list_object_namess(self.base_path) + object_names = self.list_object_names(self.base_path) for pattern in requirements: found = False for requirement in fnmatch.filter(object_names, pattern): @@ -274,6 +274,3 @@ class RequirementNotFoundError(cdist.Error): def __str__(self): return 'Requirement could not be found: %s' % self.requirement - - - diff --git a/cdist/core/cdist_type.py b/cdist/core/cdist_type.py index 44e192fc..0efb10f4 100644 --- a/cdist/core/cdist_type.py +++ b/cdist/core/cdist_type.py @@ -42,6 +42,26 @@ class CdistType(object): """ + def __init__(self, base_path, name): + self.base_path = base_path + self.name = name + self.path = self.name + self.absolute_path = os.path.join(self.base_path, self.path) + if not os.path.isdir(self.absolute_path): + raise NoSuchTypeError(self.path, self.absolute_path) + self.manifest_path = os.path.join(self.name, "manifest") + self.explorer_path = os.path.join(self.name, "explorer") + self.gencode_local_path = os.path.join(self.name, "gencode-local") + self.gencode_remote_path = os.path.join(self.name, "gencode-remote") + self.manifest_path = os.path.join(self.name, "manifest") + + self.__explorers = None + self.__required_parameters = None + self.__required_multiple_parameters = None + self.__optional_parameters = None + self.__optional_multiple_parameters = None + self.__boolean_parameters = None + @classmethod def list_types(cls, base_path): """Return a list of type instances""" @@ -65,26 +85,6 @@ class CdistType(object): # return instance so __init__ is called return cls._instances[name] - def __init__(self, base_path, name): - self.base_path = base_path - self.name = name - self.path = self.name - self.absolute_path = os.path.join(self.base_path, self.path) - if not os.path.isdir(self.absolute_path): - raise NoSuchTypeError(self.path, self.absolute_path) - self.manifest_path = os.path.join(self.name, "manifest") - self.explorer_path = os.path.join(self.name, "explorer") - self.gencode_local_path = os.path.join(self.name, "gencode-local") - self.gencode_remote_path = os.path.join(self.name, "gencode-remote") - self.manifest_path = os.path.join(self.name, "manifest") - - self.__explorers = None - self.__required_parameters = None - self.__required_multiple_parameters = None - self.__optional_parameters = None - self.__optional_multiple_parameters = None - self.__boolean_parameters = None - def __repr__(self): return '' % self.name diff --git a/cdist/test/object/__init__.py b/cdist/test/object/__init__.py index 2612f060..531300a7 100644 --- a/cdist/test/object/__init__.py +++ b/cdist/test/object/__init__.py @@ -1,6 +1,7 @@ # -*- coding: utf-8 -*- # # 2010-2011 Steven Armstrong (steven-cdist at armstrong.cc) +# 2012 Nico Schottelius (nico-cdist at schottelius.org) # # This file is part of cdist. # @@ -202,56 +203,24 @@ class ObjectTestCase(test.CdistTestCase): self.assertEqual(other_object.object_id, 'man') -# -*- coding: utf-8 -*- -# -# 2010-2011 Steven Armstrong (steven-cdist at armstrong.cc) -# -# This file is part of cdist. -# -# cdist is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# cdist is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with cdist. If not, see . -# -# -import os -import shutil - -import cdist -from cdist import test -from cdist import core - -import os.path as op -my_dir = op.abspath(op.dirname(__file__)) -fixtures = op.join(my_dir, 'fixtures') -object_base_path = op.join(fixtures, 'object') -type_base_path = op.join(fixtures, 'type') - - -class ResolverTestCase(test.CdistTestCase): +class ObjectResolveRequirementsTestCase(test.CdistTestCase): def setUp(self): self.objects = list(core.CdistObject.list_objects(object_base_path, type_base_path)) self.object_index = dict((o.name, o) for o in self.objects) - # Create a test object with requirements - self.test_object = core.CdistObject(cdist_type, base_path, object_id=None) + self.cdist_type = core.CdistType(type_base_path, '__third') + self.cdist_object = core.CdistObject(self.cdist_type, object_base_path, 'moon') def tearDown(self): for o in self.objects: o.requirements = [] def test_find_requirements_by_name_string(self): + """Check that resolving requirements by name works""" requirements = ['__first/man', '__second/on-the', '__third/moon'] + required_objects = [self.object_index[name] for name in requirements] self.assertEqual(sorted(list(self.dependency_resolver.find_requirements_by_name(requirements))), sorted(required_objects)) @@ -278,16 +247,9 @@ class ResolverTestCase(test.CdistTestCase): [third_moon, second_on_the, first_man] ) - def test_circular_reference(self): - first_man = self.object_index['__first/man'] - first_woman = self.object_index['__first/woman'] - first_man.requirements = [first_woman.name] - first_woman.requirements = [first_man.name] - with self.assertRaises(resolver.CircularReferenceError): - self.dependency_resolver.dependencies - def test_requirement_not_found(self): first_man = self.object_index['__first/man'] first_man.requirements = ['__does/not/exist'] - with self.assertRaises(cdist.Error): - self.dependency_resolver.dependencies + with self.assertRaises(core.cdist_object.RequirementNotFoundError): + first_man.find_requirements_by_name(first_man.requirements) + From 4faec434936ce21e6b7a4662fa3b3efa49d83a4f Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Tue, 18 Dec 2012 17:37:07 +0100 Subject: [PATCH 26/37] cleanup: remove call to old resolver Signed-off-by: Nico Schottelius --- cdist/config_install.py | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/cdist/config_install.py b/cdist/config_install.py index 4d6c10f2..d6628cc2 100644 --- a/cdist/config_install.py +++ b/cdist/config_install.py @@ -145,6 +145,10 @@ class ConfigInstall(object): all_resolved = False objects_changed = False while not all_resolved: + + object_state_list=' '.join('%s:%s:%s' % (o, o.state, o.all_requirements()) for o in objects) + self.log.debug("Object state (name:state:requirements): %s" % object_state_list) + all_resolved = True for cdist_object in objects: if not cdist_object.state == cdist_object.STATE_DONE: @@ -165,13 +169,3 @@ class ConfigInstall(object): raise cdist.Error("Cannot solve requirements for the following objects: %s" % (",".join(evil_objects))) - - - return - - dependency_resolver = resolver.DependencyResolver(objects) - self.log.debug(pprint.pformat(dependency_resolver.dependencies)) - - for cdist_object in dependency_resolver: - self.log.debug("Run object: %s", cdist_object) - self.object_run(cdist_object) From 8031c7770041cff1be74655cec0745af5cdba480 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Tue, 18 Dec 2012 18:21:51 +0100 Subject: [PATCH 27/37] fix some tests, break some others :-) Signed-off-by: Nico Schottelius --- cdist/core/cdist_object.py | 19 +++++-- cdist/test/object/__init__.py | 57 ++++++++++++------- .../object/__first/child/.cdist/.keep | 0 .../fixtures/object/__first/dog/.cdist/.keep | 0 .../object/__first/woman/.cdist/.keep | 0 .../object/__second/under-the/.cdist/.keep | 0 6 files changed, 52 insertions(+), 24 deletions(-) create mode 100644 cdist/test/object/fixtures/object/__first/child/.cdist/.keep create mode 100644 cdist/test/object/fixtures/object/__first/dog/.cdist/.keep create mode 100644 cdist/test/object/fixtures/object/__first/woman/.cdist/.keep create mode 100644 cdist/test/object/fixtures/object/__second/under-the/.cdist/.keep diff --git a/cdist/core/cdist_object.py b/cdist/core/cdist_object.py index 7d757e67..3e5e33b7 100644 --- a/cdist/core/cdist_object.py +++ b/cdist/core/cdist_object.py @@ -20,6 +20,7 @@ # # +import fnmatch import logging import os import collections @@ -232,11 +233,19 @@ class CdistObject(object): [, , ] """ - object_names = self.list_object_names(self.base_path) + + # FIXME: think about where to store this - probably not here + self.objects = dict((o.name, o) for o in self.list_objects(self.base_path, self.cdist_type.base_path)) + object_names = self.objects.keys() + + print("a:%s" % self.objects) + print("b:%s" % object_names) + for pattern in requirements: found = False for requirement in fnmatch.filter(object_names, pattern): found = True + print("c:%s" % self.objects[requirement]) yield self.objects[requirement] if not found: # FIXME: get rid of the singleton object_id, it should be invisible to the code -> hide it in Object @@ -252,11 +261,11 @@ class CdistObject(object): a complete list of requirements is returned """ - all_requirements = [] - all_requirements.extend(self.find_requirements_by_name(self.requirements)) - all_requirements.extend(self.find_requirements_by_name(self.autorequire)) + all_reqs= [] + all_reqs.extend(self.find_requirements_by_name(self.requirements)) + all_reqs.extend(self.find_requirements_by_name(self.autorequire)) - return unique(all_requirements) + return set(all_reqs) class CircularReferenceError(cdist.Error): diff --git a/cdist/test/object/__init__.py b/cdist/test/object/__init__.py index 531300a7..ec6e25c1 100644 --- a/cdist/test/object/__init__.py +++ b/cdist/test/object/__init__.py @@ -37,8 +37,16 @@ type_base_path = op.join(fixtures, 'type') class ObjectClassTestCase(test.CdistTestCase): def test_list_object_names(self): - object_names = list(core.CdistObject.list_object_names(object_base_path)) - self.assertEqual(object_names, ['__first/man', '__second/on-the', '__third/moon']) + found_object_names = sorted(list(core.CdistObject.list_object_names(object_base_path))) + expected_object_names = sorted([ + '__first/child', + '__first/dog', + '__first/man', + '__first/woman', + '__second/on-the', + '__second/under-the', + '__third/moon']) + self.assertEqual(found_object_names, expected_object_names) def test_list_type_names(self): type_names = list(cdist.core.CdistObject.list_type_names(object_base_path)) @@ -209,6 +217,9 @@ class ObjectResolveRequirementsTestCase(test.CdistTestCase): def setUp(self): self.objects = list(core.CdistObject.list_objects(object_base_path, type_base_path)) self.object_index = dict((o.name, o) for o in self.objects) + self.object_names = [o.name for o in self.objects] + + print(self.objects) self.cdist_type = core.CdistType(type_base_path, '__third') self.cdist_object = core.CdistObject(self.cdist_type, object_base_path, 'moon') @@ -218,23 +229,28 @@ class ObjectResolveRequirementsTestCase(test.CdistTestCase): o.requirements = [] def test_find_requirements_by_name_string(self): - """Check that resolving requirements by name works""" - requirements = ['__first/man', '__second/on-the', '__third/moon'] + """Check that resolving requirements by name works (require all objects)""" + requirements = self.object_names - required_objects = [self.object_index[name] for name in requirements] - self.assertEqual(sorted(list(self.dependency_resolver.find_requirements_by_name(requirements))), - sorted(required_objects)) + self.cdist_object.requirements = requirements + + found_requirements = sorted(self.cdist_object.find_requirements_by_name(self.cdist_object.requirements)) + expected_requirements = sorted(self.objects) + + self.assertEqual(found_requirements, expected_requirements) def test_find_requirements_by_name_pattern(self): + """Test whether pattern matching on requirements works""" + + # Matches all objects in the end requirements = ['__first/*', '__second/*-the', '__third/moon'] - requirements_expanded = [ - '__first/child', '__first/dog', '__first/man', '__first/woman', - '__second/on-the', '__second/under-the', - '__third/moon' - ] - required_objects = [self.object_index[name] for name in requirements_expanded] - self.assertEqual(sorted(list(self.dependency_resolver.find_requirements_by_name(requirements))), - sorted(required_objects)) + + self.cdist_object.requirements = requirements + + expected_requirements = sorted(self.objects) + found_requirements = sorted(self.cdist_object.find_requirements_by_name(self.cdist_object.requirements)) + + self.assertEqual(expected_requirements, found_requirements) def test_dependency_resolution(self): first_man = self.object_index['__first/man'] @@ -242,10 +258,13 @@ class ObjectResolveRequirementsTestCase(test.CdistTestCase): third_moon = self.object_index['__third/moon'] first_man.requirements = [second_on_the.name] second_on_the.requirements = [third_moon.name] - self.assertEqual( - self.dependency_resolver.dependencies['__first/man'], - [third_moon, second_on_the, first_man] - ) + + # FIXME :-) + self.assertTrue(False) +# self.assertEqual( +# self.dependency_resolver.dependencies['__first/man'], +# [third_moon, second_on_the, first_man] +# ) def test_requirement_not_found(self): first_man = self.object_index['__first/man'] diff --git a/cdist/test/object/fixtures/object/__first/child/.cdist/.keep b/cdist/test/object/fixtures/object/__first/child/.cdist/.keep new file mode 100644 index 00000000..e69de29b diff --git a/cdist/test/object/fixtures/object/__first/dog/.cdist/.keep b/cdist/test/object/fixtures/object/__first/dog/.cdist/.keep new file mode 100644 index 00000000..e69de29b diff --git a/cdist/test/object/fixtures/object/__first/woman/.cdist/.keep b/cdist/test/object/fixtures/object/__first/woman/.cdist/.keep new file mode 100644 index 00000000..e69de29b diff --git a/cdist/test/object/fixtures/object/__second/under-the/.cdist/.keep b/cdist/test/object/fixtures/object/__second/under-the/.cdist/.keep new file mode 100644 index 00000000..e69de29b From 369305f37643c1d26cf7b0551e83a99f033796ad Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Wed, 19 Dec 2012 15:33:41 +0100 Subject: [PATCH 28/37] remove circularreferenceerror, because it is not being used anymore Signed-off-by: Nico Schottelius --- cdist/core/cdist_object.py | 15 +-------------- 1 file changed, 1 insertion(+), 14 deletions(-) diff --git a/cdist/core/cdist_object.py b/cdist/core/cdist_object.py index 3e5e33b7..cc9aeaa5 100644 --- a/cdist/core/cdist_object.py +++ b/cdist/core/cdist_object.py @@ -234,18 +234,14 @@ class CdistObject(object): """ - # FIXME: think about where to store this - probably not here + # FIXME: think about where/when to store this - probably not here self.objects = dict((o.name, o) for o in self.list_objects(self.base_path, self.cdist_type.base_path)) object_names = self.objects.keys() - print("a:%s" % self.objects) - print("b:%s" % object_names) - for pattern in requirements: found = False for requirement in fnmatch.filter(object_names, pattern): found = True - print("c:%s" % self.objects[requirement]) yield self.objects[requirement] if not found: # FIXME: get rid of the singleton object_id, it should be invisible to the code -> hide it in Object @@ -268,15 +264,6 @@ class CdistObject(object): return set(all_reqs) -class CircularReferenceError(cdist.Error): - def __init__(self, cdist_object, required_object): - self.cdist_object = cdist_object - self.required_object = required_object - - def __str__(self): - return 'Circular reference detected: %s -> %s' % (self.cdist_object.name, self.required_object.name) - - class RequirementNotFoundError(cdist.Error): def __init__(self, requirement): self.requirement = requirement From eb93d1bebd0a410970d6c161c0cd565e9ef2f29d Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Wed, 19 Dec 2012 15:33:50 +0100 Subject: [PATCH 29/37] begin tests for config_install Signed-off-by: Nico Schottelius --- cdist/test/config_install/__init__.py | 93 +++++++++++++++++++ .../fixtures/object/__first/.keep | 0 .../object/__first/child/.cdist/.keep | 0 .../fixtures/object/__first/dog/.cdist/.keep | 0 .../fixtures/object/__first/man/.cdist/.keep | 0 .../object/__first/woman/.cdist/.keep | 0 .../fixtures/object/__second/.keep | 0 .../object/__second/on-the/.cdist/.keep | 0 .../object/__second/under-the/.cdist/.keep | 0 .../fixtures/object/__third/.keep | 0 .../fixtures/object/__third/moon/.cdist/.keep | 0 .../object/__third/moon/.cdist/parameter/name | 1 + .../__third/moon/.cdist/parameter/planet | 1 + .../fixtures/type/__first/.keep | 0 .../fixtures/type/__second/.keep | 0 .../fixtures/type/__third/.keep | 0 16 files changed, 95 insertions(+) create mode 100644 cdist/test/config_install/__init__.py create mode 100644 cdist/test/config_install/fixtures/object/__first/.keep create mode 100644 cdist/test/config_install/fixtures/object/__first/child/.cdist/.keep create mode 100644 cdist/test/config_install/fixtures/object/__first/dog/.cdist/.keep create mode 100644 cdist/test/config_install/fixtures/object/__first/man/.cdist/.keep create mode 100644 cdist/test/config_install/fixtures/object/__first/woman/.cdist/.keep create mode 100644 cdist/test/config_install/fixtures/object/__second/.keep create mode 100644 cdist/test/config_install/fixtures/object/__second/on-the/.cdist/.keep create mode 100644 cdist/test/config_install/fixtures/object/__second/under-the/.cdist/.keep create mode 100644 cdist/test/config_install/fixtures/object/__third/.keep create mode 100644 cdist/test/config_install/fixtures/object/__third/moon/.cdist/.keep create mode 100644 cdist/test/config_install/fixtures/object/__third/moon/.cdist/parameter/name create mode 100644 cdist/test/config_install/fixtures/object/__third/moon/.cdist/parameter/planet create mode 100644 cdist/test/config_install/fixtures/type/__first/.keep create mode 100644 cdist/test/config_install/fixtures/type/__second/.keep create mode 100644 cdist/test/config_install/fixtures/type/__third/.keep diff --git a/cdist/test/config_install/__init__.py b/cdist/test/config_install/__init__.py new file mode 100644 index 00000000..326632b7 --- /dev/null +++ b/cdist/test/config_install/__init__.py @@ -0,0 +1,93 @@ +# -*- coding: utf-8 -*- +# +# 2010-2011 Steven Armstrong (steven-cdist at armstrong.cc) +# 2012 Nico Schottelius (nico-cdist at schottelius.org) +# +# This file is part of cdist. +# +# cdist is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# cdist is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with cdist. If not, see . +# +# + +import os +import shutil + +from cdist import test +from cdist import core + +import cdist +import cdist.context + +import os.path as op +my_dir = op.abspath(op.dirname(__file__)) +fixtures = op.join(my_dir, 'fixtures') +object_base_path = op.join(fixtures, 'object') +type_base_path = op.join(fixtures, 'type') +add_conf_dir = op.join(fixtures, 'conf') + +class ConfigInstallRunTestCase(test.CdistTestCase): + + def setUp(self): + + self.context = cdist.context.Context( + target_host=self.target_host, + remote_copy=self.remote_copy, + remote_exec=self.remote_exec, + initial_manifest=args.manifest, + add_conf_dirs=add_conf_dir, + exec_path=test.cdist_exec_path, + debug=False) + + self.config = config.Config(self.context) + + def setUp(self): + self.objects = list(core.CdistObject.list_objects(object_base_path, type_base_path)) + self.object_index = dict((o.name, o) for o in self.objects) + self.object_names = [o.name for o in self.objects] + + print(self.objects) + + self.cdist_type = core.CdistType(type_base_path, '__third') + self.cdist_object = core.CdistObject(self.cdist_type, object_base_path, 'moon') + + def tearDown(self): + for o in self.objects: + o.requirements = [] + + def test_dependency_resolution(self): + first = self.object_index['__first/man'] + second = self.object_index['__second/on-the'] + third = self.object_index['__third/moon'] + + first.requirements = [second.name] + second.requirements = [third.name] + + self.config.stage_run_prepare() + + # First run: + # solves first and maybe second (depending on the order in the set) + self.config.stage_run_iterate() + + # FIXME :-) + self.assertTrue(False) +# self.assertEqual( +# self.dependency_resolver.dependencies['__first/man'], +# [third_moon, second_on_the, first_man] +# ) + + def test_requirement_not_found(self): + first_man = self.object_index['__first/man'] + first_man.requirements = ['__does/not/exist'] + with self.assertRaises(core.cdist_object.RequirementNotFoundError): + first_man.find_requirements_by_name(first_man.requirements) diff --git a/cdist/test/config_install/fixtures/object/__first/.keep b/cdist/test/config_install/fixtures/object/__first/.keep new file mode 100644 index 00000000..e69de29b diff --git a/cdist/test/config_install/fixtures/object/__first/child/.cdist/.keep b/cdist/test/config_install/fixtures/object/__first/child/.cdist/.keep new file mode 100644 index 00000000..e69de29b diff --git a/cdist/test/config_install/fixtures/object/__first/dog/.cdist/.keep b/cdist/test/config_install/fixtures/object/__first/dog/.cdist/.keep new file mode 100644 index 00000000..e69de29b diff --git a/cdist/test/config_install/fixtures/object/__first/man/.cdist/.keep b/cdist/test/config_install/fixtures/object/__first/man/.cdist/.keep new file mode 100644 index 00000000..e69de29b diff --git a/cdist/test/config_install/fixtures/object/__first/woman/.cdist/.keep b/cdist/test/config_install/fixtures/object/__first/woman/.cdist/.keep new file mode 100644 index 00000000..e69de29b diff --git a/cdist/test/config_install/fixtures/object/__second/.keep b/cdist/test/config_install/fixtures/object/__second/.keep new file mode 100644 index 00000000..e69de29b diff --git a/cdist/test/config_install/fixtures/object/__second/on-the/.cdist/.keep b/cdist/test/config_install/fixtures/object/__second/on-the/.cdist/.keep new file mode 100644 index 00000000..e69de29b diff --git a/cdist/test/config_install/fixtures/object/__second/under-the/.cdist/.keep b/cdist/test/config_install/fixtures/object/__second/under-the/.cdist/.keep new file mode 100644 index 00000000..e69de29b diff --git a/cdist/test/config_install/fixtures/object/__third/.keep b/cdist/test/config_install/fixtures/object/__third/.keep new file mode 100644 index 00000000..e69de29b diff --git a/cdist/test/config_install/fixtures/object/__third/moon/.cdist/.keep b/cdist/test/config_install/fixtures/object/__third/moon/.cdist/.keep new file mode 100644 index 00000000..e69de29b diff --git a/cdist/test/config_install/fixtures/object/__third/moon/.cdist/parameter/name b/cdist/test/config_install/fixtures/object/__third/moon/.cdist/parameter/name new file mode 100644 index 00000000..4129a761 --- /dev/null +++ b/cdist/test/config_install/fixtures/object/__third/moon/.cdist/parameter/name @@ -0,0 +1 @@ +Prometheus diff --git a/cdist/test/config_install/fixtures/object/__third/moon/.cdist/parameter/planet b/cdist/test/config_install/fixtures/object/__third/moon/.cdist/parameter/planet new file mode 100644 index 00000000..8e6ee422 --- /dev/null +++ b/cdist/test/config_install/fixtures/object/__third/moon/.cdist/parameter/planet @@ -0,0 +1 @@ +Saturn diff --git a/cdist/test/config_install/fixtures/type/__first/.keep b/cdist/test/config_install/fixtures/type/__first/.keep new file mode 100644 index 00000000..e69de29b diff --git a/cdist/test/config_install/fixtures/type/__second/.keep b/cdist/test/config_install/fixtures/type/__second/.keep new file mode 100644 index 00000000..e69de29b diff --git a/cdist/test/config_install/fixtures/type/__third/.keep b/cdist/test/config_install/fixtures/type/__third/.keep new file mode 100644 index 00000000..e69de29b From c2705380727b0d341a680e6b510246622cb5b33a Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Wed, 19 Dec 2012 19:36:00 +0100 Subject: [PATCH 30/37] add dry_run option to object_run Signed-off-by: Nico Schottelius --- cdist/config_install.py | 90 +++++++++++++++++++++++++---------------- 1 file changed, 56 insertions(+), 34 deletions(-) diff --git a/cdist/config_install.py b/cdist/config_install.py index d6628cc2..b7662ad8 100644 --- a/cdist/config_install.py +++ b/cdist/config_install.py @@ -104,11 +104,10 @@ class ConfigInstall(object): self.manifest.run_type_manifest(cdist_object) cdist_object.state = core.CdistObject.STATE_PREPARED - def object_run(self, cdist_object): + def object_run(self, cdist_object, dry_run=False): """Run gencode and code for an object""" self.log.debug("Trying to run object " + cdist_object.name) if cdist_object.state == core.CdistObject.STATE_DONE: - # TODO: remove once we are sure that this really never happens. raise cdist.Error("Attempting to run an already finished object: %s", cdist_object) cdist_type = cdist_object.cdist_type @@ -121,51 +120,74 @@ class ConfigInstall(object): cdist_object.changed = True # Execute - if cdist_object.code_local: - self.code.run_code_local(cdist_object) - if cdist_object.code_remote: - self.code.transfer_code_remote(cdist_object) - self.code.run_code_remote(cdist_object) + if not dry_run: + if cdist_object.code_local: + self.code.run_code_local(cdist_object) + if cdist_object.code_remote: + self.code.transfer_code_remote(cdist_object) + self.code.run_code_remote(cdist_object) # Mark this object as done self.log.debug("Finishing run of " + cdist_object.name) cdist_object.state = core.CdistObject.STATE_DONE + def stage_run_prepare(self): + """Prepare the run stage""" + + self.objects = core.CdistObject.list_objects( + self.local.object_path, + self.local.type_path) + + self.all_resolved = False + self.objects_changed = False + + print("srp: %s - %s objects: %s" % (self.local.object_path, self.local.type_path, list(self.objects))) + def stage_run(self): """The final (and real) step of deployment""" self.log.info("Generating and executing code") - - objects = core.CdistObject.list_objects( - self.local.object_path, - self.local.type_path) + self.stage_run_prepare() # FIXME: # - think about parallel execution (same for stage_prepare) # - catch unresolvable trees - all_resolved = False - objects_changed = False - while not all_resolved: + while not self.all_resolved: + self.stage_run_iterate() - object_state_list=' '.join('%s:%s:%s' % (o, o.state, o.all_requirements()) for o in objects) - self.log.debug("Object state (name:state:requirements): %s" % object_state_list) + def stage_run_iterate(self): + logging.root.setLevel(logging.DEBUG) + """ + Run one iteration of the run - all_resolved = True - for cdist_object in objects: + To be repeated until all objects are done + """ + + object_state_list=' '.join('%s:%s:%s' % (o, o.state, o.all_requirements()) for o in self.objects) + self.log.debug("Object state (name:state:requirements): %s" % object_state_list) + print("Object state (name:state:requirements): %s" % object_state_list) + + self.all_resolved = True + for cdist_object in self.objects: + if not cdist_object.state == cdist_object.STATE_DONE: + self.all_resolved = False + if cdist_object.satisfied_requirements: + self.object_run(cdist_object) + self.objects_changed = True + + # Not all are resolved, but nothing has been changed => bad dependencies! + if not self.objects_changed and not self.all_resolved: + # Create list of unfinished objects + their requirements for print + + evil_objects = [] + good_objects = [] + for cdist_object in self.objects: if not cdist_object.state == cdist_object.STATE_DONE: - all_resolved = False - if cdist_object.satisfied_requirements: - self.object_run(cdist_object) - objects_changed = True + evil_objects.append("%s: required: %s, autorequired: %s" % + (cdist_object.name, cdist_object.requirements, cdist_object.autorequire)) + else: + evil_objects.append("%s (%s): required: %s, autorequired: %s" % + (cdist_object.state, cdist_object.name, + cdist_object.requirements, cdist_object.autorequire)) - # Reran, but no object was solved - if not objects_changed: - # Create list of unfinished objects + their requirements for print - - evil_objects = [] - for cdist_object in objects: - if not cdist_object.state == cdist_object.STATE_DONE: - evil_objects.append("%s (required: %s, autorequired: %s" % - (cdist_object.name, cdist_object.requirements, cdist_object.autorequire)) - - raise cdist.Error("Cannot solve requirements for the following objects: %s" % - (",".join(evil_objects))) + errormessage = "Cannot solve requirements for the following objects: %s - solved: %s" % (",".join(evil_objects), ",".join(good_objects)) + raise cdist.Error(errormessage) From 4fd27e76a7ca4aea212097e4bdb6a12bded8ce0a Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Wed, 19 Dec 2012 19:37:43 +0100 Subject: [PATCH 31/37] support dry_run in stage_run_iterate as well Signed-off-by: Nico Schottelius --- cdist/config_install.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/cdist/config_install.py b/cdist/config_install.py index b7662ad8..9de5ecb8 100644 --- a/cdist/config_install.py +++ b/cdist/config_install.py @@ -56,6 +56,9 @@ class ConfigInstall(object): self.manifest = core.Manifest(self.context.target_host, self.local) self.code = core.Code(self.context.target_host, self.local, self.remote) + # Add switch to disable code execution + self.dry_run = False + def cleanup(self): # FIXME: move to local? destination = os.path.join(self.local.cache_path, self.context.target_host) @@ -171,7 +174,7 @@ class ConfigInstall(object): if not cdist_object.state == cdist_object.STATE_DONE: self.all_resolved = False if cdist_object.satisfied_requirements: - self.object_run(cdist_object) + self.object_run(cdist_object, self.dry_run) self.objects_changed = True # Not all are resolved, but nothing has been changed => bad dependencies! From 527ec0889bb64ce3977df908cf89d7319cadd4ab Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Wed, 19 Dec 2012 20:53:38 +0100 Subject: [PATCH 32/37] remove unneeded fixtures Signed-off-by: Nico Schottelius --- .../config_install/fixtures/object/__first/child/.cdist/.keep | 0 .../test/config_install/fixtures/object/__first/dog/.cdist/.keep | 0 .../config_install/fixtures/object/__first/woman/.cdist/.keep | 0 .../fixtures/object/__second/under-the/.cdist/.keep | 0 4 files changed, 0 insertions(+), 0 deletions(-) delete mode 100644 cdist/test/config_install/fixtures/object/__first/child/.cdist/.keep delete mode 100644 cdist/test/config_install/fixtures/object/__first/dog/.cdist/.keep delete mode 100644 cdist/test/config_install/fixtures/object/__first/woman/.cdist/.keep delete mode 100644 cdist/test/config_install/fixtures/object/__second/under-the/.cdist/.keep diff --git a/cdist/test/config_install/fixtures/object/__first/child/.cdist/.keep b/cdist/test/config_install/fixtures/object/__first/child/.cdist/.keep deleted file mode 100644 index e69de29b..00000000 diff --git a/cdist/test/config_install/fixtures/object/__first/dog/.cdist/.keep b/cdist/test/config_install/fixtures/object/__first/dog/.cdist/.keep deleted file mode 100644 index e69de29b..00000000 diff --git a/cdist/test/config_install/fixtures/object/__first/woman/.cdist/.keep b/cdist/test/config_install/fixtures/object/__first/woman/.cdist/.keep deleted file mode 100644 index e69de29b..00000000 diff --git a/cdist/test/config_install/fixtures/object/__second/under-the/.cdist/.keep b/cdist/test/config_install/fixtures/object/__second/under-the/.cdist/.keep deleted file mode 100644 index e69de29b..00000000 From 2732a4ba5c3ea75fccb7aa833b6799d021255140 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Wed, 19 Dec 2012 21:10:18 +0100 Subject: [PATCH 33/37] finally finish the dynamic resolver Signed-off-by: Nico Schottelius --- cdist/config_install.py | 64 ++++++++------------- cdist/core/cdist_object.py | 6 +- cdist/test/config_install/__init__.py | 82 ++++++++++++++++++++------- cdist/test/object/__init__.py | 25 ++++---- 4 files changed, 105 insertions(+), 72 deletions(-) diff --git a/cdist/config_install.py b/cdist/config_install.py index 9de5ecb8..f1529cc1 100644 --- a/cdist/config_install.py +++ b/cdist/config_install.py @@ -43,29 +43,25 @@ class ConfigInstall(object): self.context = context self.log = logging.getLogger(self.context.target_host) - # For easy access - self.local = context.local - self.remote = context.remote - # Initialise local directory structure - self.local.create_files_dirs() + self.context.local.create_files_dirs() # Initialise remote directory structure - self.remote.create_files_dirs() + self.context.remote.create_files_dirs() - self.explorer = core.Explorer(self.context.target_host, self.local, self.remote) - self.manifest = core.Manifest(self.context.target_host, self.local) - self.code = core.Code(self.context.target_host, self.local, self.remote) + self.explorer = core.Explorer(self.context.target_host, self.context.local, self.context.remote) + self.manifest = core.Manifest(self.context.target_host, self.context.local) + self.code = core.Code(self.context.target_host, self.context.local, self.context.remote) # Add switch to disable code execution self.dry_run = False def cleanup(self): # FIXME: move to local? - destination = os.path.join(self.local.cache_path, self.context.target_host) - self.log.debug("Saving " + self.local.out_path + " to " + destination) + destination = os.path.join(self.context.local.cache_path, self.context.target_host) + self.log.debug("Saving " + self.context.local.out_path + " to " + destination) if os.path.exists(destination): shutil.rmtree(destination) - shutil.move(self.local.out_path, destination) + shutil.move(self.context.local.out_path, destination) def deploy_to(self): """Mimic the old deploy to: Deploy to one host""" @@ -82,7 +78,7 @@ class ConfigInstall(object): def stage_prepare(self): """Do everything for a deploy, minus the actual code stage""" - self.explorer.run_global_explorers(self.local.global_explorer_out_path) + self.explorer.run_global_explorers(self.context.local.global_explorer_out_path) self.manifest.run_initial_manifest(self.context.initial_manifest) self.log.info("Running object manifests and type explorers") @@ -91,8 +87,8 @@ class ConfigInstall(object): new_objects_created = True while new_objects_created: new_objects_created = False - for cdist_object in core.CdistObject.list_objects(self.local.object_path, - self.local.type_path): + for cdist_object in core.CdistObject.list_objects(self.context.local.object_path, + self.context.local.type_path): if cdist_object.state == core.CdistObject.STATE_PREPARED: self.log.debug("Skipping re-prepare of object %s", cdist_object) continue @@ -134,56 +130,46 @@ class ConfigInstall(object): self.log.debug("Finishing run of " + cdist_object.name) cdist_object.state = core.CdistObject.STATE_DONE - def stage_run_prepare(self): - """Prepare the run stage""" - - self.objects = core.CdistObject.list_objects( - self.local.object_path, - self.local.type_path) - - self.all_resolved = False - self.objects_changed = False - - print("srp: %s - %s objects: %s" % (self.local.object_path, self.local.type_path, list(self.objects))) - def stage_run(self): """The final (and real) step of deployment""" self.log.info("Generating and executing code") - self.stage_run_prepare() - # FIXME: - # - think about parallel execution (same for stage_prepare) - # - catch unresolvable trees + # FIXME: think about parallel execution (same for stage_prepare) + self.all_resolved = False while not self.all_resolved: self.stage_run_iterate() def stage_run_iterate(self): - logging.root.setLevel(logging.DEBUG) """ Run one iteration of the run To be repeated until all objects are done """ + objects = list(core.CdistObject.list_objects(self.context.local.object_path, self.context.local.type_path)) + object_state_list=' '.join('%s:%s:%s:%s' % (o, o.state, o.all_requirements, o.satisfied_requirements) for o in objects) - object_state_list=' '.join('%s:%s:%s' % (o, o.state, o.all_requirements()) for o in self.objects) - self.log.debug("Object state (name:state:requirements): %s" % object_state_list) - print("Object state (name:state:requirements): %s" % object_state_list) + self.log.debug("Object state (name:state:requirements:satisfied): %s" % object_state_list) + objects_changed = False self.all_resolved = True - for cdist_object in self.objects: + for cdist_object in objects: if not cdist_object.state == cdist_object.STATE_DONE: self.all_resolved = False + self.log.debug("Object %s not done" % cdist_object.name) if cdist_object.satisfied_requirements: + self.log.debug("Running object %s with satisfied requirements" % cdist_object.name) self.object_run(cdist_object, self.dry_run) - self.objects_changed = True + objects_changed = True + + self.log.debug("All resolved: %s Objects changed: %s" % (self.all_resolved, objects_changed)) # Not all are resolved, but nothing has been changed => bad dependencies! - if not self.objects_changed and not self.all_resolved: + if not objects_changed and not self.all_resolved: # Create list of unfinished objects + their requirements for print evil_objects = [] good_objects = [] - for cdist_object in self.objects: + for cdist_object in objects: if not cdist_object.state == cdist_object.STATE_DONE: evil_objects.append("%s: required: %s, autorequired: %s" % (cdist_object.name, cdist_object.requirements, cdist_object.autorequire)) diff --git a/cdist/core/cdist_object.py b/cdist/core/cdist_object.py index cc9aeaa5..7beea130 100644 --- a/cdist/core/cdist_object.py +++ b/cdist/core/cdist_object.py @@ -211,15 +211,18 @@ class CdistObject(object): except EnvironmentError as error: raise cdist.Error('Error creating directories for cdist object: %s: %s' % (self, error)) + @property def satisfied_requirements(self): """Return state whether all of our dependencies have been resolved already""" satisfied = True - for requirement in self.all_requirements(): + for requirement in self.all_requirements: + log.debug("%s: Checking requirement %s (%s) .." % (self.name, requirement.name, requirement.state)) if not requirement.state == self.STATE_DONE: satisfied = False break + log.debug("%s is satisfied: %s" % (self.name, satisfied)) return satisfied @@ -251,6 +254,7 @@ class CdistObject(object): else: raise RequirementNotFoundError(pattern) + @property def all_requirements(self): """ Return resolved autorequirements and requirements so that diff --git a/cdist/test/config_install/__init__.py b/cdist/test/config_install/__init__.py index 326632b7..2a05d4c5 100644 --- a/cdist/test/config_install/__init__.py +++ b/cdist/test/config_install/__init__.py @@ -28,6 +28,7 @@ from cdist import core import cdist import cdist.context +import cdist.config import os.path as op my_dir = op.abspath(op.dirname(__file__)) @@ -40,30 +41,40 @@ class ConfigInstallRunTestCase(test.CdistTestCase): def setUp(self): + # Change env for context + self.orig_environ = os.environ + os.environ = os.environ.copy() + self.temp_dir = self.mkdtemp() + + self.out_dir = os.path.join(self.temp_dir, "out") + self.remote_out_dir = os.path.join(self.temp_dir, "remote") + + os.environ['__cdist_out_dir'] = self.out_dir + os.environ['__cdist_remote_out_dir'] = self.remote_out_dir + self.context = cdist.context.Context( target_host=self.target_host, remote_copy=self.remote_copy, remote_exec=self.remote_exec, - initial_manifest=args.manifest, - add_conf_dirs=add_conf_dir, exec_path=test.cdist_exec_path, - debug=False) + debug=True) - self.config = config.Config(self.context) + self.context.local.object_path = object_base_path + self.context.local.type_path = type_base_path + + self.config = cdist.config.Config(self.context) - def setUp(self): self.objects = list(core.CdistObject.list_objects(object_base_path, type_base_path)) self.object_index = dict((o.name, o) for o in self.objects) self.object_names = [o.name for o in self.objects] - print(self.objects) - - self.cdist_type = core.CdistType(type_base_path, '__third') - self.cdist_object = core.CdistObject(self.cdist_type, object_base_path, 'moon') - def tearDown(self): for o in self.objects: o.requirements = [] + o.state = "" + + os.environ = self.orig_environ + shutil.rmtree(self.temp_dir) def test_dependency_resolution(self): first = self.object_index['__first/man'] @@ -73,21 +84,50 @@ class ConfigInstallRunTestCase(test.CdistTestCase): first.requirements = [second.name] second.requirements = [third.name] - self.config.stage_run_prepare() - # First run: # solves first and maybe second (depending on the order in the set) self.config.stage_run_iterate() + self.assertTrue(third.state == third.STATE_DONE) - # FIXME :-) - self.assertTrue(False) -# self.assertEqual( -# self.dependency_resolver.dependencies['__first/man'], -# [third_moon, second_on_the, first_man] -# ) + self.config.stage_run_iterate() + self.assertTrue(second.state == second.STATE_DONE) + + + try: + self.config.stage_run_iterate() + except cdist.Error: + # Allow failing, because the third run may or may not be unecessary already, + # depending on the order of the objects + pass + self.assertTrue(first.state == first.STATE_DONE) + + + def test_non_empty_object_list(self): + """Ensure the object list returned is not empty""" + pass def test_requirement_not_found(self): - first_man = self.object_index['__first/man'] - first_man.requirements = ['__does/not/exist'] + """Ensure an exception is thrown for missing depedencies""" + cdist_object = self.object_index['__first/man'] + cdist_object.requirements = ['__does/not/exist'] + with self.assertRaises(core.cdist_object.RequirementNotFoundError): - first_man.find_requirements_by_name(first_man.requirements) + # Use list, as generator does not (yet) raise the error + list(cdist_object.find_requirements_by_name(cdist_object.requirements)) + + def test_unresolvable_requirements(self): + """Ensure an exception is thrown for unresolvable depedencies""" + + # Create to objects depending on each other - no solution possible + first = self.object_index['__first/man'] + second = self.object_index['__second/on-the'] + + first.requirements = [second.name] + second.requirements = [first.name] + + # First round solves __third/moon + self.config.stage_run_iterate() + + # Second round detects it cannot solve the rest + with self.assertRaises(cdist.Error): + self.config.stage_run_iterate() diff --git a/cdist/test/object/__init__.py b/cdist/test/object/__init__.py index ec6e25c1..f3a526aa 100644 --- a/cdist/test/object/__init__.py +++ b/cdist/test/object/__init__.py @@ -36,9 +36,8 @@ type_base_path = op.join(fixtures, 'type') class ObjectClassTestCase(test.CdistTestCase): - def test_list_object_names(self): - found_object_names = sorted(list(core.CdistObject.list_object_names(object_base_path))) - expected_object_names = sorted([ + def setUp(self): + self.expected_object_names = sorted([ '__first/child', '__first/dog', '__first/man', @@ -46,20 +45,24 @@ class ObjectClassTestCase(test.CdistTestCase): '__second/on-the', '__second/under-the', '__third/moon']) - self.assertEqual(found_object_names, expected_object_names) + + self.expected_objects = [] + for cdist_object_name in self.expected_object_names: + cdist_type, cdist_object_id = cdist_object_name.split("/", maxsplit=1) + cdist_object = core.CdistObject(core.CdistType(type_base_path, cdist_type), object_base_path, cdist_object_id) + self.expected_objects.append(cdist_object) + + def test_list_object_names(self): + found_object_names = sorted(list(core.CdistObject.list_object_names(object_base_path))) + self.assertEqual(found_object_names, self.expected_object_names) def test_list_type_names(self): type_names = list(cdist.core.CdistObject.list_type_names(object_base_path)) self.assertEqual(type_names, ['__first', '__second', '__third']) def test_list_objects(self): - objects = list(core.CdistObject.list_objects(object_base_path, type_base_path)) - objects_expected = [ - core.CdistObject(core.CdistType(type_base_path, '__first'), object_base_path, 'man'), - core.CdistObject(core.CdistType(type_base_path, '__second'), object_base_path, 'on-the'), - core.CdistObject(core.CdistType(type_base_path, '__third'), object_base_path, 'moon'), - ] - self.assertEqual(objects, objects_expected) + found_objects = list(core.CdistObject.list_objects(object_base_path, type_base_path)) + self.assertEqual(found_objects, self.expected_objects) class ObjectIdTestCase(test.CdistTestCase): From ef45cd5ce063ce4aace83bbba1091153dd6ef32e Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Wed, 19 Dec 2012 21:14:07 +0100 Subject: [PATCH 34/37] fix removal of context alias in config_install Signed-off-by: Nico Schottelius --- cdist/test/autorequire/__init__.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cdist/test/autorequire/__init__.py b/cdist/test/autorequire/__init__.py index 2a647954..330680df 100644 --- a/cdist/test/autorequire/__init__.py +++ b/cdist/test/autorequire/__init__.py @@ -65,10 +65,10 @@ class AutorequireTestCase(test.CdistTestCase): shutil.rmtree(self.temp_dir) def test_implicit_dependencies(self): - self.context.initial_manifest = os.path.join(self.config.local.manifest_path, 'implicit_dependencies') + self.context.initial_manifest = os.path.join(self.context.local.manifest_path, 'implicit_dependencies') self.config.stage_prepare() - objects = core.CdistObject.list_objects(self.config.local.object_path, self.config.local.type_path) + objects = core.CdistObject.list_objects(self.context.local.object_path, self.context.local.type_path) dependency_resolver = resolver.DependencyResolver(objects) expected_dependencies = [ dependency_resolver.objects['__package_special/b'], @@ -79,7 +79,7 @@ class AutorequireTestCase(test.CdistTestCase): self.assertEqual(resolved_dependencies, expected_dependencies) def test_circular_dependency(self): - self.context.initial_manifest = os.path.join(self.config.local.manifest_path, 'circular_dependency') + self.context.initial_manifest = os.path.join(self.context.local.manifest_path, 'circular_dependency') self.config.stage_prepare() # raises CircularDependecyError self.config.stage_run() From e03f5d08dd616742d558f72545e9d9eedcf7a192 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Wed, 19 Dec 2012 21:14:23 +0100 Subject: [PATCH 35/37] move / correct tests in object and config_install Signed-off-by: Nico Schottelius --- cdist/test/config_install/__init__.py | 14 -------------- cdist/test/object/__init__.py | 24 ++++++------------------ 2 files changed, 6 insertions(+), 32 deletions(-) diff --git a/cdist/test/config_install/__init__.py b/cdist/test/config_install/__init__.py index 2a05d4c5..2abf7614 100644 --- a/cdist/test/config_install/__init__.py +++ b/cdist/test/config_install/__init__.py @@ -101,20 +101,6 @@ class ConfigInstallRunTestCase(test.CdistTestCase): pass self.assertTrue(first.state == first.STATE_DONE) - - def test_non_empty_object_list(self): - """Ensure the object list returned is not empty""" - pass - - def test_requirement_not_found(self): - """Ensure an exception is thrown for missing depedencies""" - cdist_object = self.object_index['__first/man'] - cdist_object.requirements = ['__does/not/exist'] - - with self.assertRaises(core.cdist_object.RequirementNotFoundError): - # Use list, as generator does not (yet) raise the error - list(cdist_object.find_requirements_by_name(cdist_object.requirements)) - def test_unresolvable_requirements(self): """Ensure an exception is thrown for unresolvable depedencies""" diff --git a/cdist/test/object/__init__.py b/cdist/test/object/__init__.py index f3a526aa..c4f46cd1 100644 --- a/cdist/test/object/__init__.py +++ b/cdist/test/object/__init__.py @@ -255,23 +255,11 @@ class ObjectResolveRequirementsTestCase(test.CdistTestCase): self.assertEqual(expected_requirements, found_requirements) - def test_dependency_resolution(self): - first_man = self.object_index['__first/man'] - second_on_the = self.object_index['__second/on-the'] - third_moon = self.object_index['__third/moon'] - first_man.requirements = [second_on_the.name] - second_on_the.requirements = [third_moon.name] - - # FIXME :-) - self.assertTrue(False) -# self.assertEqual( -# self.dependency_resolver.dependencies['__first/man'], -# [third_moon, second_on_the, first_man] -# ) - def test_requirement_not_found(self): - first_man = self.object_index['__first/man'] - first_man.requirements = ['__does/not/exist'] - with self.assertRaises(core.cdist_object.RequirementNotFoundError): - first_man.find_requirements_by_name(first_man.requirements) + """Ensure an exception is thrown for missing depedencies""" + cdist_object = self.object_index['__first/man'] + cdist_object.requirements = ['__does/not/exist'] + with self.assertRaises(core.cdist_object.RequirementNotFoundError): + # Use list, as generator does not (yet) raise the error + list(cdist_object.find_requirements_by_name(cdist_object.requirements)) From fbda50dfb0a5fef44be698187bf4f786ab4dd0f0 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Wed, 19 Dec 2012 21:16:22 +0100 Subject: [PATCH 36/37] ++changes Signed-off-by: Nico Schottelius --- docs/changelog | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/changelog b/docs/changelog index 319f7a38..12baa1a2 100644 --- a/docs/changelog +++ b/docs/changelog @@ -6,6 +6,7 @@ Changelog next: * Type __jail: State absent should implies stopped (Jake Guffey) + * Core: Use dynamic dependency resolver to allow indirect self dependencies 2.1.0: 2012-12-09 * Core: Ensure global explorers are executable From acb31b2632778e1dc2eaf32fbf9d2bc5cf8d4da6 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Wed, 19 Dec 2012 22:41:20 +0100 Subject: [PATCH 37/37] do not push to ethz Signed-off-by: Nico Schottelius --- build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build b/build index 572fb768..1f408b94 100755 --- a/build +++ b/build @@ -349,7 +349,7 @@ eof ;; p|pu|pub) - for remote in "" github sf ethz; do + for remote in "" github sf; do echo "Pushing to $remote" git push --mirror $remote done