From 332f5dcff9b27e110c60bc94aa9c2f3c40f44121 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Sat, 23 Nov 2019 00:25:51 +0100 Subject: [PATCH] Redefine/reimplement CDIST_ORDER_DEPENDENCY Update documentation. --- docs/src/cdist-best-practice.rst | 91 +---------------------- docs/src/cdist-manifest.rst | 121 ++++++++++++++++++++++++++++++- docs/src/cdist-reference.rst.sh | 2 +- docs/src/man1/cdist.rst | 2 + 4 files changed, 126 insertions(+), 90 deletions(-) diff --git a/docs/src/cdist-best-practice.rst b/docs/src/cdist-best-practice.rst index a91f2cc0..39ec453e 100644 --- a/docs/src/cdist-best-practice.rst +++ b/docs/src/cdist-best-practice.rst @@ -226,8 +226,8 @@ and also to store all important files in one repository. -Perils of CDIST_ORDER_DEPENDENCY --------------------------------- +Notes on CDIST_ORDER_DEPENDENCY +------------------------------- With CDIST_ORDER_DEPENDENCY all types are executed in the order in which they are created in the manifest. The current created object automatically depends on the previously created object. @@ -235,96 +235,11 @@ on the previously created object. It essentially helps you to build up blocks of code that build upon each other (like first creating the directory xyz than the file below the directory). -This can be helpful, but it can also be the source of *evil*. - - -CDIST_ORDER_DEPENDENCY easily causes unobvious dependency cycles -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Let's see an example. Suppose you have special init manifest where among other -things you are assuring that remote host has packages `sudo` and `curl` -installed. - -**init1** - -.. code-block:: sh - - CDIST_ORDER_DEPENDENCY=1 - export CDIST_ORDER_DEPENDENCY - - for p in sudo curl - do - __package "${p}" - done - -Then you have some other special init manifest where among other things you are -assuring `sudo` package is installed. - -**init2** - -.. code-block:: sh - - CDIST_ORDER_DEPENDENCY=1 - export CDIST_ORDER_DEPENDENCY - - __package sudo - -Then you have third init manifest where you combine those two init manifests, -by including them: - -**init** - -.. code-block:: sh - - sh -e "$__manifest/init1" - sh -e "$__manifest/init2" - -The resulting init manifest is then equal to: - -.. code-block:: sh - - CDIST_ORDER_DEPENDENCY=1 - export CDIST_ORDER_DEPENDENCY - - for p in sudo curl - do - __package "${p}" - done - - CDIST_ORDER_DEPENDENCY=1 - export CDIST_ORDER_DEPENDENCY - - __package sudo - -In the end you get the following dependencies: - -* `__package/curl` depends on `__package/sudo` -* `__package/sudo` depends on `__package/curl` - -And here you have a circular dependency! - -In the real world manifest can be quite complex, dependencies can become -complicated and circual dependencies are not so obvious. Resolving it can -become cumbersome. - -**Practical solution?** - -Instead of managing complex init manifests you can write custom types. -Each custom type can do one thing, it has well defined dependencies that will -not leak into init manifest. In custom type you can also add special explorers -and gencode. - -Then, in init manifest you combine your complex types. It is: - -* cleaner -* easier to follow -* easier to maintain -* easier to debug. +This can be helpful, but one must be aware of its side effects. CDIST_ORDER_DEPENDENCY kills parallelization ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Suppose you have defined CDIST_ORDER_DEPENDENCY and then, among other things, you specify creation of three, by nature independent, files. diff --git a/docs/src/cdist-manifest.rst b/docs/src/cdist-manifest.rst index 4dd3e74b..5dbca479 100644 --- a/docs/src/cdist-manifest.rst +++ b/docs/src/cdist-manifest.rst @@ -163,7 +163,126 @@ automatically depends on the previously created object. It essentially helps you to build up blocks of code that build upon each other (like first creating the directory xyz than the file below the directory). -Read also about `perils of CDIST_ORDER_DEPENDENCY `_. +Read also about `notes on CDIST_ORDER_DEPENDENCY `_. + +In version 6.2.0 semantic CDIST_ORDER_DEPENDENCY is finally fixed and well defined. + +CDIST_ORDER_DEPENDENCY defines type order dependency context. Order dependency context +starts when CDIST_ORDER_DEPENDENCY is set, and ends when it is unset. After each +manifest execution finishes, any existing order dependency context is automatically +unset. This ensures that CDIST_ORDER_DEPENDENCY is valid within the manifest where it +is used. When order dependency context is defined then cdist executes types in the +order in which they are created in the manifest inside order dependency context. + +Sometimes the best way to see how something works is to see examples. + +Suppose you have defined **initial manifest**: + +.. code-block:: sh + + __cycle1 cycle1 + export CDIST_ORDER_DEPENDENCY=1 + __cycle2 cycle2 + __cycle3 cycle3 + +with types **__cycle1**: + +.. code-block:: sh + + export CDIST_ORDER_DEPENDENCY=1 + __file /tmp/cycle11 + __file /tmp/cycle12 + __file /tmp/cycle13 + +**__cycle2**: + +.. code-block:: sh + + __file /tmp/cycle21 + export CDIST_ORDER_DEPENDENCY=1 + __file /tmp/cycle22 + __file /tmp/cycle23 + unset CDIST_ORDER_DEPENDENCY + __file /tmp/cycle24 + +**__cycle3**: + +.. code-block:: sh + + __file /tmp/cycle31 + __file /tmp/cycle32 + export CDIST_ORDER_DEPENDENCY=1 + __file /tmp/cycle33 + __file /tmp/cycle34 + +For the above config, cdist results in the following expected *dependency graph* +(type *__cycleX* is shown as *cX*, *__file/tmp/cycleXY* is shown as *fcXY*): + +:: + + c1---->fc11 + | /\ + | | + +----->fc12 + | /\ + | | + +----->fc13 + + c2--+--->fc21 + /\ | + | | + | +----->fc22 + | | /\ + | | | + | +----->fc23 + | | + | | + | +----->fc24 + | + | + c3---->fc31 + | + | + +----->fc32 + | + | + +----->fc33 + | /\ + | | + +----->fc34 + +Before version 6.2.0 the above configuration would result in cycle: + +:: + + ERROR: 185.203.112.26: Cycle detected in object dependencies: + __file/tmp/cycle11 -> __cycle3/cycle3 -> __cycle2/cycle2 -> __cycle1/cycle1 -> __file/tmp/cycle11! + +The following manifest shows an example for order dependency contexts: + +.. code-block:: sh + + __file /tmp/fileA + export CDIST_ORDER_DEPENDENCY=1 + __file /tmp/fileB + __file /tmp/fileC + __file /tmp/fileD + unset CDIST_ORDER_DEPENDENCY + __file /tmp/fileE + __file /tmp/fileF + export CDIST_ORDER_DEPENDENCY=1 + __file /tmp/fileG + __file /tmp/fileH + unset CDIST_ORDER_DEPENDENCY + __file /tmp/fileI + +This means: + +* C depends on B +* D depends on C +* H depends on G + +and there are no other dependencies from this manifest. Overrides diff --git a/docs/src/cdist-reference.rst.sh b/docs/src/cdist-reference.rst.sh index 3ab12fe2..e77d98f6 100755 --- a/docs/src/cdist-reference.rst.sh +++ b/docs/src/cdist-reference.rst.sh @@ -330,7 +330,7 @@ CDIST_OVERRIDE CDIST_ORDER_DEPENDENCY Create dependencies based on the execution order (see \`cdist manifest \`_). - Read also about \`perils of CDIST_ORDER_DEPENDENCY \`_. + Note that in version 6.2.0 semantic of this processing mode is finally fixed and well defined. CDIST_REMOTE_EXEC Use this command for remote execution (should behave like ssh). diff --git a/docs/src/man1/cdist.rst b/docs/src/man1/cdist.rst index 7f368e68..f40491e9 100644 --- a/docs/src/man1/cdist.rst +++ b/docs/src/man1/cdist.rst @@ -827,6 +827,8 @@ CDIST_OVERRIDE CDIST_ORDER_DEPENDENCY Create dependencies based on the execution order. + Note that in version 6.2.0 semantic of this processing mode is + finally fixed and well defined. CDIST_REMOTE_EXEC Use this command for remote execution (should behave like ssh).