From 735f57b3a032fad842c9d4306080b5d7a4c53a38 Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Sun, 5 May 2019 17:55:02 +0200 Subject: [PATCH 1/3] Add 'Perils of CDIST_ORDER_DEPENDENCY' sub-section --- docs/src/cdist-best-practice.rst | 92 ++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) diff --git a/docs/src/cdist-best-practice.rst b/docs/src/cdist-best-practice.rst index a99ba88e..7a1255d6 100644 --- a/docs/src/cdist-best-practice.rst +++ b/docs/src/cdist-best-practice.rst @@ -224,3 +224,95 @@ in the repository for such content: It allows you to easily distinguish what is used by cdist and what is not and also to store all important files in one repository. + + +Perils of CDIST_ORDER_DEPENDENCY +-------------------------------- +With CDIST_ORDER_DEPENDENCY all types are executed in the order in which they +are created in the manifest. The current created object automatically depends +on the previously created object. + +It essentially helps you to build up blocks of code that build upon each other +(like first creating the directory xyz than the file below the directory). + +This can be helpful, but it can also be the source of *evil*. + +Let's see an example. Suppose you have special init manifest where among other +things you are assuring that remote host has packages `sudo` and `curl` +installed. + +**init1** + +.. code-block:: sh + + CDIST_ORDER_DEPENDENCY=1 + export CDIST_ORDER_DEPENDENCY + + for p in sudo curl + do + __package "${p}" + done + +Then you have some other special init manifest where among other things you are +assuring `sudo` package is installed. + +**init2** + +.. code-block:: sh + + CDIST_ORDER_DEPENDENCY=1 + export CDIST_ORDER_DEPENDENCY + + __package sudo + +Then you have third init manifest where you combine those two init manifests, +by including them: + +**init** + +.. code-block:: sh + + sh -e "$__manifest/init1" + sh -e "$__manifest/init2" + +The resulting init manifest is then equal to: + +.. code-block:: sh + + CDIST_ORDER_DEPENDENCY=1 + export CDIST_ORDER_DEPENDENCY + + for p in sudo curl + do + __package "${p}" + done + + CDIST_ORDER_DEPENDENCY=1 + export CDIST_ORDER_DEPENDENCY + + __package sudo + +In the end you get the following dependencies: + +* `__package/curl` depends on `__package/sudo` +* `__package/sudo` depends on `__package/curl` + +And here you have a circular dependency! + +In the real world manifest can be quite complex, dependencies can become +complicated and circual dependencies are not so obvious. Resolving it can +become cumbersome. + +**Practical solution?** + +Instead of managing complex init manifests you can write custom types. +Each custom type can do one thing, it has well defined dependencies that will +not leak into init manifest. In custom type you can also add special explorers +and gencode. + +Then, in init manifest you combine your complex types. It is: + +* cleaner +* easier to follow +* easier to maintain +* easier to debug. From 28082c710aa02a1861a4396513469ecfa7ef2d4f Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Mon, 6 May 2019 11:11:10 +0200 Subject: [PATCH 2/3] Add refs to perils of CDIST_ORDER_DEPENDENCY --- docs/src/cdist-manifest.rst | 2 ++ docs/src/cdist-reference.rst.sh | 1 + 2 files changed, 3 insertions(+) diff --git a/docs/src/cdist-manifest.rst b/docs/src/cdist-manifest.rst index 0e266943..4dd3e74b 100644 --- a/docs/src/cdist-manifest.rst +++ b/docs/src/cdist-manifest.rst @@ -163,6 +163,8 @@ automatically depends on the previously created object. It essentially helps you to build up blocks of code that build upon each other (like first creating the directory xyz than the file below the directory). +Read also about `perils of CDIST_ORDER_DEPENDENCY `_. + Overrides --------- diff --git a/docs/src/cdist-reference.rst.sh b/docs/src/cdist-reference.rst.sh index 59ce018b..2c9c7b40 100755 --- a/docs/src/cdist-reference.rst.sh +++ b/docs/src/cdist-reference.rst.sh @@ -323,6 +323,7 @@ CDIST_OVERRIDE CDIST_ORDER_DEPENDENCY Create dependencies based on the execution order (see \`cdist manifest \`_). + Read also about \`perils of CDIST_ORDER_DEPENDENCY \`_. CDIST_REMOTE_EXEC Use this command for remote execution (should behave like ssh). From 02eb6c75a7979eb2fa1ec2b4921ab9fafe582edd Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Mon, 6 May 2019 17:11:23 +0200 Subject: [PATCH 3/3] Add 'CDIST_ORDER_DEPENDENCY kills parallelization' --- docs/src/cdist-best-practice.rst | 45 ++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/docs/src/cdist-best-practice.rst b/docs/src/cdist-best-practice.rst index 7a1255d6..a91f2cc0 100644 --- a/docs/src/cdist-best-practice.rst +++ b/docs/src/cdist-best-practice.rst @@ -237,6 +237,10 @@ It essentially helps you to build up blocks of code that build upon each other This can be helpful, but it can also be the source of *evil*. + +CDIST_ORDER_DEPENDENCY easily causes unobvious dependency cycles +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + Let's see an example. Suppose you have special init manifest where among other things you are assuring that remote host has packages `sudo` and `curl` installed. @@ -316,3 +320,44 @@ Then, in init manifest you combine your complex types. It is: * easier to follow * easier to maintain * easier to debug. + + +CDIST_ORDER_DEPENDENCY kills parallelization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Suppose you have defined CDIST_ORDER_DEPENDENCY and then, among other things, +you specify creation of three, by nature independent, files. + +**init** + +.. code-block:: sh + + CDIST_ORDER_DEPENDENCY=1 + export CDIST_ORDER_DEPENDENCY + + ... + __file /tmp/file1 + __file /tmp/file2 + __file /tmp/file3 + ... + +Due to defined CDIST_ORDER_DEPENDENCY cdist will execute them in specified order. +It is better to use CDIST_ORDER_DEPENDENCY in well defined blocks: + +**init** + +.. code-block:: sh + + CDIST_ORDER_DEPENDENCY=1 + export CDIST_ORDER_DEPENDENCY + ... + unset CDIST_ORDER_DEPENDENCY + + __file /tmp/file1 + __file /tmp/file2 + __file /tmp/file3 + + CDIST_ORDER_DEPENDENCY=1 + export CDIST_ORDER_DEPENDENCY + ... + unset CDIST_ORDER_DEPENDENCY