From 80304d33407d27e0fd3f7ef36b95bbb7b352bbce Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Wed, 17 Aug 2016 19:32:16 +0200 Subject: [PATCH] Add 'Parallelization' chapter to docs. --- docs/changelog | 1 + docs/src/cdist-parallelization.rst | 73 ++++++++++++++++++++++++++++++ docs/src/index.rst | 1 + 3 files changed, 75 insertions(+) create mode 100644 docs/src/cdist-parallelization.rst diff --git a/docs/changelog b/docs/changelog index d0d2c157..db86de45 100644 --- a/docs/changelog +++ b/docs/changelog @@ -1,6 +1,7 @@ Changelog --------- next: + * Documentation: Add Parallelization chapter (Darko Poljak) * Core: Add -b, --enable-beta option for enabling beta functionalities (Darko Poljak) * Core: Add -j, --jobs option for parallel execution and add parallel support for global explorers (currently in beta) (Darko Poljak) * Core: Add derived env vars for target hostname and fqdn (Darko Poljak) diff --git a/docs/src/cdist-parallelization.rst b/docs/src/cdist-parallelization.rst new file mode 100644 index 00000000..ce9f5bca --- /dev/null +++ b/docs/src/cdist-parallelization.rst @@ -0,0 +1,73 @@ +Parallelization +=============== + +Description +----------- +cdist has two modes of parallel operation. + +One of them is to operate on each host in separate process. This is enabled +with :strong:`-p/--parallel` option. + +The other way is to operate in parallel within one host where you specify +the number of jobs. This is enabled with :strong:`-j/--jobs` option where you +can specify the number of parallel jobs. By default, +:strong:`multiprocessing.cpu_count()` is used. For this mode only global +explorers are currently supported and this option is still in :strong:`beta`. + +You can, of course, use those two options together. This means that each host +will be processed by its own process. Within each process cdist will operate +using specified number of parallel jobs. + +For more info on those options see :strong:`cdist`\ (1). + + +Examples +-------- + +.. code-block:: sh + + # Configure hosts read from file hosts.file in parallel + $ cdist config -p -f hosts.file + + # Configure hosts read from file hosts.file sequentially but using default + # number of parallel jobs + $ cdist config -b -j -f hosts.file + + # Configure hosts read from file hosts.file in parallel using 16 + # parallel jobs + $ cdist config -b -j 16 -p -f hosts.file + + +Caveats +------- +When operating in parallel, either by operating in parallel for each host +(-p/--parallel) or by parallel jobs within a host (-j/--jobs), and depending +on target SSH server and its configuration you may encounter connection drops. +This is controlled with sshd :strong:MaxStartups configuration options. +You may also encounter session open refusal. This happens with ssh multiplexing +when you reach maximum number of open sessions permitted per network +connection. In this case ssh will disable multiplexing. +This limit is controlled with sshd :strong:MaxSessions configuration +options. For more details refer to :strong:`sshd_config`\ (5). + +For example, if you reach :strong:`MaxSessions` sessions you may get the +following output: + +.. code-block:: sh + + $ cdist config -b -j 11 -v 78.47.116.244 + INFO: cdist: version 4.2.2-55-g640b7f9 + INFO: 78.47.116.244: Running global explorers + INFO: 78.47.116.244: Remote transfer in 11 parallel jobs + channel 22: open failed: administratively prohibited: open failed + mux_client_request_session: session request failed: Session open refused by peer + ControlSocket /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/ssh-control-path already exists, disabling multiplexing + INFO: 78.47.116.244: Running global explorers in 11 parallel jobs + channel 22: open failed: administratively prohibited: open failed + mux_client_request_session: session request failed: Session open refused by peer + ControlSocket /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/ssh-control-path already exists, disabling multiplexing + INFO: 78.47.116.244: Running initial manifest /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/data/conf/manifest/init + INFO: 78.47.116.244: Running manifest and explorers for __file/root/host.file + INFO: 78.47.116.244: Generating code for __file/root/host.file + INFO: 78.47.116.244: Finished successful run in 18.655028820037842 seconds + INFO: cdist: Total processing time for 1 host(s): 19.159148693084717 diff --git a/docs/src/index.rst b/docs/src/index.rst index a34231f8..b33b707d 100644 --- a/docs/src/index.rst +++ b/docs/src/index.rst @@ -23,6 +23,7 @@ Contents: cdist-types cdist-explorer cdist-messaging + cdist-parallelization cdist-reference cdist-best-practice cdist-stages