Add 'Parallelization' chapter to docs.

This commit is contained in:
Darko Poljak 2016-08-17 19:32:16 +02:00
parent 640b7f9613
commit 80304d3340
3 changed files with 75 additions and 0 deletions

View file

@ -1,6 +1,7 @@
Changelog
---------
next:
* Documentation: Add Parallelization chapter (Darko Poljak)
* Core: Add -b, --enable-beta option for enabling beta functionalities (Darko Poljak)
* Core: Add -j, --jobs option for parallel execution and add parallel support for global explorers (currently in beta) (Darko Poljak)
* Core: Add derived env vars for target hostname and fqdn (Darko Poljak)

View file

@ -0,0 +1,73 @@
Parallelization
===============
Description
-----------
cdist has two modes of parallel operation.
One of them is to operate on each host in separate process. This is enabled
with :strong:`-p/--parallel` option.
The other way is to operate in parallel within one host where you specify
the number of jobs. This is enabled with :strong:`-j/--jobs` option where you
can specify the number of parallel jobs. By default,
:strong:`multiprocessing.cpu_count()` is used. For this mode only global
explorers are currently supported and this option is still in :strong:`beta`.
You can, of course, use those two options together. This means that each host
will be processed by its own process. Within each process cdist will operate
using specified number of parallel jobs.
For more info on those options see :strong:`cdist`\ (1).
Examples
--------
.. code-block:: sh
# Configure hosts read from file hosts.file in parallel
$ cdist config -p -f hosts.file
# Configure hosts read from file hosts.file sequentially but using default
# number of parallel jobs
$ cdist config -b -j -f hosts.file
# Configure hosts read from file hosts.file in parallel using 16
# parallel jobs
$ cdist config -b -j 16 -p -f hosts.file
Caveats
-------
When operating in parallel, either by operating in parallel for each host
(-p/--parallel) or by parallel jobs within a host (-j/--jobs), and depending
on target SSH server and its configuration you may encounter connection drops.
This is controlled with sshd :strong:MaxStartups configuration options.
You may also encounter session open refusal. This happens with ssh multiplexing
when you reach maximum number of open sessions permitted per network
connection. In this case ssh will disable multiplexing.
This limit is controlled with sshd :strong:MaxSessions configuration
options. For more details refer to :strong:`sshd_config`\ (5).
For example, if you reach :strong:`MaxSessions` sessions you may get the
following output:
.. code-block:: sh
$ cdist config -b -j 11 -v 78.47.116.244
INFO: cdist: version 4.2.2-55-g640b7f9
INFO: 78.47.116.244: Running global explorers
INFO: 78.47.116.244: Remote transfer in 11 parallel jobs
channel 22: open failed: administratively prohibited: open failed
mux_client_request_session: session request failed: Session open refused by peer
ControlSocket /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/ssh-control-path already exists, disabling multiplexing
INFO: 78.47.116.244: Running global explorers in 11 parallel jobs
channel 22: open failed: administratively prohibited: open failed
mux_client_request_session: session request failed: Session open refused by peer
ControlSocket /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/ssh-control-path already exists, disabling multiplexing
INFO: 78.47.116.244: Running initial manifest /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/data/conf/manifest/init
INFO: 78.47.116.244: Running manifest and explorers for __file/root/host.file
INFO: 78.47.116.244: Generating code for __file/root/host.file
INFO: 78.47.116.244: Finished successful run in 18.655028820037842 seconds
INFO: cdist: Total processing time for 1 host(s): 19.159148693084717

View file

@ -23,6 +23,7 @@ Contents:
cdist-types
cdist-explorer
cdist-messaging
cdist-parallelization
cdist-reference
cdist-best-practice
cdist-stages