forked from ungleich-public/cdist
Add 'Parallelization' chapter to docs.
This commit is contained in:
parent
640b7f9613
commit
80304d3340
3 changed files with 75 additions and 0 deletions
|
@ -1,6 +1,7 @@
|
||||||
Changelog
|
Changelog
|
||||||
---------
|
---------
|
||||||
next:
|
next:
|
||||||
|
* Documentation: Add Parallelization chapter (Darko Poljak)
|
||||||
* Core: Add -b, --enable-beta option for enabling beta functionalities (Darko Poljak)
|
* Core: Add -b, --enable-beta option for enabling beta functionalities (Darko Poljak)
|
||||||
* Core: Add -j, --jobs option for parallel execution and add parallel support for global explorers (currently in beta) (Darko Poljak)
|
* Core: Add -j, --jobs option for parallel execution and add parallel support for global explorers (currently in beta) (Darko Poljak)
|
||||||
* Core: Add derived env vars for target hostname and fqdn (Darko Poljak)
|
* Core: Add derived env vars for target hostname and fqdn (Darko Poljak)
|
||||||
|
|
73
docs/src/cdist-parallelization.rst
Normal file
73
docs/src/cdist-parallelization.rst
Normal file
|
@ -0,0 +1,73 @@
|
||||||
|
Parallelization
|
||||||
|
===============
|
||||||
|
|
||||||
|
Description
|
||||||
|
-----------
|
||||||
|
cdist has two modes of parallel operation.
|
||||||
|
|
||||||
|
One of them is to operate on each host in separate process. This is enabled
|
||||||
|
with :strong:`-p/--parallel` option.
|
||||||
|
|
||||||
|
The other way is to operate in parallel within one host where you specify
|
||||||
|
the number of jobs. This is enabled with :strong:`-j/--jobs` option where you
|
||||||
|
can specify the number of parallel jobs. By default,
|
||||||
|
:strong:`multiprocessing.cpu_count()` is used. For this mode only global
|
||||||
|
explorers are currently supported and this option is still in :strong:`beta`.
|
||||||
|
|
||||||
|
You can, of course, use those two options together. This means that each host
|
||||||
|
will be processed by its own process. Within each process cdist will operate
|
||||||
|
using specified number of parallel jobs.
|
||||||
|
|
||||||
|
For more info on those options see :strong:`cdist`\ (1).
|
||||||
|
|
||||||
|
|
||||||
|
Examples
|
||||||
|
--------
|
||||||
|
|
||||||
|
.. code-block:: sh
|
||||||
|
|
||||||
|
# Configure hosts read from file hosts.file in parallel
|
||||||
|
$ cdist config -p -f hosts.file
|
||||||
|
|
||||||
|
# Configure hosts read from file hosts.file sequentially but using default
|
||||||
|
# number of parallel jobs
|
||||||
|
$ cdist config -b -j -f hosts.file
|
||||||
|
|
||||||
|
# Configure hosts read from file hosts.file in parallel using 16
|
||||||
|
# parallel jobs
|
||||||
|
$ cdist config -b -j 16 -p -f hosts.file
|
||||||
|
|
||||||
|
|
||||||
|
Caveats
|
||||||
|
-------
|
||||||
|
When operating in parallel, either by operating in parallel for each host
|
||||||
|
(-p/--parallel) or by parallel jobs within a host (-j/--jobs), and depending
|
||||||
|
on target SSH server and its configuration you may encounter connection drops.
|
||||||
|
This is controlled with sshd :strong:MaxStartups configuration options.
|
||||||
|
You may also encounter session open refusal. This happens with ssh multiplexing
|
||||||
|
when you reach maximum number of open sessions permitted per network
|
||||||
|
connection. In this case ssh will disable multiplexing.
|
||||||
|
This limit is controlled with sshd :strong:MaxSessions configuration
|
||||||
|
options. For more details refer to :strong:`sshd_config`\ (5).
|
||||||
|
|
||||||
|
For example, if you reach :strong:`MaxSessions` sessions you may get the
|
||||||
|
following output:
|
||||||
|
|
||||||
|
.. code-block:: sh
|
||||||
|
|
||||||
|
$ cdist config -b -j 11 -v 78.47.116.244
|
||||||
|
INFO: cdist: version 4.2.2-55-g640b7f9
|
||||||
|
INFO: 78.47.116.244: Running global explorers
|
||||||
|
INFO: 78.47.116.244: Remote transfer in 11 parallel jobs
|
||||||
|
channel 22: open failed: administratively prohibited: open failed
|
||||||
|
mux_client_request_session: session request failed: Session open refused by peer
|
||||||
|
ControlSocket /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/ssh-control-path already exists, disabling multiplexing
|
||||||
|
INFO: 78.47.116.244: Running global explorers in 11 parallel jobs
|
||||||
|
channel 22: open failed: administratively prohibited: open failed
|
||||||
|
mux_client_request_session: session request failed: Session open refused by peer
|
||||||
|
ControlSocket /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/ssh-control-path already exists, disabling multiplexing
|
||||||
|
INFO: 78.47.116.244: Running initial manifest /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/data/conf/manifest/init
|
||||||
|
INFO: 78.47.116.244: Running manifest and explorers for __file/root/host.file
|
||||||
|
INFO: 78.47.116.244: Generating code for __file/root/host.file
|
||||||
|
INFO: 78.47.116.244: Finished successful run in 18.655028820037842 seconds
|
||||||
|
INFO: cdist: Total processing time for 1 host(s): 19.159148693084717
|
|
@ -23,6 +23,7 @@ Contents:
|
||||||
cdist-types
|
cdist-types
|
||||||
cdist-explorer
|
cdist-explorer
|
||||||
cdist-messaging
|
cdist-messaging
|
||||||
|
cdist-parallelization
|
||||||
cdist-reference
|
cdist-reference
|
||||||
cdist-best-practice
|
cdist-best-practice
|
||||||
cdist-stages
|
cdist-stages
|
||||||
|
|
Loading…
Reference in a new issue