2016-08-17 19:32:16 +02:00
|
|
|
Parallelization
|
|
|
|
|
===============
|
|
|
|
|
|
|
|
|
|
Description
|
|
|
|
|
-----------
|
|
|
|
|
cdist has two modes of parallel operation.
|
|
|
|
|
|
|
|
|
|
One of them is to operate on each host in separate process. This is enabled
|
|
|
|
|
with :strong:`-p/--parallel` option.
|
|
|
|
|
|
|
|
|
|
The other way is to operate in parallel within one host where you specify
|
|
|
|
|
the number of jobs. This is enabled with :strong:`-j/--jobs` option where you
|
|
|
|
|
can specify the number of parallel jobs. By default,
|
2017-06-13 22:54:40 +02:00
|
|
|
:strong:`multiprocessing.cpu_count()` is used. For this mode global explorers,
|
|
|
|
|
object preparation and object run are supported and this option is still in
|
|
|
|
|
:strong:`beta`.
|
2016-08-17 19:32:16 +02:00
|
|
|
|
|
|
|
|
You can, of course, use those two options together. This means that each host
|
|
|
|
|
will be processed by its own process. Within each process cdist will operate
|
|
|
|
|
using specified number of parallel jobs.
|
|
|
|
|
|
|
|
|
|
For more info on those options see :strong:`cdist`\ (1).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Examples
|
|
|
|
|
--------
|
|
|
|
|
|
|
|
|
|
.. code-block:: sh
|
|
|
|
|
|
|
|
|
|
# Configure hosts read from file hosts.file in parallel
|
|
|
|
|
$ cdist config -p -f hosts.file
|
|
|
|
|
|
|
|
|
|
# Configure hosts read from file hosts.file sequentially but using default
|
|
|
|
|
# number of parallel jobs
|
|
|
|
|
$ cdist config -b -j -f hosts.file
|
|
|
|
|
|
|
|
|
|
# Configure hosts read from file hosts.file in parallel using 16
|
|
|
|
|
# parallel jobs
|
|
|
|
|
$ cdist config -b -j 16 -p -f hosts.file
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Caveats
|
|
|
|
|
-------
|
|
|
|
|
When operating in parallel, either by operating in parallel for each host
|
|
|
|
|
(-p/--parallel) or by parallel jobs within a host (-j/--jobs), and depending
|
|
|
|
|
on target SSH server and its configuration you may encounter connection drops.
|
|
|
|
|
This is controlled with sshd :strong:MaxStartups configuration options.
|
|
|
|
|
You may also encounter session open refusal. This happens with ssh multiplexing
|
|
|
|
|
when you reach maximum number of open sessions permitted per network
|
|
|
|
|
connection. In this case ssh will disable multiplexing.
|
|
|
|
|
This limit is controlled with sshd :strong:MaxSessions configuration
|
|
|
|
|
options. For more details refer to :strong:`sshd_config`\ (5).
|
|
|
|
|
|
|
|
|
|
For example, if you reach :strong:`MaxSessions` sessions you may get the
|
|
|
|
|
following output:
|
|
|
|
|
|
|
|
|
|
.. code-block:: sh
|
|
|
|
|
|
|
|
|
|
$ cdist config -b -j 11 -v 78.47.116.244
|
|
|
|
|
INFO: cdist: version 4.2.2-55-g640b7f9
|
|
|
|
|
INFO: 78.47.116.244: Running global explorers
|
|
|
|
|
INFO: 78.47.116.244: Remote transfer in 11 parallel jobs
|
|
|
|
|
channel 22: open failed: administratively prohibited: open failed
|
|
|
|
|
mux_client_request_session: session request failed: Session open refused by peer
|
|
|
|
|
ControlSocket /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/ssh-control-path already exists, disabling multiplexing
|
|
|
|
|
INFO: 78.47.116.244: Running global explorers in 11 parallel jobs
|
|
|
|
|
channel 22: open failed: administratively prohibited: open failed
|
|
|
|
|
mux_client_request_session: session request failed: Session open refused by peer
|
|
|
|
|
ControlSocket /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/ssh-control-path already exists, disabling multiplexing
|
|
|
|
|
INFO: 78.47.116.244: Running initial manifest /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/data/conf/manifest/init
|
|
|
|
|
INFO: 78.47.116.244: Running manifest and explorers for __file/root/host.file
|
|
|
|
|
INFO: 78.47.116.244: Generating code for __file/root/host.file
|
|
|
|
|
INFO: 78.47.116.244: Finished successful run in 18.655028820037842 seconds
|
|
|
|
|
INFO: cdist: Total processing time for 1 host(s): 19.159148693084717
|
