ungleich-public/cdist
ungleich-public/cdist
7
4
Fork 10
Code Issues 58 Pull requests 15 Projects Releases Wiki Activity

cdist -j doens't seem to default to cpu cores, and fails with sufficiently large number #325

New issue
Open
opened 2021-12-21 13:29:36 +00:00 by allan_wind · 1 comment
allan_wind commented 2021-12-21 13:29:36 +00:00
Copy link

https://www.cdi.st/manual/6.9.4/man1/cdist.html says:

-j [JOBS], --jobs [JOBS]
Operate in parallel in specified maximum number of jobs. Global explorers, object prepare and object run are supported. Without argument CPU count is used by default.

however, when you run it without a -j it defaults to sequential mode:

$ cdist config -vvv vent|egrep 'mode|seconds' | uniq
DEBUG: vent: Iteration in sequential mode
INFO: vent: Finished successful run in 5.47 seconds

which appears similar to -j1:

$ cdist config -j1 -vvv vent|egrep 'mode|seconds'| uniq
DEBUG: [862030]: vent: Iteration in parallel mode in 1 jobs
INFO: [862030]: vent: Finished successful run in 5.14 seconds

With sufficient large -j in my case around ~12 or ~13 the global explorers fail in no particular pattern:

$ cdist config -j12 -vvv vent|egrep 'mode|seconds'| uniq
Error processing global explorer 'hostname'
===========================================
name: hostname
path: /tmp/user/1000/tmpn1t453bf/39070aff4100488e27d58a587b526fc4/data/conf/explorer/hostname

$ cdist config -j12 -vvv vent|egrep 'mode|seconds'| uniq
ERROR: [863719]: vent: /usr/share/doc/cdist/examples/remote/sudo/exec vent /bin/sh -c ' export __target_host=vent;  export __target_hostname=vent.lifeintegrity.com;  export __target_fqdn=vent.lifeintegrity.com;  export __explorer=/var/lib/cdist/conf/explorer;  export __target_host_tags=desktop;  export __cdist_log_level=10;  export __cdist_log_level_name=DEBUG;  export PATH=/bin:/usr/bin:/sbin:/usr/sbin;/bin/sh -e /var/lib/cdist/conf/explorer/os_version': ['/usr/share/doc/cdist/examples/remote/sudo/exec', 'vent', "/bin/sh -c ' export __target_host=vent;  export __target_hostname=vent.lifeintegrity.com;  export __target_fqdn=vent.lifeintegrity.com;  export __explorer=/var/lib/cdist/conf/explorer;  export __target_host_tags=desktop;  export __cdist_log_level=10;  export __cdist_log_level_name=DEBUG;  export PATH=/bin:/usr/bin:/sbin:/usr/sbin;/bin/sh -e /var/lib/cdist/conf/explorer/os_version'"]

Error processing global explorer 'os_version'
=============================================
name: os_version
path: /tmp/user/1000/tmp63bkjf1n/39070aff4100488e27d58a587b526fc4/data/conf/explorer/os_version
https://www.cdi.st/manual/6.9.4/man1/cdist.html says: > -j [JOBS], --jobs [JOBS] > Operate in parallel in specified maximum number of jobs. Global explorers, object prepare and object run are supported. Without argument CPU count is used by default. however, when you run it without a -j it defaults to sequential mode: ``` $ cdist config -vvv vent|egrep 'mode|seconds' | uniq DEBUG: vent: Iteration in sequential mode INFO: vent: Finished successful run in 5.47 seconds ``` which appears similar to -j1: ``` $ cdist config -j1 -vvv vent|egrep 'mode|seconds'| uniq DEBUG: [862030]: vent: Iteration in parallel mode in 1 jobs INFO: [862030]: vent: Finished successful run in 5.14 seconds ``` With sufficient large -j in my case around ~12 or ~13 the global explorers fail in no particular pattern: ``` $ cdist config -j12 -vvv vent|egrep 'mode|seconds'| uniq Error processing global explorer 'hostname' =========================================== name: hostname path: /tmp/user/1000/tmpn1t453bf/39070aff4100488e27d58a587b526fc4/data/conf/explorer/hostname ``` ``` $ cdist config -j12 -vvv vent|egrep 'mode|seconds'| uniq ERROR: [863719]: vent: /usr/share/doc/cdist/examples/remote/sudo/exec vent /bin/sh -c ' export __target_host=vent; export __target_hostname=vent.lifeintegrity.com; export __target_fqdn=vent.lifeintegrity.com; export __explorer=/var/lib/cdist/conf/explorer; export __target_host_tags=desktop; export __cdist_log_level=10; export __cdist_log_level_name=DEBUG; export PATH=/bin:/usr/bin:/sbin:/usr/sbin;/bin/sh -e /var/lib/cdist/conf/explorer/os_version': ['/usr/share/doc/cdist/examples/remote/sudo/exec', 'vent', "/bin/sh -c ' export __target_host=vent; export __target_hostname=vent.lifeintegrity.com; export __target_fqdn=vent.lifeintegrity.com; export __explorer=/var/lib/cdist/conf/explorer; export __target_host_tags=desktop; export __cdist_log_level=10; export __cdist_log_level_name=DEBUG; export PATH=/bin:/usr/bin:/sbin:/usr/sbin;/bin/sh -e /var/lib/cdist/conf/explorer/os_version'"] Error processing global explorer 'os_version' ============================================= name: os_version path: /tmp/user/1000/tmp63bkjf1n/39070aff4100488e27d58a587b526fc4/data/conf/explorer/os_version ```
allan_wind commented 2021-12-21 13:37:04 +00:00
Author
Copy link

The errors appears to be due to "sshd[630]: error: beginning MaxStartups throttling". They go away if I bump up MaxStartups 16:30:100. It would good if the error message could indicate that it's an ssh issue and not something with a particular explorer.

The errors appears to be due to "sshd[630]: error: beginning MaxStartups throttling". They go away if I bump up MaxStartups 16:30:100. It would good if the error message could indicate that it's an ssh issue and not something with a particular explorer.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
7.0
6.9
py3.10
ander/__package_apt_update_index
haproxy-dualstack
ander/__sed
beta
ander/os_version_debian_sid
evilham-compatibility-fixes
ander/__rsync
__snakeoil_cert
ander/update_readme
__download_improvements
feature/onchange
cleanup/string-formatting
feature/type-relationship-graph
cleanup/ssh-auth-keys-types
__letsencrypt_cert-fix-hooks
bugfix/preos-debug
bugfix/in-script-import
6.8
bugfix/sphinx-docs-build
6.7
cherry-pick-2f433a14
bugfix/make-code-consistent
6.6
regain-py3.2-support
6.5
bugfix/multiple-log-lines
matterbridge
coturn
alpinefix
matrix
new-type/network-interface
feature/process
6.4
feature/info-command
feature/libexec
6.3
preos-plugins-dir-opt
gitlab-ci
6.2
order-dep-fix
6.1
6.0
build/support-pip-from-git
feature/shell-lib
5.1
feature/support-type-deprecation
5.0
feature/python-types
4.11
4.10
shellcheck
4.9
4.8
freebsd-improvements
new-prometheus
key_value-onchange
feature/output_streams
AnotherKamila-patch-1
__letsencrypt_cert-fixes
letsencrypt-cron-fix
4.7
newtype-__letsencrypt_cert
os_explorer_devuan_fix
prometheus-exporter-fixes
daemontools-for-fbsd
type-prometheus-exporter-from-master
prometheus-more-fixes
4.6
4.5
fix-j
steven-backport
4.4
prometheus-fixes
grafana_dashboard
prometheus
daemontools
consul_improvements
feature/trigger
4.3
4.0-pre-not-stable
4.2
4.1
4.0
feature_install_and_preos
3.1
no-dot-cdist
random_dot_cdist
feature_yum_url
feature_files_export
3.0
2.3
2.2
2.1
ssh_callback
2.0
archive_shell_function_approach
1.7
1.6
1.5
1.4
1.3
1.2
1.1
1.0
7.0.0
6.9.8
6.9.7
6.9.6
6.9.5
6.9.4
6.9.3
6.9.2
6.9.1
6.9.0
6.8.0
6.7.0
6.6.0
6.5.6
6.5.5
6.5.4
6.5.3
6.5.2
6.5.1
6.5.0
6.4.0
6.3.0
6.2.0
6.1.1
6.1.0
6.0.4
6.0.3
6.0.2
6.0.1
6.0.0
5.1.3
5.1.2
5.1.1
5.1.0
5.0.2
5.0.1
5.0.0
4.11.1
4.11.0
4.10.11
4.10.10
4.10.9
4.10.8
4.10.7
4.10.6
4.10.5
4.10.4
4.10.3
4.10.2
4.10.1
4.10.0
4.9.1
4.9.0
4.8.4
4.8.3
4.8.2
4.8.1
4.8.0
4.7.3
4.7.2
4.7.1
4.7.0
4.6.1
4.6.0
4.5.0
4.4.4
4.4.3
4.4.2
4.4.1
4.4.0
4.3.2
4.3.1
4.3.0
4.2.2
4.2.1
4.2.0
4.1.0
4.0.0
3.1.13
tn1
3.1.12
3.1.11
3.1.10
3.1.9
3.1.8
3.1.7
3.1.6
4.0.0pre3
3.1.5
3.1.4
3.1.3
3.1.2
3.1.1
3.1.0
4.0.0pre2
3.0.9
3.0.8
3.0.7
3.0.6
3.0.5
3.0.4
3.0.3
4.0.0pre1
3.0.2
3.0.1
3.0.0
2.3.7
2.3.6
2.3.5
2.3.4
2.3.3
2.3.2
2.3.1
2.3.0
2.2.0
2.1.2
2.1.1
2.1.0
2.1.0pre8
2.1.0pre7
2.1.0pre6
2.0.15
2.1.0pre5
2.1.0pre4
2.1.0pre3
2.1.0pre2
2.1.0pre1
2.0.14
2.0.13
2.0.12
2.0.11
2.0.10
2.0.9
2.0.8
2.0.7
2.0.6
2.0.5
2.0.4
2.0.3
2.0.2
2.0.1
2.0.0
1.7.1
1.7.0
1.6.2
1.6.1
1.6.0
1.5.0
1.4.1
1.4.0
1.3.2
1.3.1
1.3.0
1.2.0
1.1.0
1.0.4
1.0.3
1.0.2
1.0.1
1.0.0
0.9.9
0.9.8
0.9.7
0.9.6
0.9.5
0.9.4
0.9.3
0.9.2
0.9.1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Labels
Clear labels   
bugfix

   
cleanup

   
discussion

   
documentation

   
doing

   
done

   
feature

   
improvement

   
packaging

   
Stale

   
testing

   
TODO
No labels
bugfix
cleanup
discussion
documentation
doing
done
feature
improvement
packaging
Stale
testing
TODO
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: ungleich-public/cdist#325
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?