ungleich-public/cdist
ungleich-public/cdist
7
4
Fork 10
Code Issues 59 Pull requests 15 Projects Releases Wiki Activity

[DISCUSSION] nonparallel vs nonparallel-self #10

New issue
Open
opened 2021-11-20 11:24:46 +00:00 by ungleich-gitea · 4 comments
ungleich-gitea commented 2021-11-20 11:24:46 +00:00
Owner
Copy link

Conceptually there are many types that shouldn't be run in parallel with antyhing at all. Something that changes the system's global state.

On the other hand, some (most?) types actually only need to not be run in parallel with another instance of themselves, those that are accessing one well defined resource (or set of resources).

I am unsure which one the current nonparallel implementation meant to address, and whether it would make sense to address the other one.

I'd be delighted to know your thoughts...

Conceptually there are many types that shouldn't be run in parallel with _antyhing_ at all. Something that changes the system's global state. On the other hand, some (most?) types actually only need to not be run in parallel with another instance of themselves, those that are accessing one well defined resource (or set of resources). I am unsure which one the current `nonparallel` implementation meant to address, and whether it would make sense to address the other one. I'd be delighted to know your thoughts...
ungleich-gitea commented 2021-11-20 15:25:14 +00:00
Author
Owner
Copy link

I think you're pokinging a wasps nest, here.

The current nonparallel file is meant to be used with types that modify parts of one resource, e.g.

  • __package_*: modifies the package database,
  • __uci: modified the UCI database,
  • __sshd_config: modifies the "global" sshd_config,
  • __debconf_set_selections: modifies the debconf database; is currently parallel, but I'm working on a patch.)

But there are many different cases where this does not apply, e.g.:

  • __cron: modifies crontab, but interlocking is only required if multiple objects touch the same user.
  • __key_value/__line: interlocking only required if multiple objects touch the same file.
  • __dpkg_architecture: is nonparallel, but maybe it should also interlock with e.g. __package_apt or __apt_*?
  • __ssh_authorized_key: only requires interlocking if multiple objects touch the same file.

But IMHO nonparallel is actually even more complex.
There are two issues I see that are non-trivial to fix.

  • There are many utilities that want to acquire a lock when they are started but terminate if the lock cannot be acquired, instead of waiting for it to be become available.
    Even if all cdist objects would be locked by a nonparallel-theworld cdist runs can still fail if e.g. an auto updater cron job launches and starts to upgrade packages at the same time.
  • Interlocking does not only apply to code-remote (as nonparallel does) but it can also apply to explorers.

EDIT: @fancsali can you make some examples of objects that should not be run with anything else in parallel?

I think you're pokinging a wasps nest, here. The current `nonparallel` file is meant to be used with types that modify parts of one resource, e.g. - `__package_*`: modifies the package database, - `__uci`: modified the UCI database, - `__sshd_config`: modifies the "global" sshd_config, - `__debconf_set_selections`: modifies the debconf database; is currently parallel, but I'm working on a patch.) But there are many different cases where this does not apply, e.g.: - `__cron`: modifies crontab, but interlocking is only required if multiple objects touch the same user. - `__key_value`/`__line`: interlocking only required if multiple objects touch the same file. - `__dpkg_architecture`: is `nonparallel`, but maybe it should also interlock with e.g. `__package_apt` or `__apt_*`? - `__ssh_authorized_key`: only requires interlocking if multiple objects touch the same file. But IMHO `nonparallel` is actually even more complex. There are two issues I see that are non-trivial to fix. * There are many utilities that want to acquire a lock when they are started but terminate if the lock cannot be acquired, instead of waiting for it to be become available. Even if all cdist objects would be locked by a `nonparallel-theworld` cdist runs can still fail if e.g. an auto updater cron job launches and starts to upgrade packages at the same time. * Interlocking does not only apply to `code-remote` (as `nonparallel` does) but it _can_ also apply to explorers. EDIT: @fancsali can you make some examples of objects that should not be run with anything else in parallel?
ungleich-gitea commented 2021-11-20 15:25:16 +00:00
Author
Owner
Copy link

changed the description

changed the description
ungleich-gitea commented 2021-11-20 15:25:17 +00:00
Author
Owner
Copy link

changed the description

changed the description
ungleich-gitea commented 2021-11-20 15:25:18 +00:00
Author
Owner
Copy link

changed title from {-{-} to {+[DISCUSSION] nonparallel vs nonparallel-self+}

changed title from **{-{-}** to **{+[DISCUSSION] nonparallel vs nonparallel-self+}**
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
7.0
6.9
py3.10
ander/__package_apt_update_index
haproxy-dualstack
ander/__sed
beta
ander/os_version_debian_sid
evilham-compatibility-fixes
ander/__rsync
__snakeoil_cert
ander/update_readme
__download_improvements
feature/onchange
cleanup/string-formatting
feature/type-relationship-graph
cleanup/ssh-auth-keys-types
__letsencrypt_cert-fix-hooks
bugfix/preos-debug
bugfix/in-script-import
6.8
bugfix/sphinx-docs-build
6.7
cherry-pick-2f433a14
bugfix/make-code-consistent
6.6
regain-py3.2-support
6.5
bugfix/multiple-log-lines
matterbridge
coturn
alpinefix
matrix
new-type/network-interface
feature/process
6.4
feature/info-command
feature/libexec
6.3
preos-plugins-dir-opt
gitlab-ci
6.2
order-dep-fix
6.1
6.0
build/support-pip-from-git
feature/shell-lib
5.1
feature/support-type-deprecation
5.0
feature/python-types
4.11
4.10
shellcheck
4.9
4.8
freebsd-improvements
new-prometheus
key_value-onchange
feature/output_streams
AnotherKamila-patch-1
__letsencrypt_cert-fixes
letsencrypt-cron-fix
4.7
newtype-__letsencrypt_cert
os_explorer_devuan_fix
prometheus-exporter-fixes
daemontools-for-fbsd
type-prometheus-exporter-from-master
prometheus-more-fixes
4.6
4.5
fix-j
steven-backport
4.4
prometheus-fixes
grafana_dashboard
prometheus
daemontools
consul_improvements
feature/trigger
4.3
4.0-pre-not-stable
4.2
4.1
4.0
feature_install_and_preos
3.1
no-dot-cdist
random_dot_cdist
feature_yum_url
feature_files_export
3.0
2.3
2.2
2.1
ssh_callback
2.0
archive_shell_function_approach
1.7
1.6
1.5
1.4
1.3
1.2
1.1
1.0
7.0.0
6.9.8
6.9.7
6.9.6
6.9.5
6.9.4
6.9.3
6.9.2
6.9.1
6.9.0
6.8.0
6.7.0
6.6.0
6.5.6
6.5.5
6.5.4
6.5.3
6.5.2
6.5.1
6.5.0
6.4.0
6.3.0
6.2.0
6.1.1
6.1.0
6.0.4
6.0.3
6.0.2
6.0.1
6.0.0
5.1.3
5.1.2
5.1.1
5.1.0
5.0.2
5.0.1
5.0.0
4.11.1
4.11.0
4.10.11
4.10.10
4.10.9
4.10.8
4.10.7
4.10.6
4.10.5
4.10.4
4.10.3
4.10.2
4.10.1
4.10.0
4.9.1
4.9.0
4.8.4
4.8.3
4.8.2
4.8.1
4.8.0
4.7.3
4.7.2
4.7.1
4.7.0
4.6.1
4.6.0
4.5.0
4.4.4
4.4.3
4.4.2
4.4.1
4.4.0
4.3.2
4.3.1
4.3.0
4.2.2
4.2.1
4.2.0
4.1.0
4.0.0
3.1.13
tn1
3.1.12
3.1.11
3.1.10
3.1.9
3.1.8
3.1.7
3.1.6
4.0.0pre3
3.1.5
3.1.4
3.1.3
3.1.2
3.1.1
3.1.0
4.0.0pre2
3.0.9
3.0.8
3.0.7
3.0.6
3.0.5
3.0.4
3.0.3
4.0.0pre1
3.0.2
3.0.1
3.0.0
2.3.7
2.3.6
2.3.5
2.3.4
2.3.3
2.3.2
2.3.1
2.3.0
2.2.0
2.1.2
2.1.1
2.1.0
2.1.0pre8
2.1.0pre7
2.1.0pre6
2.0.15
2.1.0pre5
2.1.0pre4
2.1.0pre3
2.1.0pre2
2.1.0pre1
2.0.14
2.0.13
2.0.12
2.0.11
2.0.10
2.0.9
2.0.8
2.0.7
2.0.6
2.0.5
2.0.4
2.0.3
2.0.2
2.0.1
2.0.0
1.7.1
1.7.0
1.6.2
1.6.1
1.6.0
1.5.0
1.4.1
1.4.0
1.3.2
1.3.1
1.3.0
1.2.0
1.1.0
1.0.4
1.0.3
1.0.2
1.0.1
1.0.0
0.9.9
0.9.8
0.9.7
0.9.6
0.9.5
0.9.4
0.9.3
0.9.2
0.9.1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Labels
Clear labels   
bugfix

   
cleanup

   
discussion

   
documentation

   
doing

   
done

   
feature

   
improvement

   
packaging

   
Stale

   
testing

   
TODO
No labels
bugfix
cleanup
discussion
documentation
doing
done
feature
improvement
packaging
Stale
testing
TODO
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
ungleich-public/cdist#10
No description provided.