WIP: __directory to support --source path #376

Draft

smwltr wants to merge 1 commit from smwltr/cdist:directory_source into master

pull from: smwltr/cdist:directory_source

merge into: ungleich-public:master

ungleich-public:master

ungleich-public:7.0

ungleich-public:6.9

ungleich-public:py3.10

ungleich-public:ander/__package_apt_update_index

ungleich-public:haproxy-dualstack

ungleich-public:ander/__sed

ungleich-public:beta

ungleich-public:ander/os_version_debian_sid

ungleich-public:evilham-compatibility-fixes

ungleich-public:ander/__rsync

ungleich-public:__snakeoil_cert

ungleich-public:ander/update_readme

ungleich-public:__download_improvements

ungleich-public:feature/onchange

ungleich-public:cleanup/string-formatting

ungleich-public:feature/type-relationship-graph

ungleich-public:cleanup/ssh-auth-keys-types

ungleich-public:__letsencrypt_cert-fix-hooks

ungleich-public:bugfix/preos-debug

ungleich-public:bugfix/in-script-import

ungleich-public:6.8

ungleich-public:bugfix/sphinx-docs-build

ungleich-public:6.7

ungleich-public:cherry-pick-2f433a14

ungleich-public:bugfix/make-code-consistent

ungleich-public:6.6

ungleich-public:regain-py3.2-support

ungleich-public:6.5

ungleich-public:bugfix/multiple-log-lines

ungleich-public:matterbridge

ungleich-public:coturn

ungleich-public:alpinefix

ungleich-public:matrix

ungleich-public:new-type/network-interface

ungleich-public:feature/process

ungleich-public:6.4

ungleich-public:feature/info-command

ungleich-public:feature/libexec

ungleich-public:6.3

ungleich-public:preos-plugins-dir-opt

ungleich-public:gitlab-ci

ungleich-public:6.2

ungleich-public:order-dep-fix

ungleich-public:6.1

ungleich-public:6.0

ungleich-public:build/support-pip-from-git

ungleich-public:feature/shell-lib

ungleich-public:5.1

ungleich-public:feature/support-type-deprecation

ungleich-public:5.0

ungleich-public:feature/python-types

ungleich-public:4.11

ungleich-public:4.10

ungleich-public:shellcheck

ungleich-public:4.9

ungleich-public:4.8

ungleich-public:freebsd-improvements

ungleich-public:new-prometheus

ungleich-public:key_value-onchange

ungleich-public:feature/output_streams

ungleich-public:AnotherKamila-patch-1

ungleich-public:__letsencrypt_cert-fixes

ungleich-public:letsencrypt-cron-fix

ungleich-public:4.7

ungleich-public:newtype-__letsencrypt_cert

ungleich-public:os_explorer_devuan_fix

ungleich-public:prometheus-exporter-fixes

ungleich-public:daemontools-for-fbsd

ungleich-public:type-prometheus-exporter-from-master

ungleich-public:prometheus-more-fixes

ungleich-public:4.6

ungleich-public:4.5

ungleich-public:fix-j

ungleich-public:steven-backport

ungleich-public:4.4

ungleich-public:prometheus-fixes

ungleich-public:grafana_dashboard

ungleich-public:prometheus

ungleich-public:daemontools

ungleich-public:consul_improvements

ungleich-public:feature/trigger

ungleich-public:4.3

ungleich-public:4.0-pre-not-stable

ungleich-public:4.2

ungleich-public:4.1

ungleich-public:4.0

ungleich-public:feature_install_and_preos

ungleich-public:3.1

ungleich-public:no-dot-cdist

ungleich-public:random_dot_cdist

ungleich-public:feature_yum_url

ungleich-public:feature_files_export

ungleich-public:3.0

ungleich-public:2.3

ungleich-public:2.2

ungleich-public:2.1

ungleich-public:ssh_callback

ungleich-public:2.0

ungleich-public:archive_shell_function_approach

ungleich-public:1.7

ungleich-public:1.6

ungleich-public:1.5

ungleich-public:1.4

ungleich-public:1.3

ungleich-public:1.2

ungleich-public:1.1

ungleich-public:1.0

Conversation 5 Commits 1 Files changed 4 +74

smwltr commented 2024-05-07 15:13:57 +00:00

First-time contributor

Copy link

I've been using this code with an older version of cdist (3.1.13-237-g7135a3fd) for a few years now and thought it is generic enough to be of use to someone else.

If this is of any interest, I'll polish it up. Please fire away with the critiques.

I'd be interested in comments particularly to the FIXME lines.

I've been using this code with an older version of cdist (3.1.13-237-g7135a3fd) for a few years now and thought it is generic enough to be of use to someone else. If this is of any interest, I'll polish it up. Please fire away with the critiques. I'd be interested in comments particularly to the FIXME lines.

smwltr added 1 commit 2024-05-07 15:13:58 +00:00

__directory to support --source path a072dc9367

fnux commented 2024-05-13 06:01:57 +00:00

Collaborator

Copy link

There are a few edge cases (do we want to replace the remote content, to merge the local and remote directories, apply/conserve specific permissions, etc.) that will be hard to handle in this type. A separate type (e.g. __mirror_directory) with its own set of parameter might be more adapted.

There are a few edge cases (do we want to replace the remote content, to merge the local and remote directories, apply/conserve specific permissions, etc.) that will be hard to handle in this type. A separate type (e.g. `__mirror_directory`) with its own set of parameter might be more adapted.

fnux commented 2024-05-13 06:03:49 +00:00

Collaborator

Copy link

FIXME: What does cdist use for unit testing?

We have unit tests for cdist's python code, but no integration tests for the types.

FIXME: Output to stderr is no longer visible by default. Is there a correct
way to do this?

cdist will display stderr when the config runs exits with a non-zero code.

> FIXME: What does cdist use for unit testing? We have unit tests for cdist's python code, but no integration tests for the types. > FIXME: Output to stderr is no longer visible by default. Is there a correct > way to do this? cdist will display stderr when the config runs exits with a non-zero code.

smwltr commented 2024-05-19 09:57:40 +00:00

Author

First-time contributor

Copy link

FIXME: What does cdist use for unit testing?

We have unit tests for cdist's python code, but no integration tests for the types.

OK. I don't have anything set up for this yet, but if there is nothing for this yet, I may start on something that can be run locally and in CI so that you could also use it for helping check PRs.

FIXME: Output to stderr is no longer visible by default. Is there a correct
way to do this?

cdist will display stderr when the config runs exits with a non-zero code.

I suppose it's part of the design to have minimal output.

There are a few edge cases (do we want to replace the remote content, to merge the local and remote directories, apply/conserve specific permissions, etc.) that will be hard to handle in this type. A separate type (e.g. __mirror_directory) with its own set of parameter might be more adapted.

IIRC, I was trying to create an analog of the --source parameter in the __file type. What is the behaviour there? Does __file have options for merging content? Wouldn't it just overwrite the file? In that case, maybe I can add --delete to the $__remote_copy parameters. WDYT?

> > FIXME: What does cdist use for unit testing? > > We have unit tests for cdist's python code, but no integration tests for the types. OK. I don't have anything set up for this yet, but if there is nothing for this yet, I may start on something that can be run locally and in CI so that you **could** also use it for helping check PRs. > > > FIXME: Output to stderr is no longer visible by default. Is there a correct > > way to do this? > > cdist will display stderr when the config runs exits with a non-zero code. I suppose it's part of the design to have minimal output. > There are a few edge cases (do we want to replace the remote content, to merge the local and remote directories, apply/conserve specific permissions, etc.) that will be hard to handle in this type. A separate type (e.g. `__mirror_directory`) with its own set of parameter might be more adapted. IIRC, I was trying to create an analog of the `--source` parameter in the `__file` type. What is the behaviour there? Does `__file` have options for merging content? Wouldn't it just overwrite the file? In that case, maybe I can add `--delete` to the [`$__remote_copy` parameters](https://code.ungleich.ch/ungleich-public/cdist/src/commit/a072dc9367414ad799800d2d0d86d7c95505baf3/cdist/conf/type/__directory/gencode-local#L30). WDYT?

fnux commented 2024-05-21 12:42:50 +00:00

Collaborator

Copy link

OK. I don't have anything set up for this yet, but if there is nothing for this yet, I may start on something that can be run locally and in CI so that you could also use it for helping check PRs.

We could use gitea's new action/CI system, but this is a pretty big project.

IIRC, I was trying to create an analog of the --source parameter in the __file type. What is the behaviour there? Does __file have options for merging content? Wouldn't it just overwrite the file? In that case, maybe I can add --delete to the $__remote_copy parameters. WDYT?

I figured out you tried to make something similar to what we do with the __file type (note: there are no options for merge content with __file). I'd rather keep the __directory type as simple as possible and have this copy logic somewhere else (I'm open to comments from other users/maintainers, through).

Wouldn't it just overwrite the file? In that case, maybe I can add --delete to the $__remote_copy parameters. WDYT?

There are no guaranties that $__remote_copy supports --delete (e.g. scp).

> OK. I don't have anything set up for this yet, but if there is nothing for this yet, I may start on something that can be run locally and in CI so that you could also use it for helping check PRs. We could use gitea's new action/CI system, but this is a pretty big project. > IIRC, I was trying to create an analog of the --source parameter in the __file type. What is the behaviour there? Does __file have options for merging content? Wouldn't it just overwrite the file? In that case, maybe I can add --delete to the $__remote_copy parameters. WDYT? I figured out you tried to make something similar to what we do with the __file type (note: there are no options for merge content with __file). I'd rather keep the __directory type as simple as possible and have this copy logic somewhere else (I'm open to comments from other users/maintainers, through). > Wouldn't it just overwrite the file? In that case, maybe I can add --delete to the $__remote_copy parameters. WDYT? There are no guaranties that $__remote_copy supports --delete (e.g. scp).

smwltr commented 2024-05-29 13:53:15 +00:00

Author

First-time contributor

Copy link

There are no guaranties that $__remote_copy supports --delete (e.g. scp).

That is true. Oh well, if it's not wanted, we can close this.

I find it very useful to push a bunch of data to a server.
For example, a web server that has lots of customers websites. I use cdist to deploy everything.

There are other ways of doing that such as what I'd do before: compress the dir, use __file, and then decompress it in gencode-remote, which seems more "clunky".

> There are no guaranties that $__remote_copy supports --delete (e.g. scp). That is true. Oh well, if it's not wanted, we can close this. I find it very useful to push a bunch of data to a server. For example, a web server that has lots of customers websites. I use cdist to deploy everything. There are other ways of doing that such as what I'd do before: compress the dir, use __file, and then decompress it in gencode-remote, which seems more "clunky".

This pull request is marked as a work in progress.

View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u directory_source:smwltr-directory_source

git checkout smwltr-directory_source

Merge

Merge the changes and update on Forgejo.

git checkout master

git merge --no-ff smwltr-directory_source

git checkout master

git merge --ff-only smwltr-directory_source

git checkout smwltr-directory_source

git rebase master

git checkout master

git merge --no-ff smwltr-directory_source

git checkout master

git merge --squash smwltr-directory_source

git checkout master

git merge --ff-only smwltr-directory_source

git checkout master

git merge smwltr-directory_source

git push origin master

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bugfix

  

cleanup

  

discussion

  

documentation

  

doing

  

done

  

feature

  

improvement

  

packaging

  

Stale

  

testing

  

TODO

No labels

bugfix

cleanup

discussion

documentation

doing

done

feature

improvement

packaging

Stale

testing

TODO

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: ungleich-public/cdist#376

No description provided.