WIP: __directory to support --source path #376

Draft
smwltr wants to merge 1 commits from smwltr/cdist:directory_source into master
First-time contributor

I've been using this code with an older version of cdist (3.1.13-237-g7135a3fd) for a few years now and thought it is generic enough to be of use to someone else.

If this is of any interest, I'll polish it up. Please fire away with the critiques.

I'd be interested in comments particularly to the FIXME lines.

I've been using this code with an older version of cdist (3.1.13-237-g7135a3fd) for a few years now and thought it is generic enough to be of use to someone else. If this is of any interest, I'll polish it up. Please fire away with the critiques. I'd be interested in comments particularly to the FIXME lines.
smwltr added 1 commit 2024-05-07 15:13:58 +00:00
Collaborator

There are a few edge cases (do we want to replace the remote content, to merge the local and remote directories, apply/conserve specific permissions, etc.) that will be hard to handle in this type. A separate type (e.g. __mirror_directory) with its own set of parameter might be more adapted.

There are a few edge cases (do we want to replace the remote content, to merge the local and remote directories, apply/conserve specific permissions, etc.) that will be hard to handle in this type. A separate type (e.g. `__mirror_directory`) with its own set of parameter might be more adapted.
Collaborator

FIXME: What does cdist use for unit testing?

We have unit tests for cdist's python code, but no integration tests for the types.

FIXME: Output to stderr is no longer visible by default. Is there a correct
way to do this?

cdist will display stderr when the config runs exits with a non-zero code.

> FIXME: What does cdist use for unit testing? We have unit tests for cdist's python code, but no integration tests for the types. > FIXME: Output to stderr is no longer visible by default. Is there a correct > way to do this? cdist will display stderr when the config runs exits with a non-zero code.
Author
First-time contributor

FIXME: What does cdist use for unit testing?

We have unit tests for cdist's python code, but no integration tests for the types.

OK. I don't have anything set up for this yet, but if there is nothing for this yet, I may start on something that can be run locally and in CI so that you could also use it for helping check PRs.

FIXME: Output to stderr is no longer visible by default. Is there a correct
way to do this?

cdist will display stderr when the config runs exits with a non-zero code.

I suppose it's part of the design to have minimal output.

There are a few edge cases (do we want to replace the remote content, to merge the local and remote directories, apply/conserve specific permissions, etc.) that will be hard to handle in this type. A separate type (e.g. __mirror_directory) with its own set of parameter might be more adapted.

IIRC, I was trying to create an analog of the --source parameter in the __file type. What is the behaviour there? Does __file have options for merging content? Wouldn't it just overwrite the file? In that case, maybe I can add --delete to the $__remote_copy parameters. WDYT?

> > FIXME: What does cdist use for unit testing? > > We have unit tests for cdist's python code, but no integration tests for the types. OK. I don't have anything set up for this yet, but if there is nothing for this yet, I may start on something that can be run locally and in CI so that you **could** also use it for helping check PRs. > > > FIXME: Output to stderr is no longer visible by default. Is there a correct > > way to do this? > > cdist will display stderr when the config runs exits with a non-zero code. I suppose it's part of the design to have minimal output. > There are a few edge cases (do we want to replace the remote content, to merge the local and remote directories, apply/conserve specific permissions, etc.) that will be hard to handle in this type. A separate type (e.g. `__mirror_directory`) with its own set of parameter might be more adapted. IIRC, I was trying to create an analog of the `--source` parameter in the `__file` type. What is the behaviour there? Does `__file` have options for merging content? Wouldn't it just overwrite the file? In that case, maybe I can add `--delete` to the [`$__remote_copy` parameters](https://code.ungleich.ch/ungleich-public/cdist/src/commit/a072dc9367414ad799800d2d0d86d7c95505baf3/cdist/conf/type/__directory/gencode-local#L30). WDYT?
Collaborator

OK. I don't have anything set up for this yet, but if there is nothing for this yet, I may start on something that can be run locally and in CI so that you could also use it for helping check PRs.

We could use gitea's new action/CI system, but this is a pretty big project.

IIRC, I was trying to create an analog of the --source parameter in the __file type. What is the behaviour there? Does __file have options for merging content? Wouldn't it just overwrite the file? In that case, maybe I can add --delete to the $__remote_copy parameters. WDYT?

I figured out you tried to make something similar to what we do with the __file type (note: there are no options for merge content with __file). I'd rather keep the __directory type as simple as possible and have this copy logic somewhere else (I'm open to comments from other users/maintainers, through).

Wouldn't it just overwrite the file? In that case, maybe I can add --delete to the $__remote_copy parameters. WDYT?

There are no guaranties that $__remote_copy supports --delete (e.g. scp).

> OK. I don't have anything set up for this yet, but if there is nothing for this yet, I may start on something that can be run locally and in CI so that you could also use it for helping check PRs. We could use gitea's new action/CI system, but this is a pretty big project. > IIRC, I was trying to create an analog of the --source parameter in the __file type. What is the behaviour there? Does __file have options for merging content? Wouldn't it just overwrite the file? In that case, maybe I can add --delete to the $__remote_copy parameters. WDYT? I figured out you tried to make something similar to what we do with the __file type (note: there are no options for merge content with __file). I'd rather keep the __directory type as simple as possible and have this copy logic somewhere else (I'm open to comments from other users/maintainers, through). > Wouldn't it just overwrite the file? In that case, maybe I can add --delete to the $__remote_copy parameters. WDYT? There are no guaranties that $__remote_copy supports --delete (e.g. scp).
Author
First-time contributor

There are no guaranties that $__remote_copy supports --delete (e.g. scp).

That is true. Oh well, if it's not wanted, we can close this.

I find it very useful to push a bunch of data to a server.
For example, a web server that has lots of customers websites. I use cdist to deploy everything.

There are other ways of doing that such as what I'd do before: compress the dir, use __file, and then decompress it in gencode-remote, which seems more "clunky".

> There are no guaranties that $__remote_copy supports --delete (e.g. scp). That is true. Oh well, if it's not wanted, we can close this. I find it very useful to push a bunch of data to a server. For example, a web server that has lots of customers websites. I use cdist to deploy everything. There are other ways of doing that such as what I'd do before: compress the dir, use __file, and then decompress it in gencode-remote, which seems more "clunky".
This pull request is marked as a work in progress.
You can also view command line instructions.

Step 1:

From your project repository, check out a new branch and test the changes.
git checkout -b smwltr-directory_source master
git pull directory_source

Step 2:

Merge the changes and update on Gitea.
git checkout master
git merge --no-ff smwltr-directory_source
git push origin master
Sign in to join this conversation.
No reviewers
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: ungleich-public/cdist#376
No description provided.