How to print messages from manifests? #110

New Issue

ungleich-gitea · 2021-11-20T13:24:04Z

ungleich-gitea commented

2021-11-20 13:24:04 +00:00

Created by: matthijskooijman

Something I couldn't figure out (might not be possible): How to print any messages for the user when running a manifest? I'm mostly thinking of the initial manifest here, but I guess the same applies to the type manifests.

Inside my manifest, I do some autodetection of needed config files based on the hostname, and I would like to show a warning when certain config files a re missing for this host. Another case is to show some instructions when a particular file was changed (e.g. when a similar change must be made elsewhere), but I guess this is more complicated, as it is not clear yet during the initial manifest run that a change will actually happen...

*Created by: matthijskooijman* Something I couldn't figure out (might not be possible): How to print any messages for the user when running a manifest? I'm mostly thinking of the initial manifest here, but I guess the same applies to the type manifests. Inside my manifest, I do some autodetection of needed config files based on the hostname, and I would like to show a warning when certain config files a re missing for this host. Another case is to show some instructions when a particular file was changed (e.g. when a similar change must be made elsewhere), but I guess this is more complicated, as it is not clear yet during the initial manifest run that a change will actually happen...

ungleich-gitea added the

Stale

label 2021-11-20 13:24:04 +00:00

ungleich-gitea closed this issue

2021-11-20 13:24:04 +00:00

ungleich-gitea commented

2021-11-20 15:39:55 +00:00

closed

ungleich-gitea commented

2021-11-20 15:39:57 +00:00

Created by: darko-poljak

@darko-poljak This would simplify all the dealing with logging vs output a lot. What say you?

Sounds interesting.

*Created by: darko-poljak* > @darko-poljak This would simplify all the dealing with logging vs output a lot. What say you? Sounds interesting.

ungleich-gitea commented

2021-11-20 15:39:58 +00:00

Created by: matthijskooijman

I was originally thinking to have a single pipe per process, so then you would not have any concurrency problems, but it doesn't scale too well. Using a single socket server makes a lot more sense indeed. I can imagine sockets are easier there (I was thinking about datagram sockets, which do not have any framing issues, but your example uses streaming sockets, but perhaps those have better concurrency guarantees as well).

Looking at your code, that looks good. I didn't realize that the logging package already has a unix socket implementation, all the more reason to use that :-)

I'm still thinking about somehow pre-opening a socket on e.g. fd 3 for subprcesses (i.e. so you do not have to create a socket on the filesystem somewhere and access to the socket is really limited to the related processes), but I couldn't figure out if that's possible (technically you need to create a socket, not bind it but do connect to it). I found socketpair() which could perhaps do this, but then you either need a separate socketpair per process again, or share a single fd among all processes (not sure if that voids the concurrency advantages).

Anyway, just some food for thought. Thanks for picking this up :-D

*Created by: matthijskooijman* I was originally thinking to have a single pipe per process, so then you would not have any concurrency problems, but it doesn't scale too well. Using a single socket server makes a lot more sense indeed. I can imagine sockets are easier there (I was thinking about datagram sockets, which do not have any framing issues, but your example uses streaming sockets, but perhaps those have better concurrency guarantees as well). Looking at your code, that looks good. I didn't realize that the logging package already has a unix socket implementation, all the more reason to use that :-) I'm still thinking about somehow pre-opening a socket on e.g. fd 3 for subprcesses (i.e. so you do not have to create a socket on the filesystem somewhere and access to the socket is really limited to the related processes), but I couldn't figure out if that's possible (technically you need to create a socket, not bind it but do connect to it). I found [`socketpair()`](http://man7.org/linux/man-pages/man2/socketpair.2.html) which could perhaps do this, but then you either need a separate socketpair per process again, or share a single fd among all processes (not sure if that voids the concurrency advantages). Anyway, just some food for thought. Thanks for picking this up :-D

ungleich-gitea commented

2021-11-20 15:39:59 +00:00

Created by: asteven

@darko-poljak This would simplify all the dealing with logging vs output a lot. What say you?

*Created by: asteven* @darko-poljak This would simplify all the dealing with logging vs output a lot. What say you?

ungleich-gitea commented

2021-11-20 15:40:01 +00:00

Created by: asteven

UNIX socket would be the obvious choice.
I tested using fd's and named pipes but it's much harder to deal with then a socket.
Remember that you have n processes writing there concurrently.

What I mean is something like this: https://gist.github.com/asteven/7e32461754cc417737d76da1adcd990a

This would then obviously be used by all parts of cdist that do not run in the main parent process.

*Created by: asteven* UNIX socket would be the obvious choice. I tested using fd's and named pipes but it's much harder to deal with then a socket. Remember that you have n processes writing there concurrently. What I mean is something like this: https://gist.github.com/asteven/7e32461754cc417737d76da1adcd990a This would then obviously be used by all parts of cdist that do not run in the main parent process.

ungleich-gitea commented

2021-11-20 15:40:02 +00:00

Created by: matthijskooijman

I was thinking more of the parent process running a log server on a socket.

You mean on a UNIX socket or TCP socket? These have the downside that they expose the logging access to other processes too.

With a solution like this all the logging is detached from stdout/stderr.

My suggestion also has that. stdout/stderr are on fd 1 and 2, and you would open up a pipe on fd 3 for the logging (by opening the pipe before forking off the subprocess, it is really only accessible to the subprocess).

*Created by: matthijskooijman* > I was thinking more of the parent process running a log server on a socket. You mean on a UNIX socket or TCP socket? These have the downside that they expose the logging access to other processes too. > With a solution like this all the logging is detached from stdout/stderr. My suggestion also has that. stdout/stderr are on fd 1 and 2, and you would open up a pipe on fd 3 for the logging (by opening the pipe before forking off the subprocess, it is really only accessible to the subprocess).

ungleich-gitea commented

2021-11-20 15:40:03 +00:00

Created by: asteven

I was thinking more of the parent process running a log server on a socket.
It shares the config for this via environment.
Client process picks this up and configures it's logging package to log to that logging server.

With a solution like this all the logging is detached from stdout/stderr.

*Created by: asteven* I was thinking more of the parent process running a log server on a socket. It shares the config for this via environment. Client process picks this up and configures it's logging package to log to that logging server. With a solution like this all the logging is detached from stdout/stderr.

ungleich-gitea commented

2021-11-20 15:40:05 +00:00

Created by: matthijskooijman

In terms of implementation, I think that would mean opening up an extra pipe to the process (on e.g. fd 3) and printing messages in some format (probably just lines with the level followed by a space and the message)?

*Created by: matthijskooijman* In terms of implementation, I think that would mean opening up an extra pipe to the process (on e.g. fd 3) and printing messages in some format (probably just lines with the level followed by a space and the message)?

ungleich-gitea commented

2021-11-20 15:40:06 +00:00

Created by: asteven

Thinking even more closely, Ideally I would be able to hook into the log level system from inside a manifest, but I can imagine that's not so trival to add..

I was thinking about this for some time also. I'd want to have something like this that I can use inside my manifest/gencode-* scripts:

cdist log INFO 'here goes my info message'

And this ends up in cdist's log output looking something like:

INFO $target_host: __my_type: here goes my info message

*Created by: asteven* > Thinking even more closely, Ideally I would be able to hook into the log level system from inside a manifest, but I can imagine that's not so trival to add.. I was thinking about this for some time also. I'd want to have something like this that I can use inside my manifest/gencode-* scripts: ``` cdist log INFO 'here goes my info message' ``` And this ends up in cdist's log output looking something like: ``` INFO $target_host: __my_type: here goes my info message ```

ungleich-gitea commented

2021-11-20 15:40:07 +00:00

Created by: matthijskooijman

Thinking even more closely, Ideally I would be able to hook into the log level system from inside a manifest, but I can imagine that's not so trival to add..

*Created by: matthijskooijman* Thinking even more closely, Ideally I would be able to hook into the log level system from inside a manifest, but I can imagine that's not so trival to add..

ungleich-gitea commented

2021-11-20 15:40:09 +00:00

Created by: matthijskooijman

Looking more closely at the --disable-saving-streams option, that makes all output from all manifests show up, not just the initial manifest. In my case, all of the type manifests seem to be silent, but if any are generating output, I'm not sure if I want to see that?

*Created by: matthijskooijman* Looking more closely at the `--disable-saving-streams` option, that makes *all* output from all manifests show up, not just the initial manifest. In my case, all of the type manifests seem to be silent, but if any are generating output, I'm not sure if I want to see that?

ungleich-gitea commented

2021-11-20 15:40:10 +00:00

Created by: matthijskooijman

Thanks for your swift reply! Unfortunately, I missed it back then, but found it now while I'm working with cdist again.

I tried your suggestion, but my output is not being shown. I tried with one to three -v options, with no luck. I also just updated to 8.10.3, which does not help.

Looking at the code, it seems that it redirects to files in /tmp by default, and passing --disable-saving-streams disables redirection, making output go directly to the terminal (however, that makes both stdout and stderr show up, not just stderr).

What's the recommended way to use this? By default, apparently output ends up in an anonymous folder in /tmp (which, on an unrelated note, does not seem to be cleaned up after a run?).

6895ad0207/cdist/core/manifest.py (L157-L171)

*Created by: matthijskooijman* Thanks for your swift reply! Unfortunately, I missed it back then, but found it now while I'm working with cdist again. I tried your suggestion, but my output is not being shown. I tried with one to three `-v` options, with no luck. I also just updated to 8.10.3, which does not help. Looking at the code, it seems that it redirects to files in /tmp by default, and passing `--disable-saving-streams` disables redirection, making output go directly to the terminal (however, that makes both stdout and stderr show up, not just stderr). What's the recommended way to use this? By default, apparently output ends up in an anonymous folder in /tmp (which, on an unrelated note, does not seem to be cleaned up after a run?). https://github.com/ungleich/cdist/blob/6895ad0207a58a5d22efd2943b872afbd1225204/cdist/core/manifest.py#L157-L171

ungleich-gitea commented

2021-11-20 15:40:11 +00:00

Created by: thriqon

Send it to standard error:

#!/bin/bash

__cdistmarker

echo "Hello, user!" >&2

__file /etc/config-file --state exists .....

You can also abort execution from inside the manifest, if neccessary, e.g. with exit 3.

*Created by: thriqon* Send it to standard error: ```` #!/bin/bash __cdistmarker echo "Hello, user!" >&2 __file /etc/config-file --state exists ..... ```` You can also abort execution from inside the manifest, if neccessary, e.g. with `exit 3`.

Sign in to join this conversation.