Commit Graph

2292 Commits

Author SHA1 Message Date
fnux 13e2ad175f
[scanner] add host class, name mapper and pre-config logic 2021-04-25 12:45:34 +02:00
fnux bb24d632d6
[scanner] implement the --list flag 2021-04-22 10:20:49 +02:00
fnux a4464209b6
[scanner] add minimal error handling, consolidate CLI args processing 2021-04-22 09:31:06 +02:00
fnux acf9bf91f1
[scanner] error to stderr and exit when scapy is not available 2021-04-22 08:55:14 +02:00
Dennis Camera 1c047353a9 [bin/cdist] Fix Python version check 2021-04-17 09:57:10 +02:00
Dennis Camera 19bf37be1a [type/__postgres_conf] Update man.rst 2021-04-15 15:56:15 +02:00
Dennis Camera 686e4f0f2d [type/__postgres_conf] Reverse state logic (decide based on source first) 2021-04-15 15:50:03 +02:00
Dennis Camera bef1433ba3 [type/__postgres_conf] Accept empty values 2021-04-15 15:50:03 +02:00
Dennis Camera 12c2995494 [type/__postgres_conf] Implement complex state compare logic 2021-04-15 15:50:02 +02:00
Dennis Camera e0416403c4 [type/__postgres_conf] Add psql_conf_source function to state explorer 2021-04-15 15:50:02 +02:00
Dennis Camera 2ccc03fef1 [type/__postgres_conf] Add psql_conf_cmp function to state explorer 2021-04-15 15:50:02 +02:00
Dennis Camera 92b8942a8c [type/__postgres_conf] Add psql_exec function to state explorer 2021-04-15 15:50:02 +02:00
ander d2eec60668
[__download] make --sum optional 2021-04-11 23:16:00 +03:00
Darko Poljak 750c71fb5a Minor refactoring and remove code duplication 2021-04-07 10:25:26 +02:00
Darko Poljak 199effb7ef Improve unfinished object requirements bool check
When we need only boolean value for unfinished object requirements then
we don't need to determine the whole list of unfinished objects.
2021-04-06 19:35:14 +02:00
poljakowski ce79a2069c Merge branch 'fix/type/__pyvenv/numeric-owner' into 'master'
__pyvenv: Fix if --owner / --group is numeric

See merge request ungleich-public/cdist!988
2021-04-01 15:36:02 +02:00
poljakowski c981f654f1 Merge branch 'fix/type/__git/numeric-owner' into 'master'
__git: Fix if --owner / --group is numeric

See merge request ungleich-public/cdist!987
2021-04-01 15:35:46 +02:00
Darko Poljak 4c2d273f07 Unify string formatting
Use one way of string formatting: replace old `%` style with new `str.format`.

Resolve #855.
2021-03-31 08:19:34 +02:00
Darko Poljak f984a918b9 Fix log message string formatting
Use logging message format with args, instead of direct `%` or `str.format`.

Resolve #855.
2021-03-31 08:19:28 +02:00
Dennis Camera 985252585c [type/__pyvenv] Fix if --owner / --group is numeric
Before, if --owner and/or --group was numeric, gencode-remote would generate
`chown` code every time.
2021-03-30 13:26:21 +02:00
Dennis Camera 167c2ad7ea [type/__git] Fix if --owner / --group is numeric
Before, if --owner and/or --group was numeric, gencode-remote would generate
`chown` code every time.
2021-03-30 13:24:56 +02:00
Darko Poljak 7a0b697f4c Implement maintaining object relationship graph
For each object maintain parent-child relationship graph, i.e. list of
parent objects ('parents' property) and list of children objects ('children'
property).

Objects without parent(s) are objects specified in init manifest.
Objects without children are object of types that do not reuse other types.
2021-03-30 12:09:59 +02:00
poljakowski c55397766e Merge branch 'feature/type/__sshd_config/whitelist-openbmc' into 'master'
__sshd_config: Whitelist OpenBMC

See merge request ungleich-public/cdist!980
2021-03-12 08:20:35 +01:00
Dennis Camera e47c4dd8a4 [type/__sshd_config] Whitelist OpenBMC in manifest 2021-03-11 14:17:44 +01:00
Dennis Camera fb19f34266 [type/__ssh_authorized_key] Only grep if file exists 2021-03-09 21:15:26 +01:00
Steven Armstrong ea0126dd81 Make local state dir available to custom remote scripts
Signed-off-by: Steven Armstrong <steven@icarus.ethz.ch>
2021-03-05 16:11:49 +01:00
poljakowski 1bc0d912bf Merge branch 'fix/type/__pyvenv/man-typo' into 'master'
__pyvenv: Fix user example

See merge request ungleich-public/cdist!978
2021-03-02 09:28:50 +01:00
Dennis Camera 8ef19d47f6 [type/__pyvenv] Fix example (--user -> --owner) 2021-03-01 17:59:45 +01:00
fancsali dc66efa690 Fix shellcheck issues 2021-02-23 11:59:09 +00:00
fancsali 1a74470c4d __apt_pin: Always use $__object_id as preferences.d filename 2021-02-23 09:43:02 +00:00
fancsali 0734288483 First draft of __apt_pin 2021-02-23 09:43:02 +00:00
poljakowski 6358885d26 Merge branch 'feature/__package_pip/extras' into 'master'
__package_pip: add optional (extra) dependencies

See merge request ungleich-public/cdist!975
2021-02-23 06:27:09 +01:00
poljakowski b3a9c907ad Merge branch '__letsencrypt_cert-fix-hooks' into 'master'
[__letsencrypt_cert] Fix various issues with hooks.

Closes #853

See merge request ungleich-public/cdist!977
2021-02-22 09:09:45 +01:00
poljakowski e854db096e Merge branch 'fix/type/__postgres_role/implement-alter' into 'master'
__postgres_role: implement modification of roles

See merge request ungleich-public/cdist!973
2021-02-22 08:58:58 +01:00
matze d1f45d3524 __package_pip: corrected typo in man
.. by fully replacing it with a smaller sentence.
2021-02-19 09:03:56 +01:00
Dennis Camera 0835f414a5 [type/__postgres_conf] Extract PostgreSQL service user detection to separate explorer 2021-02-16 16:03:23 +01:00
matze 2ce1fce767 __package_pip: match package names case insensitive
Pip matches them insensitive, so we need to do the same to avoid
problems by saying extras are not installed but already is there in
place.
2021-02-15 16:17:46 +01:00
matze 951712740f __package_pip: update man.rst
Adjusted comments for `explorer/extras` and updated the man page for the
new behaviour of updating the extras.
2021-02-12 13:42:51 +01:00
matze a9d7dfb2ed __package_pip: split extra 'all' to a list of all extras
This will fix if a package will be upgraded from some extras to all
extras. Previously, it will not work because some dependencies of 'all'
are already installed, so the feature 'all' is already installed.

Now, it will use a list of all extras to iterate over them separatly. This
will result it will never install all extras via `[all]`, but rather
`[foo,bar]`.
2021-02-12 09:17:02 +01:00
matze 7398382890 __package_pip: fix shellcheck
Useless `cat $file`, use `< $file` instead.
2021-02-11 23:12:10 +01:00
matze 2db0ef7c98 __package_pip: updating real detection of extras
As the previous detection took the wrong values, this explorer now
checks if packages for an extra are installed or not. If not, the extra
is not installed.

Based on the information of the explorer, it will install the package
again with the absent extras.
2021-02-11 22:53:26 +01:00
matze 8dc6ab9738 __package_pip: install not found extras
Compares the explorer against the parameters and install those extras
that are not already installed.
2021-02-11 13:49:53 +01:00
matze 4717e5ceff __package_pip: add extras explorer
The two new explorers detect all installed extras for this package.
2021-02-11 10:31:07 +01:00
evilham aa80c09c80 [__letsencrypt_cert] Move hook contents generation out of manifest
While there address some minor issues in the comments in the hook contents.
2021-02-10 10:10:21 +01:00
evilham b832af5e3b [__letsencrypt_cert] Don't mess with user script indentation
This could break in odd ways if they passed sth like:
cat <<eof
bla bla
eof
2021-02-09 20:53:58 +01:00
evilham e49da474c4 [__letsencrypt_cert] Remove problematic trailing slash in sed.
Happy fingers are happy and like adding slashes places.
2021-02-09 20:29:17 +01:00
evilham bc145bbc27 [__letsencrypt_cert] Fix various issues with hooks.
Closes #853, see issue for full description / discussion.

Short summary:
- There was about 6.53% chances of `--renewal-hook` not being applied
- Using --automatic-renewal in one cert and not in another was an error.
- It was not possible to use different hooks for different certificates.
- FreeBSD support was utterly broken.
2021-02-09 19:58:47 +01:00
ssrq cda17be38a [explorer/memory] Clean up, return kiB for all systems, add SunOS
BSDs were MiB before.
2021-02-08 08:27:03 +01:00
matze 73a03d75d7 __package_pip: fix shellcheck 2021-02-04 19:18:02 +01:00
matze 8eccacec59 __package_pip: add optional dependencies
This is a poor implementation of optional dependencies for pip packages.
It ensures to install them if the package will be installed, but does
not take into account if they must be added/removed after the package is
already installed. Also, it will not be autoremoved, as all dependencies
will not be removed.
2021-02-04 19:09:26 +01:00
Dennis Camera 6b18cace75 [type/__postgres_conf] Catch connection errors early 2021-01-26 14:01:44 +01:00
Dennis Camera f9ebb4333c [type/__postgres_conf] Add NetBSD PostgreSQL UNIX user 2021-01-26 14:01:44 +01:00
Dennis Camera 4967c7ebbb [type/__postgres_conf] Silence psql output 2021-01-26 14:01:44 +01:00
Dennis Camera 3f605c31ac [type/__postgres_conf] Add support for more init systems to restart service 2021-01-26 14:01:44 +01:00
Dennis Camera 0f2ff47738 [type/__postgres_conf] Restart PostgreSQL server based on pending_restart column of pg_settings 2021-01-26 14:01:44 +01:00
Dennis Camera 5051d4f40b [type/__postgres_conf] Catch invalid values 2021-01-26 14:01:44 +01:00
Dennis Camera 891c98567e [type/__postgres_conf] Compare configuration parameter names case insensitively 2021-01-26 14:01:44 +01:00
Dennis Camera 803367b316 [type/__postgres_conf] Fix default detection when default is also set in config file
e.g. port is usually also set to the default value in postgresql.conf
2021-01-26 14:01:44 +01:00
Dennis Camera 1b49fec972 [type/__postgres_conf] Refactor 2021-01-26 14:01:43 +01:00
Beni Ruef b4060720dc [type/__postgres_conf] Fix psql options for ALTER command 2021-01-26 14:01:43 +01:00
Beni Ruef 50bcd95105 [type/__postgres_conf] Remove faulty quotes 2021-01-26 14:01:43 +01:00
Beni Ruef 534d5f6bb5 [type/__postgres_conf] Fix errors found by ShellCheck 2021-01-26 14:01:43 +01:00
Beni Ruef c51d68a737 [type/__postgres_conf] New type based on ALTER SYSTEM command 2021-01-26 14:01:43 +01:00
Dennis Camera 35cde3e666 [type/__postgres_role] Fix state explorer when stored password is empty 2021-01-18 13:09:29 +01:00
Darko Poljak 92a50da487 Fix pycodestyle issues 2021-01-18 06:28:09 +01:00
poljakowski 878a65a8b7 Merge branch 'fix/type/__sshd_config/error-on-invalid' into 'master'
sshd config: Produce error if invalid config is generated, fix processing of AuthenticationMethods and AuthorizedKeysFile, document explorer bug

See merge request ungleich-public/cdist!968
2021-01-18 06:22:02 +01:00
poljakowski cce470b556 Merge branch 'bugfix/preos-debug' into 'master'
Fix debug parameter

Closes #849

See merge request ungleich-public/cdist!970
2021-01-18 06:17:36 +01:00
Dennis Camera 2954347771 [type/__postgres_role] Add note regarding empty passwords 2021-01-14 13:46:40 +01:00
Darko Poljak c819548343 Fix debug parameter
-d was removed from cdist in favor of mulitple -v and -l parameters, but
-d was not removed from preos.

Resolve #849.
2021-01-11 09:51:52 +01:00
Dennis Camera bd8ab8f26f [type/__sshd_config] Document "bug" in state explorer 2021-01-05 17:02:42 +01:00
Dennis Camera 8753b7eedf [type/__sshd_config] Make AuthenticationMethods and AuthorizedKeysFile singleton options
They were incorrectly treated as non-singleton options before.

cf. https://github.com/openssh/openssh-portable/blob/V_8_4/servconf.c#L2273
and https://github.com/openssh/openssh-portable/blob/V_8_4/servconf.c#L1899 resp.
2021-01-05 16:59:04 +01:00
Dennis Camera 766198912d [type/__sshd_config] Produce error if invalid config file is generated
Previously, cdist would silently swallow the error (no invalid config file was
generated).

Reason: `set -e` does not exit if a command in a sub-command group fails,
it merely returns with a non-zero exit status.

e.g. the following snippet does not abort the script if sshd -t returns with a
non-zero exit status:

    set -e
    cmp -s old new || {
        # check config file and update it
        sshd -t -f new \
        && cat new >old
    }

or compressed:

    set -e
    false || { false && true; }
    echo $?
    # prints 1
2021-01-05 15:50:21 +01:00
Nico Schottelius a10d43bc69 Merge branch 'master' of code.ungleich.ch:ungleich-public/cdist 2020-12-20 11:42:44 +01:00
Mark Verboom 8dc2c4207c Added optional dirmode parameter to set the mode of (optional) the directory. 2020-12-18 11:16:28 +01:00
Dennis Camera 99d82fd0d5 [type/__postgres_role] Always set psql -q 2020-12-17 17:05:58 +01:00
Dennis Camera 1180f13ed6 [type/__postgres_role] Fix setting password
We need to make sure that the password does not end up in ~/.psql_history.
2020-12-17 17:03:58 +01:00
Dennis Camera 4859c27900 [type/__postgres_role] Refactor gencode-remote 2020-12-17 16:57:43 +01:00
Dennis Camera 7b7ca4d385 [type/__postgres_role] Handle password changes 2020-12-16 19:07:05 +01:00
Dennis Camera c36df82882 [type/__postgres_role] ALTER ROLE when parameters change 2020-12-15 21:11:48 +01:00
Dennis Camera 932e2496ed [type/__postgres_role] Lint 2020-12-15 18:40:39 +01:00
poljakowski f87da8150c Merge branch 'type/__debian_backports' into 'master'
__apt_backports type

See merge request ungleich-public/cdist!964
2020-12-13 16:03:31 +01:00
matze 27aca06fb8 __apt_backports: undo __apt_update_index call
Becuase it is already done by __apt_source.
2020-12-12 17:34:51 +01:00
matze fca35fc858 __apt_backports: fix explorer call
s/-/_/ because the explorers are following an other convention :-)
2020-12-12 17:29:58 +01:00
evilham 645734c629 [explorer/os_version] Improve FreeBSD support.
It looks like uname -r is not the most reliable way to get the target patch
level for the target system.

For more information see:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=251743
2020-12-12 12:15:17 +01:00
matze fafa3d9ea5 __apt_backports: update index if required
This type now automatically calls the type __apt_update_index to update
the package index if something changed.
2020-12-12 10:00:23 +01:00
matze 49aec0b5e4 __apt_backports: list supported OSes
The manpage now lists all OSes where this type supports backports.
2020-12-12 09:40:47 +01:00
matze c4d19a2319 __debian_backports -> __apt_backports; add wider os support
As discussed in the chat, this type now supports a broader list of OSes
which it supports backports for. Because of this, it was renamed to
something more generic. "apt" should fit in.
2020-12-12 09:36:17 +01:00
Nico Schottelius 69b8bc9af0 Merge branch 'master' of code.ungleich.ch:ungleich-public/cdist 2020-12-11 19:38:03 +01:00
Nico Schottelius bc2948a8a5 ++scan stuff 2020-12-11 19:37:53 +01:00
matze 0d96b31b56 __debian_backports: pass shellcheck for sourced file
Because the sourced explorer can't be detected by shellcheck, it will be
completely disabled. Changing the path to /etc/os-release isn't
deterministic either.

The shellcheck wiki page suggests to use `source=/dev/null` instead of
`disable=SC1090`, but it was choosen to completely avoid that check ..
2020-12-11 18:13:44 +01:00
matze a5169ad858 new type __debian_backports
This new type will setup the backports distribution for the current
Debian release.
2020-12-10 21:24:26 +01:00
poljakowski 0546d6e476 Merge branch 'fix/__block/escape' into 'master'
__block: fix escaping in here-doc

Closes #838

See merge request ungleich-public/cdist!962
2020-12-08 19:36:45 +01:00
poljakowski a1987fe410 Merge branch 'feature/__iptables_rule/ipv6' into 'master'
__iptables*: add IPv6 support

See merge request ungleich-public/cdist!959
2020-12-08 07:10:29 +01:00
matze c5ca4cd2e1 __block: securly quote via the quote function
Because the function already exists, it will be used for the file to be
changed, too. Therefor, no quotes are required for that value.

The prefix and suffix match was also improved: There is no regex check
any more (the regex did checked the whole line); instead it will do it
simple.
2020-12-07 19:59:05 +01:00
poljakowski 226f665fb5 Merge branch 'imp-deprecation' into 'master'
Deal with deprecation of imp module.

See merge request ungleich-public/cdist!963
2020-12-07 19:48:08 +01:00
evilham bed08c2c5c Deal with deprecation of imp module.
importlib has been a thing since Python 3.1, and imp has been deprecated since
3.4.

Insert random complaint here about not being able to use f-strings because they
were introduced in Python 3.6 and apparently we support Python 3.5 >,<.

Output diff before to after for ./bin/cdist-build-helper test (on heavy load):
```
1,2d0
< /usr/home/evilham/s/cdist/cdist/cdist/test/__main__.py:23: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
<   import imp
72c70
< ERROR: cdisttesthost: __file/tmp/foobar requires object __file without object id. Defined at /tmp/tmp.cdist.test.g87lx7c8/tmp.cdist.test.6ramsakx
---
> ERROR: cdisttesthost: __file/tmp/foobar requires object __file without object id. Defined at /tmp/tmp.cdist.test.aqdf6vjz/tmp.cdist.test.jgv3udel
76c74
< test_nonexistent_type_requirement (cdist.test.emulator.EmulatorTestCase) ... ERROR: cdisttesthost: __file/tmp/foobar requires object __does-not-exist/some-id, but type __does-not-exist does not exist. Defined at /tmp/tmp.cdist.test.mma5j8ln/tmp.cdist.test.3zg4by4d
---
> test_nonexistent_type_requirement (cdist.test.emulator.EmulatorTestCase) ... ERROR: cdisttesthost: __file/tmp/foobar requires object __does-not-exist/some-id, but type __does-not-exist does not exist. Defined at /tmp/tmp.cdist.test.t8d6ockr/tmp.cdist.test.uimxurg9
86c84
< test_initial_manifest_environment (cdist.test.manifest.ManifestTestCase) ... VERBOSE: cdisttesthost: Running initial manifest /tmp/tmp.cdist.test.uvid60ij/759547ff4356de6e3d9e08522b0d0807/data/conf/manifest/dump_environment
---
> test_initial_manifest_environment (cdist.test.manifest.ManifestTestCase) ... VERBOSE: cdisttesthost: Running initial manifest /tmp/tmp.cdist.test._cttcnrj/759547ff4356de6e3d9e08522b0d0807/data/conf/manifest/dump_environment
89c87
< test_type_manifest_environment (cdist.test.manifest.ManifestTestCase) ... VERBOSE: cdisttesthost: Running type manifest /tmp/tmp.cdist.test.k1i2onpb/759547ff4356de6e3d9e08522b0d0807/data/conf/type/__dump_environment/manifest for object __dump_environment/whatever
---
> test_type_manifest_environment (cdist.test.manifest.ManifestTestCase) ... VERBOSE: cdisttesthost: Running type manifest /tmp/tmp.cdist.test.ukr7lrzd/759547ff4356de6e3d9e08522b0d0807/data/conf/type/__dump_environment/manifest for object __dump_environment/whatever
272c270
< Ran 225 tests in 44.457s
---
> Ran 225 tests in 43.750s
```
2020-12-06 20:24:00 +01:00
matze 3930f69456 __block: fix escaping in here-doc
This changes the here-document to do not interpret any shell-things. It
also single-quotes some more strings that are printed to code-remote.

Fixes #838
2020-12-06 16:45:58 +01:00
matze 087be130fa __iptables_apply: shorten copyright header
Do we need all the copyright header or is this sufficient? The licence
is given for cdist, but not on the target host. But it should be clear
anyway.
2020-12-04 19:23:49 +01:00
Dennis Camera 2d19856840 [type/__package_pkgng_freebsd] Set ASSUME_ALWAYS_YES instead of -y 2020-12-04 18:26:03 +01:00
matze ba7d16a155 __iptables_*: correct manpage spelling 2020-12-04 17:57:55 +01:00