Compare commits

..

17 commits

Author SHA1 Message Date
b4f381ed4c __ini_value: fix use of removed constant
_param was removed in preference of the param functions.
2021-09-29 20:24:22 +02:00
122354a0dc __ini_value: draft of comment handling
work in progress for a long time and I didn't touched it again till just
to commit it ... I have to work on it.
2021-09-29 20:23:01 +02:00
9d9577d891 Merge remote-tracking branch 'origin/master' into new/__ini_value 2021-09-14 18:03:42 +02:00
0d4868dcd5 __ini_value: new parameter --quote
This parameter ensures the value is surrounded by double quotes. As it
directly edits the value variable, the quoting will be added/removed
automaticly.
2021-03-20 15:39:38 +01:00
19c701e95d __ini_value: make type nonparallel
Because of the temporary files and because multiple types can change a
file at the same time, it can't be run in parallel.
2021-03-20 15:10:11 +01:00
46638f0839 __ini_value: change empty value check
Change the short-circuit check for an empty value to a more structual
approach. This might be better for the code or not ..
2021-03-17 18:17:21 +01:00
ad736a66d0 __ini_value: remove is_state 'nosuchfile'
Because the state 'nosuchfile' is exactly the same as 'absent', it was
removed with the code connected to it. It's not important if the file
exists or not - because in both cases, it contains no key-value.

Also, the code if the explorer returned 'nosuchfile' was wrong:
Completly overwrite the file with the assumption the file does not exist
is not correct, as it can return for multiple objects of the same file
and therefore, they overwrite themself.
2021-03-16 21:38:08 +01:00
8f687e5ce2 __ini_value: make shellcheck happy 2021-03-14 14:50:21 +01:00
e3e4a91abe __ini_value: moved awk script generation to gencode-remote
The script generation of the awk script was moved from `files/gen-awk.sh`
to `gencode-remote` because the size not really matters. If the file
does not exist, it will be created by a predefined template via an
here-doc to avoid a big awk-script where it is not needed.
2021-03-14 12:28:55 +01:00
7380fcaaf9 __ini_value: add missing bracket
(oops)
2021-03-13 18:46:04 +01:00
b76d848540 __ini_value: add delimiter space detection
This adds spaces around the delimiter to provide some flexibility than
just do it in the delimiter string directly. It can be set via the
parameter `--delimiter-space`.
2021-03-11 21:50:55 +01:00
9006994023 __ini_value: fix explorer detect value as commented
Because the value was not reseted on a fase positive.
2021-03-09 18:42:10 +01:00
ea7ac72f9a __ini_value: unify present + commented state awk scripts
Because the difference between the both states is just the comment sign
before it, it was now merged into one folder + symlink to avoid
redundancy. To acomplish the difference, a further parser step was
introduced to execute different print function based on the state (it is
a bit of hard-coded but ok).
2021-03-06 23:12:25 +01:00
3e67cdbf9b __ini_value: add space detection on insert
This detection tires to somehow inteligent places the new entry at the
end of the section, but tires to not infer with existing comments for
the section etc.

This detection is not perfect but will work in some cases. Hope it helps
(if someone whats to look at these files after they got edited).

This currently only applies to the `present` state, but not to the
`commented` state. This will be changed somehow when both scripts will
be unifed cause of there great similarities.
2021-03-06 22:09:59 +01:00
72c6306ba2 __ini_value: add man.rst 2021-03-06 17:46:40 +01:00
10427fde84 __ini_value: implement multi-line buffer
This buffer gives the ability to look a bit longer into the past than
just a single line. This is helpfull to get a bigger context when
fiddling around comments. This also erases the need of
`lastlinepopulated` as there is something in the pipe or nothing.
2021-03-06 17:06:32 +01:00
8c7a6906de __ini_value: starting base parts
Initial body of the __ini_value. It contains two awk-scripts: to detect
the state that is important to trigger a code generation, and to change
that line to the correct line. It might have problems which is not
matured enoght.

The most difficult points will be the comment detection, as this is a
critical point as you don't know if the user want to have this touched.
2021-03-05 20:38:19 +01:00
53 changed files with 755 additions and 585 deletions

View file

@ -57,11 +57,6 @@ __file "/etc/apt/preferences.d/$name" \
--owner root --group root --mode 0644 \ --owner root --group root --mode 0644 \
--state "$state" \ --state "$state" \
--source - << EOF --source - << EOF
# Created by cdist ${__type##*/}
# Do not change. Changes will be overwritten.
#
# $name
Package: $package Package: $package
Pin: $pin Pin: $pin
Pin-Priority: $priority Pin-Priority: $priority

View file

@ -1,3 +1,2 @@
state state
package package
priority

View file

@ -1 +1,2 @@
distribution distribution
priority

View file

@ -22,21 +22,7 @@
name="$__object_id" name="$__object_id"
destination="/etc/apt/sources.list.d/${name}.list" destination="/etc/apt/sources.list.d/${name}.list"
# There are special arguments to apt(8) to prevent aborts if apt woudn't been
# updated after the 19th April 2021 till the bullseye release. The additional
# arguments acknoledge the happend suite change (the apt(8) update does the
# same by itself).
#
# Using '-o $config' instead of the --allow-releaseinfo-change-* parameter
# allows backward compatablility to pre-buster Debian versions.
#
# See more: ticket #861
# https://code.ungleich.ch/ungleich-public/cdist/-/issues/861
apt_opts="-o Acquire::AllowReleaseInfoChange::Suite=true -o Acquire::AllowReleaseInfoChange::Version=true"
# run 'apt-get update' only if something changed with our sources.list file
# it will be run a second time on error as a redundancy messure to success
if grep -q "^__file${destination}" "$__messages_in"; then if grep -q "^__file${destination}" "$__messages_in"; then
printf 'apt-get %s update || apt-get %s update\n' "$apt_opts" "$apt_opts" printf 'apt-get update || apt-get update\n'
fi fi

View file

@ -18,23 +18,9 @@
# along with cdist. If not, see <http://www.gnu.org/licenses/>. # along with cdist. If not, see <http://www.gnu.org/licenses/>.
# #
# There are special arguments to apt(8) to prevent aborts if apt woudn't been
# updated after the 19th April 2021 till the bullseye release. The additional
# arguments acknoledge the happend suite change (the apt(8) update does the
# same by itself).
#
# Using '-o $config' instead of the --allow-releaseinfo-change-* parameter
# allows backward compatablility to pre-buster Debian versions.
#
# See more: ticket #861
# https://code.ungleich.ch/ungleich-public/cdist/-/issues/861
apt_opts="-o Acquire::AllowReleaseInfoChange::Suite=true -o Acquire::AllowReleaseInfoChange::Version=true"
# run 'apt-get update' if anything in /etc/apt is newer then /var/lib/apt/lists # run 'apt-get update' if anything in /etc/apt is newer then /var/lib/apt/lists
# it will be run a second time on error as a redundancy messure to success
cat << DONE cat << DONE
if find /etc/apt -mindepth 1 -cnewer /var/lib/apt/lists | grep . > /dev/null; then if find /etc/apt -mindepth 1 -cnewer /var/lib/apt/lists | grep . > /dev/null; then
apt-get $apt_opts update || apt-get $apt_opts update apt-get update || apt-get update
fi fi
DONE DONE

View file

@ -1 +0,0 @@
'file' has been deprecated in favour of 'line' in order to provide idempotency.

View file

@ -1,7 +1,7 @@
#!/bin/sh -e #!/bin/sh -e
# #
# 2011-2012 Nico Schottelius (nico-cdist at schottelius.org) # 2011-2012 Nico Schottelius (nico-cdist at schottelius.org)
# 2013-2022 Steven Armstrong (steven-cdist armstrong.cc) # 2013 Steven Armstrong (steven-cdist armstrong.cc)
# #
# This file is part of cdist. # This file is part of cdist.
# #
@ -89,26 +89,10 @@ if [ "$state_should" = "present" ] || [ "$state_should" = "exists" ]; then
touch "$__object/files/set-attributes" touch "$__object/files/set-attributes"
# upload file to temp location # upload file to temp location
upload_destination="$(mktemp -u "${destination}.cdist.XXXXXXXXXX")" tempfile_template="${destination}.cdist.XXXXXXXXXX"
# Yes, we are aware that this is a race condition.
# However:
# a) cdist usually writes to directories that are not user writable
# (probably > 99.9%)
# b) if they are user owned, the user / attacker always wins
# (probably < 0.1%)
# c) the only case which we could improve are tmp directories and we
# don't think managing tmp directories with cdist is a typical case
# ("the rest %)"
cat << DONE cat << DONE
$__remote_exec $__target_host test -e $upload_destination && { destination_upload="\$($__remote_exec $__target_host "mktemp $tempfile_template")"
echo "Refusing to upload file to existing destination: $upload_destination" >&2
exit 1
}
DONE DONE
# Tell gencode-remote to where we uploaded the file so it can move
# it to its final destination.
echo "$upload_destination" > "$__object/files/upload-destination"
if [ "$upload_file" ]; then if [ "$upload_file" ]; then
echo upload >> "$__messages_out" echo upload >> "$__messages_out"
# IPv6 fix # IPv6 fix
@ -119,8 +103,12 @@ DONE
my_target_host="${__target_host}" my_target_host="${__target_host}"
fi fi
cat << DONE cat << DONE
$__remote_copy "$source" "${my_target_host}:${upload_destination}" $__remote_copy "$source" "${my_target_host}:\$destination_upload"
DONE DONE
fi fi
# move uploaded file into place
cat << DONE
$__remote_exec $__target_host "rm -rf \"$destination\"; mv \"\$destination_upload\" \"$destination\""
DONE
fi fi
fi fi

View file

@ -1,7 +1,7 @@
#!/bin/sh -e #!/bin/sh -e
# #
# 2011-2013 Nico Schottelius (nico-cdist at schottelius.org) # 2011-2013 Nico Schottelius (nico-cdist at schottelius.org)
# 2013-2022 Steven Armstrong (steven-cdist armstrong.cc) # 2013 Steven Armstrong (steven-cdist armstrong.cc)
# #
# This file is part of cdist. # This file is part of cdist.
# #
@ -62,13 +62,6 @@ set_mode() {
case "$state_should" in case "$state_should" in
present|exists) present|exists)
if [ -f "$__object/files/upload-destination" ]; then
final_destination="$destination"
# We change the 'global' $destination variable here so we can
# change attributes of the new/uploaded file before moving it
# to it's final destination.
destination="$(cat "$__object/files/upload-destination")"
fi
# Note: Mode - needs to happen last as a chown/chgrp can alter mode by # Note: Mode - needs to happen last as a chown/chgrp can alter mode by
# clearing S_ISUID and S_ISGID bits (see chown(2)) # clearing S_ISUID and S_ISGID bits (see chown(2))
for attribute in group owner mode; do for attribute in group owner mode; do
@ -88,11 +81,6 @@ case "$state_should" in
fi fi
fi fi
done done
if [ -f "$__object/files/upload-destination" ]; then
# move uploaded file into place
printf 'rm -rf "%s"\n' "$final_destination"
printf 'mv -T "%s" "%s"\n' "$destination" "$final_destination"
fi
if [ -f "$__object/files/set-attributes" ]; then if [ -f "$__object/files/set-attributes" ]; then
# set-attributes is created if file is created or uploaded in gencode-local # set-attributes is created if file is created or uploaded in gencode-local
fire_onchange=1 fire_onchange=1

View file

@ -15,7 +15,7 @@ case $os in
# Differntation not needed anymore # Differntation not needed anymore
apt_source_distribution=stable apt_source_distribution=stable
;; ;;
10*|11*) 10*)
# Differntation not needed anymore # Differntation not needed anymore
apt_source_distribution=stable apt_source_distribution=stable
;; ;;

View file

@ -1,8 +0,0 @@
frontend http
bind BIND@:80
mode http
option httplog
default_backend http
backend http
mode http

View file

@ -1,10 +0,0 @@
frontend https
bind BIND@:443
mode tcp
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend https
backend https
mode tcp

View file

@ -1,12 +0,0 @@
frontend imaps
bind BIND@:143
bind BIND@:993
mode tcp
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend imaps
backend imaps
mode tcp

View file

@ -1,12 +0,0 @@
frontend smtps
bind BIND@:25
bind BIND@:465
mode tcp
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend smtps
backend smtps
mode tcp

View file

@ -1,121 +0,0 @@
cdist-type__haproxy_dualstack(7)
================================
NAME
----
cdist-type__haproxy_dualstack - Proxy services from a dual-stack server
DESCRIPTION
-----------
This (singleton) type installs and configures haproxy to act as a dual-stack
proxy for single-stack services.
This can be useful to add IPv4 support to IPv6-only services while only using
one IPv4 for many such services.
By default this type uses the plain TCP proxy mode, which means that there is no
need for TLS termination on this host when SNI is supported.
This also means that proxied services will not receive the client's IP address,
but will see the proxy's IP address instead (that of `$__target_host`).
This can be solved by using the PROXY protocol, but do take into account that,
e.g. nginx cannot serve both regular HTTP(S) and PROXY protocols on the same
port, so you will need to use other ports for that.
As a recommendation in this type: use TCP ports 8080 and 591 respectively to
serve HTTP and HTTPS using the PROXY protocol.
See the EXAMPLES for more details.
OPTIONAL PARAMETERS
-------------------
v4proxy
Proxy incoming IPv4 connections to the equivalent IPv6 endpoint.
In its simplest use, it must be a NAME with an `AAAA` DNS entry, which is
the IP address actually providing the proxied services.
The full format of this argument is:
`[proxy:]NAME[[:PROTOCOL_1=PORT_1]...[:PROTOCOL_N=PORT_N]]`
Where starting with `proxy:` determines that the PROXY protocol must be
used and each `:PROTOCOL=PORT` (e.g. `:http=8080` or `:https=591`) is a PORT
override for the given PROTOCOL (see `--protocol`), if not present the
PROTOCOL's default port will be used.
v6proxy
Proxy incoming IPv6 connections to the equivalent IPv4 endpoint.
In its simplest use, it must be a NAME with an `A` DNS entry, which is
the IP address actually providing the proxied services.
See `--v4proxy` for more options and details.
protocol
Can be passed multiple times or as a space-separated list of protocols.
Currently supported protocols are: `http`, `https`, `imaps`, `smtps`.
This defaults to: `http https imaps smtps`.
EXAMPLES
--------
.. code-block:: sh
# Proxy the IPv6-only services so IPv4-only clients can access them
# This uses HAProxy's TCP mode for http, https, imaps and smtps
__haproxy_dualstack \
--v4proxy ipv6.chat \
--v4proxy matrix.ungleich.ch
# Proxy the IPv6-only HTTP(S) services so IPv4-only clients can access them
# Note this means that the backend IPv6-only server will only see
# the IPv6 address of the haproxy host managed by cdist, which can be
# troublesome if this information is relevant for analytics/security/...
# See the PROXY example below
__haproxy_dualstack \
--protocol http --protocol https \
--v4proxy ipv6.chat \
--v4proxy matrix.ungleich.ch
# Use the PROXY protocol to proxy the IPv6-only HTTP(S) services enabling
# IPv4-only clients to access them while maintaining the client's IP address
__haproxy_dualstack \
--protocol http --protocol https \
--v4proxy proxy:ipv6.chat:http=8080:https=591 \
--v4proxy proxy:matrix.ungleich.ch:http=8080:https=591
# Note however that the PROXY protocol is not compatible with regular
# HTTP(S) protocols, so your nginx will have to listen on different ports
# with the PROXY settings.
# Note that you will need to restrict access to the 8080 port to prevent
# Client IP spoofing.
# This can be something like:
# server {
# # listen for regular HTTP connections
# listen [::]:80 default_server;
# listen 80 default_server;
# # listen for PROXY HTTP connections
# listen [::]:8080 proxy_protocol;
# # Accept the Client's IP from the PROXY protocol
# real_ip_header proxy_protocol;
# }
SEE ALSO
--------
- https://www.haproxy.com/blog/enhanced-ssl-load-balancing-with-server-name-indication-sni-tls-extension/
- https://www.haproxy.com/blog/haproxy/proxy-protocol/
- https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/
AUTHORS
-------
ungleich <foss--@--ungleich.ch>
Evilham <cvs--@--evilham.com>
COPYING
-------
Copyright \(C) 2021 ungleich glarus ag. You can redistribute it
and/or modify it under the terms of the GNU General Public License as
published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.

View file

@ -1,155 +0,0 @@
#!/bin/sh -eu
__package haproxy
require="__package/haproxy" __start_on_boot haproxy
tmpdir="$__object/files"
mkdir "$tmpdir"
configtmp="$__object/files/haproxy.cfg"
os=$(cat "$__global/explorer/os")
case $os in
freebsd)
CONFIG_FILE="/usr/local/etc/haproxy.conf"
cat <<EOF > "$configtmp"
global
maxconn 4000
user nobody
group nogroup
daemon
EOF
;;
*)
CONFIG_FILE="/etc/haproxy/haproxy.cfg"
cat <<EOF > "$configtmp"
global
log [::1] local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
EOF
;;
esac
cat <<EOF >> "$configtmp"
defaults
retries 3
log global
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
EOF
dig_cmd="$(command -v dig || true)"
get_ip() {
# Usage: get_ip (ipv4|ipv6) NAME
# uses "dig" if available, else fallback to "host"
case $1 in
ipv4)
if [ -n "${dig_cmd}" ]; then
${dig_cmd} +short A "$2"
else
host -t A "$2" | cut -d ' ' -f 4 | grep -v 'found:'
fi
;;
ipv6)
if [ -n "${dig_cmd}" ]; then
${dig_cmd} +short AAAA "$2"
else
host -t AAAA "$2" | cut -d ' ' -f 5 | grep -v 'NXDOMAIN'
fi
;;
esac
}
PROTOCOLS="$(cat "$__object/parameter/protocol")"
for proxy in v4proxy v6proxy; do
param=$__object/parameter/$proxy
# no backend? skip generating code
if [ ! -f "$param" ]; then
continue
fi
# turn backend name into bind parameter: v4backend -> ipv4@
bind=$(echo $proxy | sed -e 's/^/ip/' -e 's/proxy//')
case $bind in
ipv4)
backendproto=ipv6
;;
ipv6)
backendproto=ipv4
;;
esac
for proto in ${PROTOCOLS}; do
# Add protocol "header"
printf "\n# %s %s \n" "${bind}" "${proto}" >> "$configtmp"
sed -e "s/BIND/$bind/" \
-e "s/\(frontend[[:space:]].*\)/\1$bind/" \
-e "s/\(backend[[:space:]].*\)/\\1$bind/" \
"$__type/files/$proto" >> "$configtmp"
while read -r hostdefinition; do
if echo "$hostdefinition" | grep -qE '^proxy:'; then
# Proxy protocol was requested
host="$(echo "$hostdefinition" | sed -E 's/^proxy:([^:]+).*$/\1/')"
send_proxy=" send-proxy"
else
# Just use tcp proxy mode
host="$hostdefinition"
send_proxy=""
fi
if echo "$hostdefinition" | grep -qE ":${proto}="; then
# Use custom port definition if requested
port="$(echo "$hostdefinition" | sed -E "s/^(.*:)?${proto}=([0-9]+).*$/:\2/")"
else
# Else use the default
port=""
fi
servername=$host
res=$(get_ip "$bind" "$servername")
if [ -z "$res" ]; then
echo "$servername does not resolve - aborting config" >&2
exit 1
fi
# Treat protocols without TLS+SNI specially
if [ "$proto" = http ]; then
echo " use-server $servername if { hdr(host) -i $host }" >> "$configtmp"
else
echo " use-server $servername if { req_ssl_sni -i $host }" >> "$configtmp"
fi
# Create the "server" itself.
# Note that port and send_proxy will be empty unless
# they were requested by the type user
echo " server $servername ${backendproto}@${host}${port}${send_proxy}" >> "$configtmp"
done < "$param"
done
done
# Create config file
require="__package/haproxy" __file ${CONFIG_FILE} --source "$configtmp" --mode 0644
require="__file${CONFIG_FILE}" __check_messages "haproxy_reload" \
--pattern "^__file${CONFIG_FILE}" \
--execute "service haproxy reload || service haproxy restart"

View file

@ -1 +0,0 @@
http https imaps smtps

View file

@ -1,3 +0,0 @@
protocol
v4proxy
v6proxy

View file

@ -0,0 +1,165 @@
#!/bin/sh -e
# __ini_value/explorer/state
# Check the state of the key-value pair in the ini file
#
# There are following states:
# - present
# - wrongvalue
# - wrongformat
# - commented
# - absent
# - nosuchfile
# Using ' \t' for matching spaces as char classes not implemented in mawk
# see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=65617#40
# Parameters
# (maybe multi-variable object id for this ..)
#state_should="$(cat "$__object/parameter/state")"
file="$(cat "$__object/parameter/file")"
# abort if no file exist
if ! [ -f "$file" ]; then
echo absent
exit
fi
# run awk
awk -f - "$file" <<'AWK'
function trim(var) {
sub(/^[ \t]*/, "", var)
sub(/[ \t]*$/, "", var)
return var
}
function check_spaces(var) {
return match(var, /^[ \t]*$/) == 1
}
function state(val) {
print val
exit
}
BEGIN {
_param = (ENVIRON["__object"] "/parameter/")
getline state_should < (_param "state")
getline section < (_param "section")
getline key < (_param "key")
getline delimiter < (_param "delimiter")
getline value < (_param "value")
getline indentation < (_param "indentation")
getline delimiter_space < (_param "delimiter-space")
do_normalization = (system("test -f " (_param "normalize")) == 0)
i=0; _comm_param = (_param "comment-sign");
while((getline tmp < _comm_param) > 0) {
comment_signs[i++] = tmp
}
if(system("test -f " (_param "quote")) == 0) {
# quote it now that it only wins checks against quoted values
value = ("\"" value "\"")
}
found=0
curr_section=""
if(section == "")
found_section=1
else
found_section=0
}
# catch sections
/^[ \t]*\[.*\][ \t]*$/ {
curr_section = trim($0)
if(found_section)
exit # game over, section ends
if(section == curr_section)
found_section=1
next
}
# only interesting if a delimiter was found
found_section {
line = $0
# index 1 cause of trimmed string
if((idel = index(line, delimiter)) && (ikey = index(line, key))) {
is_com=0
if(ikey > 1) {
# maybe comment character or only spaces
start_string = substr(line, 1, ikey - 1)
# something inside rather than a space -> comment
if((icom = match(start_string, /[^ \t]+/)) > 0) {
# icom = RSTART
# only one free-standing char or directly before the key
if(RLENGTH == 1 || icom == ikey - 1) {
start_sign = substr(line, RSTART, 1)
for(i in comment_signs) {
if(start_sign == comment_signs[i]) {
is_com = 1; break;
}
}
if(!is_com) next
else {
aftercom_length = ikey - icom - 1
if(!check_spaces(substr(line, icom + 1, aftercom_length))) next
start_spaces = (icom - 1) + aftercom_length
}
}
else next
}
# must only contain spaces
else start_spaces = ikey - 1
}
idelspace_start = ikey + length(key)
idelspace_length = idel - idelspace_start
# check for delimiter is only preceded with spaces
if(idelspace_length == 0 || check_spaces(substr(line, idelspace_start, idelspace_length))) {
found = 1
# short-circuit on state absent to just delete
if(state_should == "absent") state("present");
# extract the value
found_value = substr(line, idel + length(delimiter))
is_value = trim(found_value)
# check if value is incorrect
if(value != is_value) state("wrongvalue")
else {
# check if the format is important
if(do_normalization) {
if(match(found_value, /^[ \t]+/) == 1) {
found_value = substr(found_value, 1 + RLENGTH)
del_val_spacelen = RLENGTH
}
else
del_val_spacelen = 0
# the format must exactly match, else it is incorrect
if(start_spaces != indentation || found_value != is_value ||
idelspace_length != delimiter_space || del_val_spacelen != delimiter_space)
state("wrongformat")
}
if(is_com)
state("commented")
else
state("present")
}
# this will never be reached
}
}
}
# in the end, check if it is absent
END {
if(!found)
state("absent")
}
AWK

View file

@ -0,0 +1,138 @@
BEGIN {
bufindex = -1
buflen = 0
maxbuflen = 10
# no section means the start to the first section
if(section == "") {
is_curr_section = 1
found_section = 1
}
}
# controls the line buffer
function flush_buffer() {
while(buflen > 0)
_pop_line()
}
function flush_lines(n) {
while(buflen > 0 && n-- > 0)
_pop_line()
}
function push_line() {
linebuf[++bufindex] = $0
buflen++
while(buflen > maxbuflen) _pop_line()
}
function revert_line() {
# no delete, because it will be overwritten by the next line if any ..
bufindex--
buflen--
}
function lastline() {
if(buflen > 0) return linebuf[bufindex]
}
function pop_line() {
if(buflen > 0) _pop_line()
}
function _pop_line() {
_index = bufindex - (--buflen)
print linebuf[_index]
delete linebuf[_index]
}
# excepts the first character is the sign to check (string is trimmed)
function is_comment(line) {
# get character and check
line_sign = substr(line, 1, 1)
for(c in comment_signs)
if(line_sign == comment_signs[c])
return 1
# nothing found
return 0
}
function was_comment(line, comment) {
line = trim(line)
if(is_comment(line)) {
return trim(substr(line, 2)) == comment
}
}
# print everything if line found instead of processing it
# maybe just a function to loop through getline for lightest overhead
found {print; next}
# main loop (til the line was found)
!found {
line = trim($0)
# process if the line is not empty (or only contains spaces)
if(line != "") {
# check for a ini section
if(substr(line, 1, 1) == "[" && substr(line, length(line), 1) == "]") {
is_section = 1
curr_section = line
if(curr_section == section) {
found_section = 1
is_curr_section = 1
}
else {
# if nothing found, print it in the valid section before the next one
if(is_curr_section) {
if(!found) {
# set found as it is there now
found=1
# %codeblock_insert%
# print line as it would else only be populated below
print
next
}
is_curr_section = 0
}
}
}
else {
# only current session is interessting
if(is_curr_section) {
# check for a comment
is_com = is_comment(line)
if(is_com) {
line = trim(substr(line, 2))
}
# check for a delimiter and a key (must be at first position due to trimming)
if((idel = index(line, delimiter)) && (ikey = index(line, key)) == 1) {
# check there are only spaces between the key and delimiter
if(check_spaces(substr(line, ikey + length(key), idel - (length(key) + 1)))) {
found = 1
# %codeblock_found%
next
}
}
}
}
}
# works cause no next statement from above *structual programming*
push_line()
}
END {
# if not found, it's not already printed
if(!found) {
flush_buffer()
# print with section if not found
if(!found_section) {
# TODO check via buffer if a empty line is necessary
print section
# %codeblock_insert%
}
}
}

View file

@ -0,0 +1,57 @@
# We try to find a comment block -- how?
# check how much paragraphs it has
# check if
#
# this code is crap - at least not well written
# Check the buffer if the comment was found
function check_comments() {
_lastline = bufindex - (buflen - 1)
_comm_size = length(comments)
lastfreeline = 0
lastfreecommline = 0
comm_index = 0
# go through all lines
for(i = bufindex; i < _lastline; i++) {
_line = trim(linebuf[i])
# empty line?
if(_line == "") {
lastfreeline = i
continue
}
# line start matched
if(_line == trim(comments[comm_index])) {
# end? else continue
if(comm_index < _comm_size) {
continue
}
else {
}
}
# reset again cause not matched
else comm_index = 0
# empty comment line
if(is_comment(_line)) {
_comment = trim(substr(_line, 2))
# check if empty comment
if(_comment == "") {
lastfreecommline = i
}
}
# check if comments fit in or is too big
if((_lastline - bufindex) < _comm_size) {
# too short
}
else {
#if()
}
}
}

View file

@ -0,0 +1,68 @@
BEGIN {
# parameter variables
section = get_param_string("section")
key = get_param_string("key")
delimiter = get_param_string("delimiter")
value = get_param_string("value")
comment = get_param_string("comment")
indentation = get_param_string("indentation")
delimiter_space = get_param_string("delimiter-space")
get_param_array("comment-sign", comment_signs)
comment_sign = comment_signs[0]
if(system("test -f " (ENVIRON["__object"] "/parameter/quote")) == 0) {
# quote it now that it only wins checks against quoted values
value = ("\"" value "\"")
}
base_spaces = spaces(indentation)
delimiter_spaces = spaces(delimiter_space)
delimiter_w_spaces = (delimiter_spaces delimiter delimiter_spaces)
}
function trim(var) {
sub(/^[ \t]*/, "", var)
sub(/[ \t]*$/, "", var)
return var
}
function spaces(a) {
rspaces = ""
for(b = 0; b < a; b++)
rspaces = (rspaces " ")
return rspaces
}
function check_spaces(part) {
return match(part, /^[ \t]*$/) == 1
}
function get_param_string(name) {
_paramfile = (ENVIRON["__object"] "/parameter/" name)
if((getline tmp < _paramfile) > 0) {
close(_paramfile)
return tmp
}
else return ""
}
function get_param_array(name, arr) {
_paramfile = (ENVIRON["__object"] "/parameter/" name)
i=0
split("", arr) # portable clear, like `delete arr`
while((getline tmp < _paramfile) > 0) {
arr[i++] = tmp
}
close(_paramfile)
}
# print value
function v_print() {
printf "%s%s%s%s%s", base_spaces, key, delimiter_w_spaces, value, ORS
}
# print commented value
function v_print_commented() {
printf "%s%s%s%s%s%s", base_spaces, comment_sign, key, delimiter_w_spaces, value, ORS
}
# print comment
function c_print() {
printf "%s%s %s%s", base_spaces, comment_sign, comment, ORS
}

View file

@ -0,0 +1,5 @@
# revert line if it was a comment
if(was_comment(lastline, comment)) revert_line()
# value line was not pushed to the buffer yet
flush_buffer()

View file

@ -0,0 +1 @@
present

View file

@ -0,0 +1,8 @@
# check if last line was the comment
was_com_there = was_comment(lastline(), comment)
# print + comment if not there
flush_buffer()
if(comment && !was_com_there) c_print()
# %code_print%

View file

@ -0,0 +1,56 @@
# check if there is a comment block before the section
firstline_index = bufindex - (buflen - 1)
insertpoint = -1 # the insertpoint marks the point before the insert
lastfreespace = -1
for(i = bufindex; i >= firstline_index; i--) {
_line = trim(linebuf[i])
if(_line == "") {
lastfreespace = i
continue
}
if(comment && was_comment(_line, comment)) {
insertpoint = i + 1
no_insert_comment = 1
if(lastfreespace != insertpoint)
insert_line_after = 1
break
}
if(!is_comment(_line) || index(_line, delimiter) > 0) {
insertpoint = i + 1
# only insert a line before if we do not have a space around
if(lastfreespace == insertpoint)
insertpoint++
else
insert_line_before = 1
# check for empty line after the insert point
# use absolute boundary cause the insertpoint can be changed
if(trim(linebuf[i + 2]) != "")
insert_line_after = 1
break
}
}
# insert into the last free space
if(insertpoint == -1) {
if(lastfreespace != -1) {
insertpoint = lastfreespace
insert_line_before = 1
}
else {
insertpoint = firstline_index
insert_line_after = 1
}
}
# print lines before
flush_lines(insertpoint - firstline_index)
# print before and comment
if(insert_line_before) print ""
if(comment && !no_insert_comment) c_print()
# %code_print%
if(insert_line_after) print ""
flush_buffer()

View file

@ -0,0 +1,82 @@
#!/bin/sh -e
# __ini_value/gencode-remote
#
# Generates the code. It will generate an AWK script to add, modify or remove
# the line. The script differ in some points depend on the state. If the file
# does not exist, it will only generate the script without the awk overhead.
# strip comments and newlines for a tighter script
strip_comments() {
grep -v '^[[:space:]]*\($\|#\)'
}
state_is="$(cat "$__object/explorer/state")"
state_should="$(cat "$__object/parameter/state")"
# short-circuit if nothing to do
if [ "$state_is" = "$state_should" ]; then exit; fi
# file to change
file="$(cat "$__object/parameter/file")"
# validation check
case "$state_should" in
present|commented|absent)
# Generate the basic awk struct if a file already exists
cat <<SHELL
tmpfile="\$(mktemp '${file}.cdist.XXXXXXXX')"
if [ -f '$file' ]; then
cp -p '$file' "\$tmpfile"
fi
awk -f - '$file' > "\$tmpfile" <<'AWK'
SHELL
# generate the awk script and strip unnecessary things
{
# basic functions which everyone needs
cat "$__type/files/common.awk"
# generate the script
awk -v state="$state_should" '
function parse(line) {
if(match(line, /^[ \t]*# %code_print%$/) > 0) {
if(state == "present")
print "v_print()"
else if(state == "commented")
print "v_print_commented()"
else
print "print \"script compile error! cdist state " state " unkown!\" > /dev/stderr"
}
else print line
}
{
if(match($0, /^[ \t]*# %codeblock_([^%]+)%$/) > 0) {
split($2, result, "_"); type = substr(result[2], 1, length(result[2]) - 1)
file = (ENVIRON["__type"] "/files/parts/" state "/" type ".awk")
while((getline line < file) > 0)
parse(line)
close(file)
}
else print
}' "$__type/files/base.awk"
} | strip_comments
# end of here-doc
cat <<SHELL
AWK
mv -f "\$tmpfile" '$file'
SHELL
# Do not threat it differently if the file does not exist. It's just
# absent. Because multiple explorers can say the file does not exist,
# so the file should not be completly overwritten all times.
;;
*)
echo "not done yet!" >&2
exit 1
;;
esac

View file

@ -0,0 +1,139 @@
cdist-type__ini_value(7)
========================
NAME
----
cdist-type__ini_value - Handles ini- and conf-style configuration options
DESCRIPTION
-----------
This cdist type allow changes to more advanced key-value based configurations.
Most commonly this would be ini- or conf-style configurations.
The type can have following states:
present
The line exists with the correct value.
commented
The key-value is outcommented.
absent
The key-value line does not exist in the given section.
REQUIRED PARAMETERS
-------------------
file
The file to modify.
delimiter
The delimiter which seperates each key-value pair.
OPTIONAL PARAMETERS
-------------------
state
One of the states defined in the above section. Defaults to `present`.
section
The section where the value is located at. It always need to be surrounded
by square brackets as common for ini files. If not, the section will not be
found. If no section is specified, the block before any section is meant.
key
The key to identify the key-value pair. Must be set if the state is not
absent.
value
The value assigned to the key. Must be set if the state is not absent.
Else, an empty value is assigned to the given key.
comment
The comment which should be placed above the configuration line.
indentation
The indentation the key-value pair should have. Will be applied on inserts,
but also be enforced if ``--normalize`` is set.
comment-sign
This declares the comment signs that are valid to use in the configuration
file. Each parameter must declare a single character only; multiple
parameters are possible. It uses the first specified sign as comment
character if this type needs to insert comments.
delimiter-space
The number of spaces before and after the delimiter which should be free.
This number applies to each site of the delimiter separately, so one space
means one space to the left and right side of the delimiter.
The delimiter will be matched independendtly of this parameter and will
only be corrected if ``--normalize`` is set.
BOOLEAN PARAMETERS
------------------
normalize
This parameter enforces that the parameter is always pretty in the
configuration file. Even if a key-value pair is correct as-is, it will
correct the line to be pretty and perfect.
quote
Wrap double quotes (``"``) around the value. If the value is previously
unquoted, the file will be modified to quote the value.
MESSAGES
--------
The type currently fails to give a correct information of what he did cause of
the following construct. It has two `awk` scripts which do the job:
1. The explorer script which will outputs a single state of the given
key-value. Because the current state can contain much more states than the
state that should be, one state is returned like `wrongvalue` even if
`commented` is correct, too. Therefor, it vanishes the information that the
line is commented, too, even this could be a nice information that the
messaging system could emit.
2. The `code-remote` script also goes through the whole file and print out the
same file except the line line that should be changed. This is done because
it can not be garanteed that an other type already modifed the file, which
may moved the key-value to an other position. Then, the script replaces the
line which a pretty-printed key-value pair.
So the detected state is not important for the remote script, as it only needs
to know that it must be run cause of differences and what the state should be.
So if there are a state like `wrongvalue`, it triggers to correction of the
line, but it do not care if it was `wrongvalue`, `wrongformat` or `commented`
which trigged the run. Because of this need, the explorer retuns only an
easy-to-use value to detect if something needs to be changed.
Therefor, it is unable to correctly emit messages with the current base.
EXAMPLES
--------
.. code-block:: sh
# set a value in a configuration
__ini_value fancy-id --file /etc/foo/bar.ini --section '[welcome]' \
--key hi --value baz --delimiter ' = '
# outcomment a value
__ini_value foo --file /etc/bar/foo.conf --state commented \
--key noop --value true --delimiter ' = ' --comment 'not this time!'
AUTHORS
-------
Matthias Stecher <matthiasstecher at gmx.de>
COPYING
-------
Copyright \(C) 2021 Matthias Stecher. You can redistribute it
and/or modify it under the terms of the GNU General Public License as
published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.

View file

@ -0,0 +1,2 @@
normalize
quote

View file

@ -0,0 +1,2 @@
;
#

View file

@ -0,0 +1 @@
0

View file

@ -0,0 +1 @@
present

View file

@ -0,0 +1,7 @@
section
key
state
value
indentation
comment
delimiter-space

View file

@ -0,0 +1 @@
comment-sign

View file

@ -0,0 +1,2 @@
file
delimiter

View file

@ -41,7 +41,7 @@ if [ -z "${certbot_fullpath}" ]; then
require="__apt_source/stretch-backports" __package_apt certbot \ require="__apt_source/stretch-backports" __package_apt certbot \
--target-release stretch-backports --target-release stretch-backports
;; ;;
10*|11*) 10*)
__package_apt certbot __package_apt certbot
;; ;;

View file

@ -81,24 +81,12 @@ aptget="DEBIAN_FRONTEND=noninteractive apt-get --quiet --yes -o Dpkg::Options::=
case "$state_should" in case "$state_should" in
present) present)
# There are special arguments to apt(8) to prevent aborts if apt woudn't been
# updated after the 19th April 2021 till the bullseye release. The additional
# arguments acknoledge the happend suite change (the apt(8) update does the
# same by itself).
#
# Using '-o $config' instead of the --allow-releaseinfo-change-* parameter
# allows backward compatablility to pre-buster Debian versions.
#
# See more: ticket #861
# https://code.ungleich.ch/ungleich-public/cdist/-/issues/861
apt_opts="-o Acquire::AllowReleaseInfoChange::Suite=true -o Acquire::AllowReleaseInfoChange::Version=true"
# following is bit ugly, but important hack. # following is bit ugly, but important hack.
# due to how cdist config run works, there isn't # due to how cdist config run works, there isn't
# currently better way to do it :( # currently better way to do it :(
cat << EOF cat << EOF
if [ ! -f /var/cache/apt/pkgcache.bin ] || [ "\$( stat --format %Y /var/cache/apt/pkgcache.bin )" -lt "\$( date +%s -d '-1 day' )" ] if [ ! -f /var/cache/apt/pkgcache.bin ] || [ "\$( stat --format %Y /var/cache/apt/pkgcache.bin )" -lt "\$( date +%s -d '-1 day' )" ]
then echo apt-get $apt_opts update > /dev/null 2>&1 || true then echo apt-get update > /dev/null 2>&1 || true
fi fi
EOF EOF
if [ -n "$version" ]; then if [ -n "$version" ]; then

View file

@ -41,19 +41,7 @@ fi
case "$type" in case "$type" in
yum) ;; yum) ;;
apt) apt)
# There are special arguments to apt(8) to prevent aborts if apt woudn't been echo "apt-get --quiet update"
# updated after the 19th April 2021 till the bullseye release. The additional
# arguments acknoledge the happend suite change (the apt(8) update does the
# same by itself).
#
# Using '-o $config' instead of the --allow-releaseinfo-change-* parameter
# allows backward compatablility to pre-buster Debian versions.
#
# See more: ticket #861
# https://code.ungleich.ch/ungleich-public/cdist/-/issues/861
apt_opts="-o Acquire::AllowReleaseInfoChange::Suite=true -o Acquire::AllowReleaseInfoChange::Version=true"
echo "apt-get --quiet $apt_opts update"
echo "apt-cache updated (age was: $currage)" >> "$__messages_out" echo "apt-cache updated (age was: $currage)" >> "$__messages_out"
;; ;;
pacman) pacman)

View file

@ -28,10 +28,6 @@ apt_clean="$__object/parameter/apt-clean"
apt_dist_upgrade="$__object/parameter/apt-dist-upgrade" apt_dist_upgrade="$__object/parameter/apt-dist-upgrade"
if [ -f "$__object/parameter/apt-with-new-pkgs" ]; then
apt_with_new_pkgs="--with-new-pkgs"
fi
if [ -f "$type" ]; then if [ -f "$type" ]; then
type="$(cat "$type")" type="$(cat "$type")"
else else
@ -58,7 +54,7 @@ case "$type" in
apt) apt)
if [ -f "$apt_dist_upgrade" ] if [ -f "$apt_dist_upgrade" ]
then echo "$aptget dist-upgrade" then echo "$aptget dist-upgrade"
else echo "$aptget $apt_with_new_pkgs upgrade" else echo "$aptget upgrade"
fi fi
if [ -f "$apt_clean" ] if [ -f "$apt_clean" ]

View file

@ -33,14 +33,6 @@ BOOLEAN PARAMETERS
apt-dist-upgrade apt-dist-upgrade
Do dist-upgrade instead of upgrade. Do dist-upgrade instead of upgrade.
apt-with-new-pkg
Allow installing new packages when used in conjunction with
upgrade. This is useful if the update of an installed package
requires new dependencies to be installed. Instead of holding the
package back upgrade will upgrade the package and install the new
dependencies. Note that upgrade with this option will never remove
packages, only allow adding new ones.
apt-clean apt-clean
Clean out the local repository of retrieved package files. Clean out the local repository of retrieved package files.

View file

@ -1,3 +1,2 @@
apt-clean apt-clean
apt-dist-upgrade apt-dist-upgrade
apt-with-new-pkgs

View file

@ -1,16 +0,0 @@
#!/bin/sh -e
if [ -f "$__object/parameter/file" ]
then
file="$( cat "$__object/parameter/file" )"
else
file="/$__object_id"
fi
if [ ! -e "$file" ]
then
echo "$file does not exist" >&2
exit 1
fi
cat "$file"

View file

@ -1,58 +0,0 @@
#!/bin/sh -e
if [ -f "$__object/parameter/file" ]
then
file="$( cat "$__object/parameter/file" )"
else
file="/$__object_id"
fi
script="$( cat "$__object/parameter/script" )"
if [ "$script" = '-' ]
then
script="$( cat "$__object/stdin" )"
fi
# since stdin is not available in explorer, we pull file from target with explorer
file_from_target="$__object/explorer/file"
sed_cmd='sed'
if [ -f "$__object/parameter/regexp-extended" ]
then
sed_cmd="$sed_cmd -E"
fi
# do sed dry run, diff result and if no change, then there's nothing to do
# also redirect diff's output to stderr for debugging purposes
if echo "$script" | "$sed_cmd" -f - "$file_from_target" | diff -u "$file_from_target" - >&2
then
exit 0
fi
# we can't use -i, because it's not posix, so we fly with tempfile and cp
# and we use cp because we want to preserve destination file's attributes
# shellcheck disable=SC2016
echo 'tmp="$__object/tempfile"'
echo "$sed_cmd -f - '$file' > \"\$tmp\" << EOF"
echo "$script"
echo 'EOF'
echo "cp \"\$tmp\" '$file'"
# shellcheck disable=SC2016
echo 'rm -f "$tmp"'
echo 'change' >> "$__messages_out"
if [ -f "$__object/parameter/onchange" ]
then
cat "$__object/parameter/onchange"
fi

View file

@ -1,57 +0,0 @@
cdist-type__sed(7)
==================
NAME
----
cdist-type__sed - Transform text files with ``sed``
DESCRIPTION
-----------
Transform text files with ``sed``.
REQUIRED MULTIPLE PARAMETERS
----------------------------
script
``sed`` script.
If ``-`` then the script is read from ``stdin``.
OPTIONAL PARAMETERS
-------------------
file
Path to the file. Defaults to ``$__object_id``.
onchange
Execute this command if ``sed`` changes file.
BOOLEAN PARAMETERS
------------------
regexp-extended
Use extended regular expressions in the script.
Might not be supported with every ``sed`` version.
EXAMPLES
--------
.. code-block:: sh
__sed /tmp/foobar --script 's/foo/bar/'
echo 's/foo/bar/' | __sed foobar --file /tmp/foobar --script -
AUTHORS
-------
Ander Punnar <ander-at-kvlt-dot-ee>
COPYING
-------
Copyright \(C) 2021 Ander Punnar. You can redistribute it and/or modify it
under the terms of the GNU General Public License as published by the Free
Software Foundation, either version 3 of the License, or (at your option)
any later version.

View file

@ -1 +0,0 @@
regexp-extended

View file

@ -1,2 +0,0 @@
file
onchange

View file

@ -1 +0,0 @@
script

View file

@ -40,7 +40,6 @@ if [ -f "$file" ]; then
grep -v -F -x '$line' '$file' >\$tmpfile grep -v -F -x '$line' '$file' >\$tmpfile
fi fi
cat "\$tmpfile" >"$file" cat "\$tmpfile" >"$file"
rm -f "\$tmpfile"
DONE DONE
} }

View file

@ -84,7 +84,7 @@ def _process_hosts_simple(action, host, manifest, verbose,
""" """
if isinstance(host, str): if isinstance(host, str):
hosts = [host, ] hosts = [host, ]
elif isinstance(host, collections.abc.Iterable): elif isinstance(host, collections.Iterable):
hosts = host hosts = host
else: else:
raise cdist.Error('Invalid host argument: {}'.format(host)) raise cdist.Error('Invalid host argument: {}'.format(host))

View file

@ -33,7 +33,7 @@ class AbsolutePathRequiredError(cdist.Error):
return 'Absolute path required, got: {}'.format(self.path) return 'Absolute path required, got: {}'.format(self.path)
class FileList(collections.abc.MutableSequence): class FileList(collections.MutableSequence):
"""A list that stores it's state in a file. """A list that stores it's state in a file.
""" """
@ -102,7 +102,7 @@ class FileList(collections.abc.MutableSequence):
self.__write(lines) self.__write(lines)
class DirectoryDict(collections.abc.MutableMapping): class DirectoryDict(collections.MutableMapping):
"""A dict that stores it's items as files in a directory. """A dict that stores it's items as files in a directory.
""" """

View file

@ -3,17 +3,6 @@ Changelog
next: next:
* Explorer machine_type: Rewrite (Dennis Camera) * Explorer machine_type: Rewrite (Dennis Camera)
* New type: __sed (Ander Punnar)
* New type: __haproxy_dualstack (Evilham and ungleich)
* Type __apt_update_index: Fix complaint about suite change (Matthias Stecher)
* Type __package_update_index: Fix complaint about suite change (Matthias Stecher)
* Type __package_upgrade_all: Add new --apt-with-new-pkgs argument (Evilham)
* Type __apt_source: Fix complaint about suite change (Matthias Stecher)
* Type __package_apt: Fix complaint about suite change (Matthias Stecher)
* Type __debconf_set_selections: Fix bug where --file was unsupported (Evilham)
* Types __letsencrypt_cert, __grafana_dashboard: Improve bullseye support (Evilham)
* Type __ssh_authorized_key: Also remove tmpfile if removing line (Mark Verboom)
* Type __apt_pin: Add default priority, add comment in generated files (Daniel Fancsali)
6.9.8: 2021-08-24 6.9.8: 2021-08-24
* Type __rsync: Rewrite (Ander Punnar) * Type __rsync: Rewrite (Ander Punnar)