performance issue regarding global explorers ( cdist/conf/explorer )

ungleich-gitea commented

2021-11-20 11:24:42 +00:00

Owner

explorers on cdist/conf/explorer make the connection slower. it would be interesting to catch the eye how this is done and what kind of optimizations could be done

if the function or process that manage the global explorers is the same as for all explorer's types, it could be responsible of a general slow down of cdist operation

I just want to document here the tests, and let other people know. and from there, at least, understand why this happens and what can be done. and how much it affects

[21:29:09] $ du -sh cdist/conf/explorer
108K	explorer/

TODO: I would like to know how to execute all the chain of explorers to know how much time they need. This does not work: time sh cdist/conf/explorer/*

TODO: another interesting test would be to add a lot of dummy explorers

reproducibility

change myhost with other ssh hosts to test other possibilities

myhost=localhost
git clone https://code.ungleich.ch/ungleich-public/cdist
cd cdist/
rm -f cdist/conf/explorer/*
git checkout cdist/conf/explorer/hostname
echo "__ping" | bin/cdist config -c . -v -i - ${myhost}
git checkout cdist/conf/explorer/
echo "__ping" | bin/cdist config -c . -v -i - ${myhost}

localhost

INFO: localhost: Starting configuration run
INFO: localhost: Processing __ping/
INFO: localhost: Finished successful run in 0.63 seconds
Updated 18 paths from the index
INFO: localhost: Starting configuration run
INFO: localhost: Processing __ping/
INFO: localhost: Finished successful run in 0.93 seconds

bad remote host on same lan

bad remote host in my case is a laptopt with a broken wifi driver (so lots of packets are drop, etc.)

Updated 1 path from the index
INFO: 192.168.41.183: Starting configuration run
INFO: 192.168.41.183: Processing __ping/
INFO: 192.168.41.183: Finished successful run in 3.81 seconds
Updated 18 paths from the index
INFO: 192.168.41.183: Starting configuration run
INFO: 192.168.41.183: Processing __ping/
INFO: 192.168.41.183: Finished successful run in 10.53 seconds

good remote host

same laptop with iwlwifi and power management OFF (that way, lots of packets are dropped) and in the same LAN

Updated 1 path from the index
INFO: bbb.exo.cat: Starting configuration run
INFO: bbb.exo.cat: Processing __ping/
INFO: bbb.exo.cat: Finished successful run in 1.02 seconds
Updated 18 paths from the index
INFO: bbb.exo.cat: Starting configuration run
INFO: bbb.exo.cat: Processing __ping/
INFO: bbb.exo.cat: Finished successful run in 2.60 seconds

explorers on `cdist/conf/explorer` make the connection slower. it would be interesting to catch the eye how this is done and what kind of optimizations could be done if the function or process that manage the global explorers is the same as for all explorer's types, it could be responsible of a general slow down of cdist operation I just want to document here the tests, and let other people know. and from there, at least, understand why this happens and what can be done. and how much it affects ``` [21:29:09] $ du -sh cdist/conf/explorer 108K explorer/ ``` TODO: I would like to know how to execute all the chain of explorers to know how much time they need. This does not work: `time sh cdist/conf/explorer/*` TODO: another interesting test would be to add a lot of dummy explorers ## reproducibility *change myhost with other ssh hosts to test other possibilities* ``` myhost=localhost git clone https://code.ungleich.ch/ungleich-public/cdist cd cdist/ rm -f cdist/conf/explorer/* git checkout cdist/conf/explorer/hostname echo "__ping" | bin/cdist config -c . -v -i - ${myhost} git checkout cdist/conf/explorer/ echo "__ping" | bin/cdist config -c . -v -i - ${myhost} ``` ## localhost ``` INFO: localhost: Starting configuration run INFO: localhost: Processing __ping/ INFO: localhost: Finished successful run in 0.63 seconds Updated 18 paths from the index INFO: localhost: Starting configuration run INFO: localhost: Processing __ping/ INFO: localhost: Finished successful run in 0.93 seconds ``` ## bad remote host on same lan bad remote host in my case is a laptopt with a broken wifi driver (so lots of packets are drop, etc.) ``` Updated 1 path from the index INFO: 192.168.41.183: Starting configuration run INFO: 192.168.41.183: Processing __ping/ INFO: 192.168.41.183: Finished successful run in 3.81 seconds Updated 18 paths from the index INFO: 192.168.41.183: Starting configuration run INFO: 192.168.41.183: Processing __ping/ INFO: 192.168.41.183: Finished successful run in 10.53 seconds ``` ## good remote host same laptop with iwlwifi and power management OFF (that way, lots of packets are dropped) and in the same LAN ``` Updated 1 path from the index INFO: bbb.exo.cat: Starting configuration run INFO: bbb.exo.cat: Processing __ping/ INFO: bbb.exo.cat: Finished successful run in 1.02 seconds Updated 18 paths from the index INFO: bbb.exo.cat: Starting configuration run INFO: bbb.exo.cat: Processing __ping/ INFO: bbb.exo.cat: Finished successful run in 2.60 seconds ```

ungleich-gitea referenced this issue

2021-11-20 15:22:53 +00:00

apt_ppa does not refresh package list (Try #2) #265

ungleich-gitea commented

2021-11-20 15:24:27 +00:00

Author

Owner

a bad remote host is a host where lots of packets are dropped

it makes more sense to fix the bad remote host instead of tuning the way explorers are executed.

well I saw this and I just wanted to share my tests. Given the entire source code of the cdist project, I am saying that maybe we should review or discuss the part where explorers are uploaded to the remote host and how are executed there. Maybe another approach would improve overall speed of cdist.

a bad remote host is a host where lots of packets are dropped > it makes more sense to fix the bad remote host instead of tuning the way explorers are executed. well I saw this and I just wanted to share my tests. Given the entire source code of the cdist project, I am saying that maybe we should review or discuss the part where explorers are uploaded to the remote host and how are executed there. Maybe another approach would improve overall speed of cdist.

ungleich-gitea commented

2021-11-20 15:24:29 +00:00

Author

Owner

changed the description

ungleich-gitea commented

2021-11-20 15:24:30 +00:00

Author

Owner

Firstly, I don't understand what the difference between a "good" and a "bad" remote host is (and how that is related to iwlwifi?).

But, given that my cdist runs are usually around 120 seconds with -j4 the difference in run times with and without explorers seems insignificant to me (0.3s for localhost, 1.6s for good remote host).

@pedro Can you elaborate a bit more?
My gut feeling is that, in this case, it makes more sense to fix the bad remote host instead of tuning the way explorers are executed.

Firstly, I don't understand what the difference between a "good" and a "bad" remote host is (and how that is related to iwlwifi?). But, given that my cdist runs are usually around 120 seconds with `-j4` the difference in run times with and without explorers seems insignificant to me (0.3s for localhost, 1.6s for good remote host). @pedro Can you elaborate a bit more? My gut feeling is that, in this case, it makes more sense to fix the bad remote host instead of tuning the way explorers are executed.

ungleich-gitea commented

2021-11-20 15:24:31 +00:00

Author

Owner

changed the description