ssh connection multiplexing race condition #149

New issue

Closed

opened 2021-11-20 15:20:15 +00:00 by ungleich-gitea · 0 comments

ungleich-gitea commented

2021-11-20 15:20:15 +00:00

Owner

Created by: asteven

Someone elses ticket says more then a thousand words https://github.com/ansible/ansible/issues/16731

In short: ssh connection fails with errror -13 if the ssh client tries to use a master connection just while that is being closed due to timeout.

...
debug1: channel 1: free: mux-control, nchannels 2
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug1: auto-mux: Trying existing master
debug1: multiplexing control connection
debug1: channel 1: new [mux-control]
debug1: channel 2: new [client-session]
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending command: /bin/sh -c ' export __target_host=eu-ms-001-02.hpc-lca.ethz.ch;  export __target_hostname=eu-ms-001-02.hpc-lca.ethz.ch;  export __target_fqdn=eu-ms-001-02.hpc-lca.ethz.ch;  export __explorer=/var/lib/cdist/conf/explorer;/bin/sh -e /var/lib/cdist/conf/explorer/cpu_cores'
debug1: mux_client_request_session: master session id: 2
debug1: client_input_channel_req: channel 2 rtype exit-status reply 0
debug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0
debug1: channel 2: free: client-session, nchannels 3
debug1: channel 1: free: mux-control, nchannels 2
VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running initial manifest /tmp/tmpel0dijcy/f6e9674130e19aa6ce21ac3dfc79d272/data/conf/manifest/init
Start: Tue Jul  4 23:29:52 CEST 2017
debug1: ControlPersist timeout expired
debug1: channel 0: free: /tmp/tmp3bv0zoi7/s, nchannels 1
Transferred: sent 115836, received 52088 bytes, in 27.4 seconds
Bytes per second: sent 4231.7, received 1902.9
debug1: Exit status -1
End: Tue Jul  4 23:30:00 CEST 2017
VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Preparing object __centos_override_default_yum_repos/
VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running manifest and explorers for __centos_override_default_yum_repos/
VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running type explorers for <CdistType __centos_override_default_yum_repos>
VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running type manifest /tmp/tmpel0dijcy/f6e9674130e19aa6ce21ac3dfc79d272/data/conf/type/__centos_override_default_yum_repos/manifest
VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Preparing object __cdist_config_version/
VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running manifest and explorers for __cdist_config_version/
VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running type explorers for <CdistType __cdist_config_version>
VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running type manifest /tmp/tmpel0dijcy/f6e9674130e19aa6ce21ac3dfc79d272/data/conf/type/__cdist_config_version/manifest
VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Preparing object __file/etc/cdist-config-versions
VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running manifest and explorers for __file/etc/cdist-config-versions
VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running type explorers for <CdistType __file>
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug1: auto-mux: Trying existing master
debug1: multiplexing control connection
debug1: channel 1: new [mux-control]
debug1: ControlPersist timeout expired
debug1: channel 0: free: /tmp/tmpjiurfiu3/s, nchannels 2
debug1: channel 1: free: mux-control, nchannels 1
Transferred: sent 43428, received 14768 bytes, in 11.2 seconds
Bytes per second: sent 3881.2, received 1319.8
debug1: Exit status -1
ERROR: eu-ms-001-02.hpc-lca.ethz.ch: Command failed: ssh -v -o User=root -o ControlPath=/tmp/tmpjiurfiu3/s -o ControlMaster=auto -o ControlPersist=10 eu-ms-001-02.hpc-lca.ethz.ch mkdir -p /var/lib/cdist/conf/type/__file/explorer with returncode: -13
stdout: None
VERBOSE: cdist: Total processing time for 1 host(s): 11.400838851928711
ERROR: cdist: Failed to configure the following hosts: eu-ms-001-02.hpc-lca.ethz.ch

*Created by: asteven* Someone elses ticket says more then a thousand words https://github.com/ansible/ansible/issues/16731 In short: ssh connection fails with errror -13 if the ssh client tries to use a master connection just while that is being closed due to timeout. ``` ... debug1: channel 1: free: mux-control, nchannels 2 OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 56: Applying options for * debug1: auto-mux: Trying existing master debug1: multiplexing control connection debug1: channel 1: new [mux-control] debug1: channel 2: new [client-session] debug1: Sending environment. debug1: Sending env LANG = en_US.UTF-8 debug1: Sending command: /bin/sh -c ' export __target_host=eu-ms-001-02.hpc-lca.ethz.ch; export __target_hostname=eu-ms-001-02.hpc-lca.ethz.ch; export __target_fqdn=eu-ms-001-02.hpc-lca.ethz.ch; export __explorer=/var/lib/cdist/conf/explorer;/bin/sh -e /var/lib/cdist/conf/explorer/cpu_cores' debug1: mux_client_request_session: master session id: 2 debug1: client_input_channel_req: channel 2 rtype exit-status reply 0 debug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0 debug1: channel 2: free: client-session, nchannels 3 debug1: channel 1: free: mux-control, nchannels 2 VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running initial manifest /tmp/tmpel0dijcy/f6e9674130e19aa6ce21ac3dfc79d272/data/conf/manifest/init Start: Tue Jul 4 23:29:52 CEST 2017 debug1: ControlPersist timeout expired debug1: channel 0: free: /tmp/tmp3bv0zoi7/s, nchannels 1 Transferred: sent 115836, received 52088 bytes, in 27.4 seconds Bytes per second: sent 4231.7, received 1902.9 debug1: Exit status -1 End: Tue Jul 4 23:30:00 CEST 2017 VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Preparing object __centos_override_default_yum_repos/ VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running manifest and explorers for __centos_override_default_yum_repos/ VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running type explorers for <CdistType __centos_override_default_yum_repos> VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running type manifest /tmp/tmpel0dijcy/f6e9674130e19aa6ce21ac3dfc79d272/data/conf/type/__centos_override_default_yum_repos/manifest VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Preparing object __cdist_config_version/ VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running manifest and explorers for __cdist_config_version/ VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running type explorers for <CdistType __cdist_config_version> VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running type manifest /tmp/tmpel0dijcy/f6e9674130e19aa6ce21ac3dfc79d272/data/conf/type/__cdist_config_version/manifest VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Preparing object __file/etc/cdist-config-versions VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running manifest and explorers for __file/etc/cdist-config-versions VERBOSE: eu-ms-001-02.hpc-lca.ethz.ch: Running type explorers for <CdistType __file> OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 56: Applying options for * debug1: auto-mux: Trying existing master debug1: multiplexing control connection debug1: channel 1: new [mux-control] debug1: ControlPersist timeout expired debug1: channel 0: free: /tmp/tmpjiurfiu3/s, nchannels 2 debug1: channel 1: free: mux-control, nchannels 1 Transferred: sent 43428, received 14768 bytes, in 11.2 seconds Bytes per second: sent 3881.2, received 1319.8 debug1: Exit status -1 ERROR: eu-ms-001-02.hpc-lca.ethz.ch: Command failed: ssh -v -o User=root -o ControlPath=/tmp/tmpjiurfiu3/s -o ControlMaster=auto -o ControlPersist=10 eu-ms-001-02.hpc-lca.ethz.ch mkdir -p /var/lib/cdist/conf/type/__file/explorer with returncode: -13 stdout: None VERBOSE: cdist: Total processing time for 1 host(s): 11.400838851928711 ERROR: cdist: Failed to configure the following hosts: eu-ms-001-02.hpc-lca.ethz.ch ```