x2openEuler升级网络中断 connection failed 5 times! Connection timed out

gogodig · 2024 年10 月 23 日 09:13

x2openEuler升级网络中断 connection failed 5 times! Connection timed out

版本 3.0.0
发布时间：2024.09.30

yanzhicong · 2024 年10 月 29 日 00:48

可以看一下升级过程中的 ssh-connect 日志或者待升级节点的 /var/log/secure 日志，关于 ssh 连接的报错信息

gogodig · 2024 年10 月 29 日 07:04

yanzhicong · 2024 年10 月 30 日 00:59

升级失败的日志时间和 /var/log/secure 时间对不上

gogodig · 2024 年10 月 30 日 02:27

ETA\rmigrate-x86.repo 100% 543 32.6KB/s 00:00 \r\n
2024-10-23 03:27:29 Wed|1923:139927265728256|ssh_util.py:370|send_cmd_getresult_exception|INFO|Send command: [ -e /opt/x2openEuler/repo_files/migrate-x86.repo ]
2024-10-23 03:27:29 Wed|1923:139927265728256|ssh_util.py:541|_exec_cmd|INFO|status code is: 0, command result is:
2024-10-23 03:27:29 Wed|1923:139927265728256|ssh_util.py:994|check_upload_success|INFO|scp upload success: /opt/x2openEuler/repo_files/migrate-x86.repo
2024-10-23 03:27:29 Wed|1923:139927283037952|ssh_util.py:1122|send_nohup_command|INFO|<<<Start send nohup command: bash /opt/x2openEuler-upgrade/upgrade_entrance.sh upgrade /etc,/usr,/boot,/var,/opt,/bin,/sbin,/lib,/lib64 /var/cache,/var/tmp /.osbak crashkernel=512M,net.ifnames=0,biosdevname=0 /opt/x2openEuler-upgrade/output/upgrade-check/a8b603c1-910a-11ef-89b4-5254003660b7/check_result-a8b603c1-910a-11ef-89b4-5254003660b7.json /opt/x2openEuler-upgrade/output/upgrade-check/a8b603c1-910a-11ef-89b4-5254003660b7/rpm_dep-a8b603c1-910a-11ef-89b4-5254003660b7.txt OTHERS /root /opt/x2openEuler/repo_files/migrate-x86.repo, nohup log path: /var/log/x2openEuler-upgrade/upgrade-plus/upgrade-plus-nohup.log, local directory of command shell: /opt/x2openEuler/nohup-scripts, remote directory: /opt/x2openEuler/nohup-scripts.
2024-10-23 03:27:29 Wed|1923:139927283037952|ssh_util.py:445|can_switch_to_root|INFO|already root user, no need to change!
2024-10-23 03:27:29 Wed|1923:139927283037952|ssh_util.py:370|send_cmd_getresult_exception|INFO|Send command: [ -e /var/log/x2openEuler-upgrade/upgrade-plus ]
2024-10-23 03:27:30 Wed|1923:139927283037952|ssh_util.py:541|_exec_cmd|INFO|status code is: 0, command result is:
2024-10-23 03:27:30 Wed|1923:139927283037952|ssh_util.py:370|send_cmd_getresult_exception|INFO|Send command: [ -e /var/log/x2openEuler-upgrade/upgrade-plus/upgrade-plus-nohup.log ]
2024-10-23 03:27:30 Wed|1923:139927283037952|ssh_util.py:541|_exec_cmd|INFO|status code is: 0, command result is:
2024-10-23 03:27:30 Wed|1923:139927283037952|ssh_util.py:370|send_cmd_getresult_exception|INFO|Send command: chmod 600 /var/log/x2openEuler-upgrade/upgrade-plus/upgrade-plus-nohup.log
2024-10-23 03:27:31 Wed|1923:139927283037952|ssh_util.py:541|_exec_cmd|INFO|status code is: 0, command result is:
2024-10-23 03:27:31 Wed|1923:139927283037952|ssh_util.py:741|scp_upload|INFO|Start upload local file(s):/opt/x2openEuler/nohup-scripts/0e8f9d5a-5fa6-41a8-853f-c6a19b3d46f7.sh to remote host:/opt/x2openEuler/nohup-scripts
2024-10-23 03:27:31 Wed|1923:139927283037952|ssh_util.py:370|send_cmd_getresult_exception|INFO|Send command: [ -e /opt/x2openEuler/nohup-scripts ]
2024-10-23 03:27:31 Wed|1923:139927283037952|ssh_util.py:541|_exec_cmd|INFO|status code is: 0, command result is:
2024-10-23 03:27:31 Wed|1923:139927283037952|ssh_util.py:832|run_scp_password_type|INFO|scp command: scp -r -P 22 -o PreferredAuthentications=password /opt/x2openEuler/nohup-scripts/0e8f9d5a-5fa6-41a8-853f-c6a19b3d46f7.sh root@[192.168.122.82]:/opt/x2openEuler/nohup-scripts
2024-10-23 03:27:32 Wed|1923:139927283037952|ssh_util.py:843|run_scp_password_type|INFO|scp continue,entering password… 1
2024-10-23 03:27:35 Wed|1923:139927283037952|ssh_util.py:683|deal_scp_password_send_password|INFO|scp finish
2024-10-23 03:27:35 Wed|1923:139927283037952|ssh_util.py:684|deal_scp_password_send_password|INFO| \r\n\r0e8f9d5a-5fa6-41a8-853f-c6a19b3d46f7.sh 0% 0 0.0KB/s --:-- ETA\r0e8f9d5a-5fa6-41a8-853f-c6a19b3d46f7.sh 100% 645 42.9KB/s 00:00 \r\n
2024-10-23 03:27:35 Wed|1923:139927283037952|ssh_util.py:370|send_cmd_getresult_exception|INFO|Send command: [ -e /opt/x2openEuler/nohup-scripts/0e8f9d5a-5fa6-41a8-853f-c6a19b3d46f7.sh ]
2024-10-23 03:27:35 Wed|1923:139927283037952|ssh_util.py:541|_exec_cmd|INFO|status code is: 0, command result is:
2024-10-23 03:27:35 Wed|1923:139927283037952|ssh_util.py:994|check_upload_success|INFO|scp upload success: /opt/x2openEuler/nohup-scripts/0e8f9d5a-5fa6-41a8-853f-c6a19b3d46f7.sh
2024-10-23 03:27:35 Wed|1923:139927283037952|ssh_util.py:1144|send_nohup_command|INFO|Execute the nohup command for the 1th time. Nohup cmd: nohup sh /opt/x2openEuler/nohup-scripts/0e8f9d5a-5fa6-41a8-853f-c6a19b3d46f7.sh >/var/log/x2openEuler-upgrade/upgrade-plus/upgrade-plus-nohup.log 2>&1 &
2024-10-23 03:27:35 Wed|1923:139927283037952|ssh_util.py:1148|send_nohup_command|INFO|Execution Completed: ps -ef|grep 0e8f9d5a-5fa6-41a8-853f-c6a19b3d46f7.sh|grep -v grep|head -n 1 |awk ‘{print $2}’|xargs -I {} kill -SIGTERM – -{}
2024-10-23 03:27:38 Wed|1923:139927283037952|ssh_util.py:370|send_cmd_getresult_exception|INFO|Send command: head -c 5 /var/log/x2openEuler-upgrade/upgrade-plus/upgrade-plus-nohup.log
2024-10-23 03:27:39 Wed|1923:139927283037952|ssh_util.py:541|_exec_cmd|INFO|status code is: 0, command result is: nohup
2024-10-23 03:27:39 Wed|1923:139927283037952|ssh_util.py:1159|send_nohup_command|INFO|>>>Send nohup command finish
2024-10-23 03:27:39 Wed|1923:139927283037952|ssh_util.py:1170|get_remote_log_by_nohup|INFO|<<<Get the remote host log to the local path: : /var/log/x2openEuler-upgrade/upgrade-plus/upgrade-plus-nohup.log → /opt/x2openEuler-upgrade/3-192.168.122.82/upgrade/upgrade.log
2024-10-23 03:27:44 Wed|1923:139927283037952|ssh_util.py:521|_is_login_success|INFO|ssh connected succeeded, logging in as root
2024-10-23 03:27:44 Wed|1923:139927283037952|ssh_util.py:445|can_switch_to_root|INFO|already root user, no need to change!
2024-10-23 04:06:19 Wed|1923:139927283037952|ssh_util.py:521|_is_login_success|INFO|ssh connected succeeded, logging in as root
2024-10-23 04:06:19 Wed|1923:139927283037952|ssh_util.py:1061|get_pid_from_log|INFO|pid of the process: 5322
2024-10-23 04:06:19 Wed|1923:139927283037952|ssh_util.py:370|send_cmd_getresult_exception|INFO|Send command: ps 5322
2024-10-23 04:06:21 Wed|1923:139927283037952|ssh_util.py:541|_exec_cmd|INFO|status code is: 0, command result is: PID TTY STAT TIME COMMAND\r\n 5322 ? S 0:00 sh /opt/x2openEuler/nohup-scripts/0e8f9d5a-5fa6-41a8-853f-c6a19b3d46f7.sh
2024-10-23 04:06:21 Wed|1923:139927283037952|ssh_util.py:1192|get_remote_log_by_nohup|INFO|The process did not end, retry 1 time connection and get remote node log.
2024-10-23 04:06:34 Wed|1923:139927283037952|ssh_util.py:521|_is_login_success|INFO|ssh connected succeeded, logging in as root
2024-10-23 04:06:34 Wed|1923:139927283037952|ssh_util.py:445|can_switch_to_root|INFO|already root user, no need to change!
2024-10-23 04:16:39 Wed|1923:139927283037952|ssh_util.py:521|_is_login_success|INFO|ssh connected succeeded, logging in as root
2024-10-23 04:16:39 Wed|1923:139927283037952|ssh_util.py:1061|get_pid_from_log|INFO|pid of the process: 5322
2024-10-23 04:16:39 Wed|1923:139927283037952|ssh_util.py:370|send_cmd_getresult_exception|INFO|Send command: ps 5322
2024-10-23 04:16:40 Wed|1923:139927283037952|ssh_util.py:541|_exec_cmd|INFO|status code is: 0, command result is: PID TTY STAT TIME COMMAND\r\n 5322 ? S 0:00 sh /opt/x2openEuler/nohup-scripts/0e8f9d5a-5fa6-41a8-853f-c6a19b3d46f7.sh
2024-10-23 04:16:40 Wed|1923:139927283037952|ssh_util.py:1192|get_remote_log_by_nohup|INFO|The process did not end, retry 2 time connection and get remote node log.
2024-10-23 04:16:53 Wed|1923:139927283037952|ssh_util.py:521|_is_login_success|INFO|ssh connected succeeded, logging in as root
2024-10-23 04:16:54 Wed|1923:139927283037952|ssh_util.py:445|can_switch_to_root|INFO|already root user, no need to change!
2024-10-23 04:27:00 Wed|1923:139927283037952|ssh_util.py:521|_is_login_success|INFO|ssh connected succeeded, logging in as root
2024-10-23 04:27:00 Wed|1923:139927283037952|ssh_util.py:1061|get_pid_from_log|INFO|pid of the process: 5322
2024-10-23 04:27:00 Wed|1923:139927283037952|ssh_util.py:370|send_cmd_getresult_exception|INFO|Send command: ps 5322
2024-10-23 04:27:01 Wed|1923:139927283037952|ssh_util.py:541|_exec_cmd|INFO|status code is: 0, command result is: PID TTY STAT TIME COMMAND\r\n 5322 ? S 0:00 sh /opt/x2openEuler/nohup-scripts/0e8f9d5a-5fa6-41a8-853f-c6a19b3d46f7.sh
2024-10-23 04:27:01 Wed|1923:139927283037952|ssh_util.py:1192|get_remote_log_by_nohup|INFO|The process did not end, retry 3 time connection and get remote node log.
2024-10-23 04:27:15 Wed|1923:139927283037952|ssh_util.py:521|_is_login_success|INFO|ssh connected succeeded, logging in as root
2024-10-23 04:27:16 Wed|1923:139927283037952|ssh_util.py:445|can_switch_to_root|INFO|already root user, no need to change!
2024-10-23 04:37:22 Wed|1923:139927283037952|ssh_util.py:521|_is_login_success|INFO|ssh connected succeeded, logging in as root
2024-10-23 04:37:22 Wed|1923:139927283037952|ssh_util.py:1061|get_pid_from_log|INFO|pid of the process: 5322
2024-10-23 04:37:22 Wed|1923:139927283037952|ssh_util.py:370|send_cmd_getresult_exception|INFO|Send command: ps 5322
2024-10-23 04:37:24 Wed|1923:139927283037952|ssh_util.py:541|_exec_cmd|INFO|status code is: 0, command result is: PID TTY STAT TIME COMMAND\r\n 5322 ? S 0:00 sh /opt/x2openEuler/nohup-scripts/0e8f9d5a-5fa6-41a8-853f-c6a19b3d46f7.sh
2024-10-23 04:37:24 Wed|1923:139927283037952|ssh_util.py:1192|get_remote_log_by_nohup|INFO|The process did not end, retry 4 time connection and get remote node log.
2024-10-23 04:37:37 Wed|1923:139927283037952|ssh_util.py:521|_is_login_success|INFO|ssh connected succeeded, logging in as root
2024-10-23 04:37:37 Wed|1923:139927283037952|ssh_util.py:445|can_switch_to_root|INFO|already root user, no need to change!
2024-10-23 04:47:43 Wed|1923:139927283037952|ssh_util.py:521|_is_login_success|INFO|ssh connected succeeded, logging in as root
2024-10-23 04:47:43 Wed|1923:139927283037952|ssh_util.py:1061|get_pid_from_log|INFO|pid of the process: 5322
2024-10-23 04:47:43 Wed|1923:139927283037952|ssh_util.py:370|send_cmd_getresult_exception|INFO|Send command: ps 5322
2024-10-23 04:47:45 Wed|1923:139927283037952|ssh_util.py:541|_exec_cmd|INFO|status code is: 0, command result is: PID TTY STAT TIME COMMAND\r\n 5322 ? S 0:00 sh /opt/x2openEuler/nohup-scripts/0e8f9d5a-5fa6-41a8-853f-c6a19b3d46f7.sh
2024-10-23 04:47:45 Wed|1923:139927283037952|ssh_util.py:1188|get_remote_log_by_nohup|ERROR|retry connection failed 5 times!
2024-10-23 04:52:17 Wed|1808:139926858172160|ssh_util.py:521|_is_login_success|INFO|ssh connected succeeded, logging in as root
2024-10-23 04:52:17 Wed|1808:139926858172160|ssh_util.py:370|send_cmd_getresult_exception|INFO|Send command: bash /usr/lib/x2openEuler-upgrade/src/utils/log/get_log.sh
2024-10-23 04:52:22 Wed|1808:139926858172160|ssh_util.py:386|send_cmd_getresult|ERROR|exec cmd failed: bash /usr/lib/x2openEuler-upgrade/src/utils/log/get_log.sh, error: sending result=$(bash /usr/lib/x2openEuler-upgrade/src/utils/log/get_log.sh); echo “$result##reluEnepo##$?” timeout,SSH Timeout exception.
2024-10-29 03:03:25 Tue|1816:139631400056576|ssh_util.py:521|_is_login_success|INFO|ssh connected succeeded, logging in as root
2024-10-29 03:03:25 Tue|1816:139631400056576|ssh_util.py:370|send_cmd_getresult_exception|INFO|Send command: bash /usr/lib/x2openEuler-upgrade/src/utils/log/get_log.sh
2024-10-29 03:03:30 Tue|1816:139631400056576|ssh_util.py:386|send_cmd_getresult|ERROR|exec cmd failed: bash /usr/lib/x2openEuler-upgrade/src/utils/log/get_log.sh, error: sending result=$(bash /usr/lib/x2openEuler-upgrade/src/utils/log/get_log.sh); echo “$result##reluEnepo##$?” timeout,SSH Timeout exception.

menghaomail · 2024 年11 月 10 日 08:33

请教下，楼主解决了吗？我也遇到了这个问题，而且重试提示操作失败，无法重试。