我用x2工具20240930版本,使用方式二迁移centos7.9的主机到2203LTS-SP4版本,迁移十台主机前端web界面总是卡在29%就不动了。之前迁移了20台也是卡在了前端29%。无奈只能每台主机手动去重启,重启所有主机之后,每台主机都已经完成了迁移升级,相关版本也是2203LTS-SP4版本,服务都是正常的,但是前端页面还是卡在了29%。之前测试的时候单独升级一台不会卡进度条。还有就是在环境检查的时候,同时检查十台主机也会报错,报错信息有SSH连接错误,无法连接到SSH或者SCP连接超时,无法在待升级节点安装x2升级工具。
升级的服务端主机系统是20.03LTSSP1版本,x2版本是20240930版本,4核CPU、8G内存。
1、x2server的配置给高点
2、日志可以查看backend下面的日志
前端还在更新的时候,使用TOP命令查看
发现5 到6个gunicorn进程占用的CPU非常高,都是97%左右,此时前端日志还在更新。
查看了日志tail -f /usr/local/x2openEuler/portal/logs/backend/ssh_connect.log
[root@tmp-openEuler-app-01 backend]# tail -f /usr/local/x2openEuler/portal/logs/backend/ssh_connect.log
2024-11-05 11:32:35 Tue|188126:140345757128256|ssh_util.py:843|run_scp_password_type|INFO|scp continue,entering password… 1
2024-11-05 11:33:26 Tue|188119:140345748735552|ssh_util.py:521|_is_login_success|INFO|ssh connected succeeded, logging in as root
2024-11-05 11:33:26 Tue|188119:140345748735552|ssh_util.py:782|scp_download|INFO|start download remote host file(s):/var/log/x2openEuler-upgrade/upgrade-plus/upgrade-plus-nohup.log, save to local path:/opt/x2openEuler-upgrade/9-10.99.60.36/upgrade/upgrade.log
2024-11-05 11:33:26 Tue|188119:140345748735552|ssh_util.py:370|send_cmd_getresult_exception|INFO|Send command: [ -e /var/log/x2openEuler-upgrade/upgrade-plus/upgrade-plus-nohup.log ]
2024-11-05 11:33:26 Tue|188119:140345748735552|ssh_util.py:541|_exec_cmd|INFO|status code is: 0, command result is:
2024-11-05 11:33:26 Tue|188119:140345748735552|ssh_util.py:832|run_scp_password_type|INFO|scp command: scp -r -P 22 -o PreferredAuthentications=password root@[10.99.60.36]:/var/log/x2openEuler-upgrade/upgrade-plus/upgrade-plus-nohup.log /opt/x2openEuler-upgrade/9-10.99.60.36/upgrade/upgrade.log
2024-11-05 11:33:27 Tue|188119:140345748735552|ssh_util.py:843|run_scp_password_type|INFO|scp continue,entering password… 1
2024-11-05 11:34:28 Tue|188120:140345740342848|ssh_util.py:671|deal_scp_password_send_password|ERROR|scp send package failed,connect timeout
2024-11-05 11:34:35 Tue|188126:140345757128256|ssh_util.py:671|deal_scp_password_send_password|ERROR|scp send package failed,connect timeout
2024-11-05 11:35:27 Tue|188119:140345748735552|ssh_util.py:671|deal_scp_password_send_password|ERROR|scp send package failed,connect timeout
^C
[root@tmp-openEuler-app-01 backend]# tail -f /usr/local/x2openEuler/portal/logs/backend/ssh_connect.log
2024-11-05 11:32:35 Tue|188126:140345757128256|ssh_util.py:843|run_scp_password_type|INFO|scp continue,entering password… 1
2024-11-05 11:33:26 Tue|188119:140345748735552|ssh_util.py:521|_is_login_success|INFO|ssh connected succeeded, logging in as root
2024-11-05 11:33:26 Tue|188119:140345748735552|ssh_util.py:782|scp_download|INFO|start download remote host file(s):/var/log/x2openEuler-upgrade/upgrade-plus/upgrade-plus-nohup.log, save to local path:/opt/x2openEuler-upgrade/9-10.99.60.36/upgrade/upgrade.log
2024-11-05 11:33:26 Tue|188119:140345748735552|ssh_util.py:370|send_cmd_getresult_exception|INFO|Send command: [ -e /var/log/x2openEuler-upgrade/upgrade-plus/upgrade-plus-nohup.log ]
2024-11-05 11:33:26 Tue|188119:140345748735552|ssh_util.py:541|_exec_cmd|INFO|status code is: 0, command result is:
2024-11-05 11:33:26 Tue|188119:140345748735552|ssh_util.py:832|run_scp_password_type|INFO|scp command: scp -r -P 22 -o PreferredAuthentications=password root@[10.99.60.36]:/var/log/x2openEuler-upgrade/upgrade-plus/upgrade-plus-nohup.log /opt/x2openEuler-upgrade/9-10.99.60.36/upgrade/upgrade.log
2024-11-05 11:33:27 Tue|188119:140345748735552|ssh_util.py:843|run_scp_password_type|INFO|scp continue,entering password… 1
2024-11-05 11:34:28 Tue|188120:140345740342848|ssh_util.py:671|deal_scp_password_send_password|ERROR|scp send package failed,connect timeout
2024-11-05 11:34:35 Tue|188126:140345757128256|ssh_util.py:671|deal_scp_password_send_password|ERROR|scp send package failed,connect timeout
2024-11-05 11:35:27 Tue|188119:140345748735552|ssh_util.py:671|deal_scp_password_send_password|ERROR|scp send package failed,connect timeout
当出现SCP 失败的时候,top命令查看进程,此时CPU消耗就没有那么大了,gunicorn的进程就都消失了,只留下了进程,消耗很小。
并且前端卡住的位置都是当所有升级节点下载完安装包之后,然后节点开始安装安装大量的安装包时。
此时ssh能连上这台服务器吗?
SSH可以连接到升级节点上面,登录升级节点,可以查看升级日志。
tail -n 10 /var/log/x2openEuler-upgrade/upgrade-plus/upgrade-plus-nohup.log
可以看到升级节点正常升级到了2203LTSP4版本,手动重启升级节点,可以正常切换到2203LTSSP4版本,但是前端界面还是会卡在进度29%
显示正在安装软件
我也是前端卡住,在84%,去待升级节点查看tail -n 10 /var/log/x2openEuler-upgrade/upgrade-plus/upgrade-plus-nohup.log显示的([ INFO ] - [upgrade] - [rebuild_boot] - rebuild_boot execution completed
[ INFO ] - [upgrade] - [rebuild_boot] - rebuild boot done
[ INFO ] - [upgrade] - [rebuild_boot] - please reboot as soon as possible
[ INFO ] - [upgrade_entrance] - upgrade openEuler repo success
0
##THE END##),就直接手动reboot了,然后等了很长时间都是失联状态,由于太晚了就睡着了,早上起来再尝试连接就连上了,查看了cat /etc/os-release显示
NAME=“openEuler”
VERSION=“24.03 (LTS)”
ID=“openEuler”
VERSION_ID=“24.03”
PRETTY_NAME=“openEuler 24.03 (LTS)”
ANSI_COLOR=“0;31”,算是从centos7.9.2009升级成功到openEuler 24.03 (LTS)