I’m currently in the process of updating an older Oracle Private Cloud Appliance X5-2 combined with a ZFS Storage Appliance ZS7-2 using the latest available patch for X5-2, version 2.4.3.2. As a precursor to the upgrade, a verification job needs to be run using Oracle’s provided tool, pca-upgrader. When initiated with the -V parameter, this tool checks the PCA’s health status and assesses all components for their readiness to be upgraded. It’s imperative that all these checks pass before initiating the actual upgrade. The first major step in this extensive procedure involves upgrading the Management Nodes.
Related Links:
- PCA Patching & Maintenance (Doc ID 1628815.2)
- [PCA 2.4.3.2] Upgrade Guide (Doc ID 2947058.1)
- Release Notes for Release 2.4.3.2
- Updating Oracle Private Cloud Appliance (Admin Guide)
PCA Management Node Pre-Upgrade Checks – 1st Run – FAILED
The initial attempt concluded with a FAILED status – refer to the subsequent steps. The task was initiated from the primary Management Node, specifically ovcamn05r1.
-- check for master node [root@ovcamn05r1 ~]# pca-check-master NODE: 192.168.4.3 MASTER: True -- run verification [root@ovcamn05r1 ~]# pca_upgrader -V -t management -c ovcamn06r1 -g 2.4.3.2 -l file:///nfs/shared_storage/pca-patches/ovca-2.4.3.2-b1368.iso.zip PCA Rack Type: PCA X3/X4/X5 BASE. Please refer to log file /nfs/shared_storage/pca_upgrader/log/pca_upgrader_2023_10_10-15.02.36_ovcamn06r1_verify.log for more details. Beginning PCA Management Node Pre-Upgrade Checks... Validate the Image Provided 1/46 Rack Type Check 2/46 OVMM Model DB Check 3/46 ONF Check 4/46 Internal ZFSSA Available Space Check 5/46 Shared Storage Owner Check 6/46 MN Disk and Shared Storage Space Check 7/46 PCA services Check 8/46 Reprovisioned/Deprovisioned Compute Nodes Check 9/46 Multiple Tenant Group Check 10/46 IB Symbol Errors Check 11/46 MySQL Desychronized Password Check 12/46 OVMM Default Network Check 13/46 MySQL Password Check 14/46 No Upstream Proxies Check 15/46 Ethernet Cards Check 16/46 Java Home Check 17/46 Management Node Uptime Check 18/46 MOTD Check 19/46 ZFSSA Resilvering Jobs Check 20/46 Shared Storage Mounted Check 21/46 Checking PCA foundational services 22/46 Default nameserver check 23/46 Server Update Configuration Check 24/46 YUM Proxy Check 25/46 RPMDB Check 26/46 Number of FC Paths Check 27/46 Available Space Check 28/46 Number of iSCSI Paths Check 29/46 OVM Manager Max Cache Size Check 30/46 Network Configuration Check 31/46 YUM Repo Check 32/46 Server Name Check 33/46 Multipath Customizations Check 34/46 Interfaces Addressing Type Check 35/46 Keystore File Location Check 36/46 OVM Warning/Error Check 37/46 NTP Xsigo Check 38/46 Xsigo Config Check 39/46 NTP Server Check 40/46 OS Check 41/46 Oracle VM Minimum Version Check 42/46 ILOM Version Check 43/46 Hardware Faults Check 44/46 OSA Disabled Check 45/46 Password Check 46/46 PCA Management Node Pre-Upgrade Checks completed after 1 minutes Beginning PCA Health Checks... Check Management Nodes Are Running 1/22 Check PCA DBs Exist 2/22 PCA Config File 3/22 Check PCA Version 4/22 Check Shares Mounted on Management Nodes 5/22 Management Nodes Have IPv6 Disabled 6/22 Repositories Defined in Oracle VM Manager 7/22 PCA Services 8/22 Oracle VM Manager Default Networks 9/22 Network Interfaces on Compute Nodes 10/22 Oracle VM Manager Settings 11/22 XSIGO PVI HA Feature 12/22 Manangement Node XSIGO Network Interface 13/22 Management Node XSIGO vHBA 14/22 All Compute Nodes Running 15/22 Check PCA OVS Package Version 16/22 Test for ovs-agent Service on Compute Nodes 17/22 IPoIB Configuration on Compute Nodes 18/22 Test for Shares Mounted on Compute Nodes 19/22 Compute Node XSIGO Network Interface 20/22 Check Compute Node's Active Network Interfaces 21/22 Compute Node XSIGO vHBA 22/22 PCA Health Checks completed after 1 minutes --------------------------------------------------------------------------- PCA Management Node Pre-Upgrade Checks Failed --------------------------------------------------------------------------- Validate the Image Provided Passed Rack Type Check Passed OVMM Model DB Check Passed ONF Check Passed Internal ZFSSA Available Space Check Passed Shared Storage Owner Check Passed MN Disk and Shared Storage Space Check Passed PCA services Check Passed Reprovisioned/Deprovisioned Compute Nodes Check Passed Multiple Tenant Group Check Passed IB Symbol Errors Check Passed MySQL Desychronized Password Check Passed OVMM Default Network Check Passed MySQL Password Check Passed No Upstream Proxies Check Passed Ethernet Cards Check Passed Java Home Check Passed Management Node Uptime Check Failed MOTD Check Passed ZFSSA Resilvering Jobs Check Passed Shared Storage Mounted Check Passed Checking PCA foundational services Passed Default nameserver check Passed Server Update Configuration Check Passed YUM Proxy Check Passed RPMDB Check Passed Number of FC Paths Check Passed Available Space Check Passed Number of iSCSI Paths Check Passed OVM Manager Max Cache Size Check Passed Network Configuration Check Passed YUM Repo Check Passed Server Name Check Passed Multipath Customizations Check Passed Interfaces Addressing Type Check Passed Keystore File Location Check Passed OVM Warning/Error Check Passed NTP Xsigo Check Passed Xsigo Config Check Passed NTP Server Check Passed OS Check Passed Oracle VM Minimum Version Check Passed ILOM Version Check Passed Hardware Faults Check Passed OSA Disabled Check Passed Password Check Passed --------------------------------------------------------------------------- PCA Health Checks Passed --------------------------------------------------------------------------- Check Management Nodes Are Running Passed Check PCA DBs Exist Passed PCA Config File Passed Check PCA Version Passed Check Shares Mounted on Management Nodes Passed Management Nodes Have IPv6 Disabled Passed Repositories Defined in Oracle VM Manager Passed PCA Services Passed Oracle VM Manager Default Networks Passed Network Interfaces on Compute Nodes Passed Oracle VM Manager Settings Passed XSIGO PVI HA Feature Passed Manangement Node XSIGO Network Interface Passed Management Node XSIGO vHBA Passed All Compute Nodes Running Passed Check PCA OVS Package Version Passed Test for ovs-agent Service on Compute Nodes Passed IPoIB Configuration on Compute Nodes Passed Test for Shares Mounted on Compute Nodes Passed Compute Node XSIGO Network Interface Passed Check Compute Node's Active Network Interfaces Passed Compute Node XSIGO vHBA Passed --------------------------------------------------------------------------- Overall Status Failed --------------------------------------------------------------------------- PCA Management Node Pre-Upgrade Checks Failed PCA Health Checks Passed
Management Node Uptime Check: Failed
To determine the reason for the check failure, let’s examine the logfile located at /nfs/shared_storage/pca_upgrader/log/pca_upgrader_2023_10_10-15.02.36_ovcamn06r1_verify.log.
[2023-10-10 15:03:08 1017357] ERROR (precheck:154) [Management Node Uptime Check (Verifying that the uptimes of the management nodes do not exceed the recommended limits for upgrading)] Failed The check failed on the following management nodes: ['ovcamn05r1', 'ovcamn06r1']. ovcamn05r1 has been up for 391 days. This exceeds the max uptime of 100 days prior to an upgrade.ovcamn06r1 has been up for 391 days. This exceeds the max uptime of 100 days prior to an upgrade.
100 Days Uptime exceeded
The upgrade process requires that the nodes have no uptime for more than 100 days prior to executing the upgrade. Let’s reboot both Management Nodes and attempt the upgrade again. This action should address the problem.
- reboot existing master node ovcamn05r1
[root@ovcamn05r1 ~]# reboot
Broadcast message from root@ovcamn06r1
(/dev/pts/0) at 13:21 ...
The system is going down for reboot NOW!
-- check new master ovcamn06r1
[root@ovcamn06r1 ~]# pca-check-master
NODE: 192.168.4.4 MASTER: True
-- wait some minutes until OVM GUI is reachable
-- reboot ovcamn06r1 to change back master
[root@ovcamn06r1 ~]# reboot
Broadcast message from root@ovcamn06r1
(/dev/pts/0) at 13:21 ...
The system is going down for reboot NOW!
-- check ovcamn05r1 is master again
[root@ovcamn05r1 ~]# pca-check-master
NODE: 192.168.4.3 MASTER: True
-- wait some minutes until OVM GUI is reachable
PCA Management Node Pre-Upgrade Checks – 2nd Run – PASSED
On the second attempt, the task was initiated from the primary Management Node, ovcamn05r1.
[root@ovcamn05r1 ~]# pca_upgrader -V -t management -c ovcamn06r1 -g 2.4.3.2 -l file:///nfs/shared_storage/pca-patches/ovca-2.4.3.2-b1368.iso.zip PCA Rack Type: PCA X3/X4/X5 BASE. Please refer to log file /nfs/shared_storage/pca_upgrader/log/pca_upgrader_2023_10_11-13.57.45_ovcamn06r1_verify.log for more details. Beginning PCA Management Node Pre-Upgrade Checks... Validate the Image Provided 1/46 Rack Type Check 2/46 OVMM Model DB Check 3/46 ONF Check 4/46 Internal ZFSSA Available Space Check 5/46 Shared Storage Owner Check 6/46 MN Disk and Shared Storage Space Check 7/46 PCA services Check 8/46 MySQL Desychronized Password Check 9/46 Reprovisioned/Deprovisioned Compute Nodes Check 10/46 Multiple Tenant Group Check 11/46 ZFSSA Resilvering Jobs Check 12/46 MySQL Password Check 13/46 Java Home Check 14/46 No Upstream Proxies Check 15/46 Ethernet Cards Check 16/46 Default nameserver check 17/46 Management Node Uptime Check 18/46 IB Symbol Errors Check 19/46 Checking PCA foundational services 20/46 YUM Proxy Check 21/46 Shared Storage Mounted Check 22/46 MOTD Check 23/46 OVMM Default Network Check 24/46 Server Update Configuration Check 25/46 Number of FC Paths Check 26/46 OVM Manager Max Cache Size Check 27/46 Available Space Check 28/46 Number of iSCSI Paths Check 29/46 OVM Warning/Error Check 30/46 Multipath Customizations Check 31/46 Network Configuration Check 32/46 Server Name Check 33/46 YUM Repo Check 34/46 RPMDB Check 35/46 Interfaces Addressing Type Check 36/46 NTP Xsigo Check 37/46 Keystore File Location Check 38/46 Xsigo Config Check 39/46 NTP Server Check 40/46 Oracle VM Minimum Version Check 41/46 OS Check 42/46 ILOM Version Check 43/46 Hardware Faults Check 44/46 Password Check 45/46 OSA Disabled Check 46/46 PCA Management Node Pre-Upgrade Checks completed after 1 minutes Beginning PCA Health Checks... Check Management Nodes Are Running 1/22 Check PCA DBs Exist 2/22 PCA Config File 3/22 Check PCA Version 4/22 Check Shares Mounted on Management Nodes 5/22 Management Nodes Have IPv6 Disabled 6/22 PCA Services 7/22 Repositories Defined in Oracle VM Manager 8/22 Oracle VM Manager Default Networks 9/22 Network Interfaces on Compute Nodes 10/22 Oracle VM Manager Settings 11/22 XSIGO PVI HA Feature 12/22 Manangement Node XSIGO Network Interface 13/22 Management Node XSIGO vHBA 14/22 All Compute Nodes Running 15/22 Test for ovs-agent Service on Compute Nodes 16/22 Check PCA OVS Package Version 17/22 IPoIB Configuration on Compute Nodes 18/22 Test for Shares Mounted on Compute Nodes 19/22 Compute Node XSIGO vHBA 20/22 Compute Node XSIGO Network Interface 21/22 Check Compute Node's Active Network Interfaces 22/22 PCA Health Checks completed after 1 minutes --------------------------------------------------------------------------- PCA Management Node Pre-Upgrade Checks Passed --------------------------------------------------------------------------- Validate the Image Provided Passed Rack Type Check Passed OVMM Model DB Check Passed ONF Check Passed Internal ZFSSA Available Space Check Passed Shared Storage Owner Check Passed MN Disk and Shared Storage Space Check Passed PCA services Check Passed MySQL Desychronized Password Check Passed Reprovisioned/Deprovisioned Compute Nodes Check Passed Multiple Tenant Group Check Passed ZFSSA Resilvering Jobs Check Passed MySQL Password Check Passed Java Home Check Passed No Upstream Proxies Check Passed Ethernet Cards Check Passed Default nameserver check Passed Management Node Uptime Check Passed IB Symbol Errors Check Passed Checking PCA foundational services Passed YUM Proxy Check Passed Shared Storage Mounted Check Passed MOTD Check Passed OVMM Default Network Check Passed Server Update Configuration Check Passed Number of FC Paths Check Passed OVM Manager Max Cache Size Check Passed Available Space Check Passed Number of iSCSI Paths Check Passed OVM Warning/Error Check Passed Multipath Customizations Check Passed Network Configuration Check Passed Server Name Check Passed YUM Repo Check Passed RPMDB Check Passed Interfaces Addressing Type Check Passed NTP Xsigo Check Passed Keystore File Location Check Passed Xsigo Config Check Passed NTP Server Check Passed Oracle VM Minimum Version Check Passed OS Check Passed ILOM Version Check Passed Hardware Faults Check Passed Password Check Passed OSA Disabled Check Passed --------------------------------------------------------------------------- PCA Health Checks Passed --------------------------------------------------------------------------- Check Management Nodes Are Running Passed Check PCA DBs Exist Passed PCA Config File Passed Check PCA Version Passed Check Shares Mounted on Management Nodes Passed Management Nodes Have IPv6 Disabled Passed PCA Services Passed Repositories Defined in Oracle VM Manager Passed Oracle VM Manager Default Networks Passed Network Interfaces on Compute Nodes Passed Oracle VM Manager Settings Passed XSIGO PVI HA Feature Passed Manangement Node XSIGO Network Interface Passed Management Node XSIGO vHBA Passed All Compute Nodes Running Passed Test for ovs-agent Service on Compute Nodes Passed Check PCA OVS Package Version Passed IPoIB Configuration on Compute Nodes Passed Test for Shares Mounted on Compute Nodes Passed Compute Node XSIGO vHBA Passed Compute Node XSIGO Network Interface Passed Check Compute Node's Active Network Interfaces Passed --------------------------------------------------------------------------- Overall Status Passed --------------------------------------------------------------------------- PCA Management Node Pre-Upgrade Checks Passed PCA Health Checks Passed Please refer to log file /nfs/shared_storage/pca_upgrader/log/pca_upgrader_2023_10_11-13.57.45_ovcamn06r1_verify.log for more details.
I’m prepared to move forward with the Management Node updates. Today’s takeaway: Uptimes exceeding 100 days aren’t conducive to upgrades.