Last Friday, we had a new patch cycle scheduled to upgrade the Oracle Private Cloud Appliance X9-2 from Patch level M3.9 to M3.10 (3.0.2-b1325160). For the first time, we encountered zero issues! All patching steps, including ZFSSA, compute nodes, platform, and OCI images, ran smoothly. For this patch, 37635338: PCA Release 3.0.2-b1325160 Patch (also known as M3.10), the following components were affected according to the ISO and patch readme:
- ZFSSA
- Compute Node
- Host (Management Node)
- Vault
- Kubernetes
- Platform
- OCI Images
- Firmwares – ILOMs(X9 and E5)
Release Notes
Oracle provides release notes that highlight new features for each version, but they lack detailed information. Here is an overview of the latest features from the documentation: Latest Features
Improved Upgrade and Patching Procedures
Both the Upgrade Guide and Patching Guide have been restructured to make it easier and clearer to follow the appropriate instructions based on active software version and target version. In the main instructional sections, it is assumed that the system is already at a minimum required version. For systems on earlier software versions, the procedures to get to the minimum required version are provided in a separate chapter. See Upgrading from Earlier Software Versions in the “Oracle Private Cloud Appliance Upgrade Guide”.
New Procedure for Grafana/Prometheus
The most surprising step was the platform upgrade. It now automatically creates a backup for Grafana and Prometheus before applying the patch and restores the data afterward. This process follows the steps we previously had to do manually in older versions, which was always a hassle.
- [PCA 3.x] Sauron Upgrade or Redeploy Causes Custom Resources to Disappear (Doc ID 3003206.1
- [PCA 3.x] How to Backup/Restore Prometheus Data (Doc ID 3021643.1)
Here is an excerpt from the logfile for the PCA-ADMIN step, which involves a full management node update and platform stack upgrade for Grafana:
[2025-03-28 14:29:25 41056] INFO (task_deploy_sauron:434) Beginning grafana.db backup.
[2025-03-28 14:29:25 41056] INFO (task_deploy_sauron:481) Running sauron-grafana pod retrieved: sauron-sauron-grafana-9755885cf-xh4lw
[2025-03-28 14:29:29 41056] INFO (task_deploy_sauron:496) Finished grafana.db backup.
[2025-03-28 14:29:29 41056] INFO (task_deploy_sauron:508) Finished grafana.db backup.
[2025-03-28 14:29:29 41056] INFO (task_deploy_sauron:513) Beginning grafana config and proxy backup.
[2025-03-28 14:29:29 41056] INFO (task_deploy_sauron:574) Finished grafana config and proxy backup.Here is the log content with the Prometheus backup steps:
PLAY [all] *********************************************************************
TASK [Gathering Facts] *********************************************************
Friday 28 March 2025 14:30:38 +0000 (0:00:00.024) 0:00:00.024 **********
ok: [localhost]
TASK [Create mount directory] **************************************************
Friday 28 March 2025 14:30:40 +0000 (0:00:01.672) 0:00:01.697 **********
changed: [localhost]
TASK [Mount NFS share] *********************************************************
Friday 28 March 2025 14:30:40 +0000 (0:00:00.574) 0:00:02.271 **********
changed: [localhost]
TASK [Delete the backup directory if it exists] ********************************
Friday 28 March 2025 14:30:41 +0000 (0:00:00.602) 0:00:02.874 **********
ok: [localhost]
TASK [Create directory to store backup] ****************************************
Friday 28 March 2025 14:30:41 +0000 (0:00:00.449) 0:00:03.324 **********
changed: [localhost]
TASK [Copy /nfs/shared_storage/prometheus_backup/mount_dir/snapshots/20250328T142929Z-40c79e0738dd6260/ to /nfs/shared_storage/prometheus_backup/backup/] ***
Friday 28 March 2025 14:30:42 +0000 (0:00:00.675) 0:00:03.999 **********
changed: [localhost]
TASK [Set permission for saved backup] *****************************************
Friday 28 March 2025 15:15:25 +0000 (0:44:42.726) 0:44:46.725 **********
changed: [localhost]
TASK [Unmount NFS share] *******************************************************
Friday 28 March 2025 15:15:25 +0000 (0:00:00.341) 0:44:47.067 **********
changed: [localhost]
PLAY RECAP *********************************************************************
localhost : ok=8 changed=6 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Friday 28 March 2025 15:15:51 +0000 (0:00:26.295) 0:45:13.362 **********
===============================================================================
Copy /nfs/shared_storage/prometheus_backup/mount_dir/snapshots/20250328T142929Z-40c79e0738dd6260/ to /nfs/shared_storage/prometheus_backup/backup/ 2682.73s
Unmount NFS share ------------------------------------------------------ 26.30s
Gathering Facts --------------------------------------------------------- 1.67s
Create directory to store backup ---------------------------------------- 0.68s
Mount NFS share --------------------------------------------------------- 0.60s
Create mount directory -------------------------------------------------- 0.57s
Delete the backup directory if it exists -------------------------------- 0.45s
Patch Plan
In total, it took approximately 12 hours to complete all the steps outlined in the patch plan for a PCA X-9 with 3 Compute Nodes. The attached screenshot shows the patch plan with all steps completed. However, not all components required patching. For instance, no new version was released for the switch software.
Here is the complete list of upgrade steps. A new step called “setup” has been added at the beginning. This step involves preparing all packages and is initiated by the pre-Upgrade command.
This is the patch time per component, but don’t trust the prepare step of more than 500 minutes, it took in real about 1hr.
Quick Summary
This was nice, now we are looking forward for the second PCA we run. Keep fingers crossed!