The roll-out of the newest Oracle Enterprise Manager 13 Release 4 is a few days old, about time to try out the upgrade process in my ESXi lab environment.
First: This blog post about the OEM upgrade process is based on my own experience.
Software
https://www.oracle.com/enterprise-manager/downloads/cloud-control-downloads.html
Documents
The Environment
- Oracle Enterprise Manager 13.3 running on Oracle Linux 7.6 – OEM Patchlevel January 2020 – Non-HA
- Oracle Enterprise Edition Repository 19.6.0 Single Tenant Database
- Oracle Restart / ASM 19.6.0
- Additional Oracle Linux Server with 19.3.0 Container Databases
- All targets up and running
- My Oracle Support connected
- Software staged directory in /u01/app/oracle/stage/em13cr4 and permission changed to execute bin file (chmod u+x em13400_linux64.bin).
oracle@kestenholz:/u01/app/oracle/stage/em13cr4/ [oms1303] ll total 12429656 drwxr-xr-x 2 oracle oinstall 4096 Feb 2 12:13 . drwxr-xr-x 3 oracle oinstall 21 Feb 2 20:13 .. -rw-r--r-- 1 oracle oinstall 2109004737 Feb 2 12:11 em13400_linux64-2.zip -rw-r--r-- 1 oracle oinstall 2047749474 Feb 2 12:11 em13400_linux64-3.zip -rw-r--r-- 1 oracle oinstall 2117312528 Feb 2 12:12 em13400_linux64-4.zip -rw-r--r-- 1 oracle oinstall 808014516 Feb 2 12:12 em13400_linux64-5.zip -rw-r--r-- 1 oracle oinstall 2143125187 Feb 2 12:13 em13400_linux64-6.zip -rw-r--r-- 1 oracle oinstall 1879163226 Feb 2 12:14 em13400_linux64-7.zip -rwxr-x--- 1 oracle oinstall 1623577684 Feb 2 12:10 em13400_linux64.bin
Prerequisites
It’s important to verify the changed prerequisites – Chapter 3 – Prerequisites for Upgrading to Enterprise Manager Cloud Control 13c Release 4 – I had do disable all adaptive features in the repository pluggable database:
alter system set "_optimizer_nlj_hj_adaptive_join"= FALSE scope=both sid='*'; alter system set "_optimizer_strans_adaptive_pruning" = FALSE scope=both sid='*'; alter system set "_px_adaptive_dist_method" = OFF scope=both sid='*'; alter system set "_sql_plan_directive_mgmt_control" = 0 scope=both sid='*'; alter system set "_optimizer_dsdir_usage_control" = 0 scope=both sid='*'; alter system set "_optimizer_use_feedback" = FALSE scope=both sid='*'; alter system set "_optimizer_gather_feedback" = FALSE scope=both sid='*'; alter system set "_optimizer_performance_feedback" = OFF scope=both sid='*';
Restart the database after the changed settings.From my view this sentence here is wrong is the documentation (my database has version 19.6.0):
If your Management Repository is using Oracle Database 12.2 or higher, none of these parameters need to be set.
Why? If these parameters are not set, the installer refuses working.
EMKEY Copy Requirements
The Key was copied to the repository. It well be removed after the successful upgrade.
oracle@kestenholz: [oms1303] $ORACLE_HOME/bin/emctl config emkey -copy_to_repos Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure. After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".
Run Installer
Start the Oracle Universal Installer
The OMS is down now (emctl stop oms -all). Start the installer.
oracle@kestenholz:/u01/app/oracle/stage/em13cr4/ [oms1303] ./em13400_linux64.bin
My Oracle Support Details
I am a registered user in My Oracle Support, I get enough information every week… I don’t enable this checkbox.
Software Updates
At the moment, there are no updates available. Maybe in future there will be any patches available for auto apply during the installation/upgrade process.
Installation Type
Upgrade an existing Enterprise Manager system, this one is my existing installation.
Installation Details
Enter a new Middleware Home Location.
Database Connection Details
Enter the SYS and SYSMAN password of the running repository. DDMP (Deferred Data Migration) Jobs are enabled. According the documentation, these jobs are running in the background when the OEM is starting up during the upgrade process to convert old data to the new format. If you have a huge amount of data in your earlier release, then the upgrade can take longer. You can run this job – if disabled – later as Post Upgrade Tasks.
Warnings
I will change to SHA communication later.
The repository has 19.6.0, all required patches are included.
I confirm that the JVMD engine is stopped, this has happened by stopping the OMS.
I had to stop the agent which was running on the management server.
Fix Parameter Settings by the Installer
For my environment, I let the installer fix this settings.
Plug-in Upgrade
Here we see the already installed plugins, they will be upgraded too.
Select Plug-ins
I don’t ant to install additional plug-ins.
Extend WebLogic Server Domain
Enter the password for the weblogic user and define the OMS Instance Base Location.
Enterprise Manager Shared Location Details
My Oracle Enterprise Manager doesn’t run in a high availability setup, I don’t need any shared location.
Port Configuration Details
I use the settings from the existing installation.
Review
Let’s start the upgrade!
Repository Upgrade failed
The upgrade process fails at the step where the repository has to be upgraded.
View Log – ORA-01950
When I scroll up the provided log from the installer, I see this error message here:
The schema manager logfile located in the 13.4 subdirectory $ORACLE_HOME/sysman/log/schemamanager confirms this error.
oracle@kestenholz:/u01/app/oracle/product/em13cr4/sysman/log/schemamanager/m_020220_0947_PM/m_020220_0947_PM.PREUPGRADE/ [oms1303] grep 1950 * emschema.log.PREUPGRADE:SEVERE: ERROR:ORA-01950: no privileges on tablespace 'USERS' grep: RCU2020-02-02_21-47_829509113: Is a directory rcu.log:Sun Feb 2 21:47:41.519 CET 2020 ERROR assistants.rcu.Rcu: oracle.sysman.assistants.rcu.backend.action.AbstractAction::handleNonIgnorableError: Received Non-Ignorable Error: ORA-01950: no privileges on tablespace 'USERS' rcu.log:Sun Feb 2 21:47:41.520 CET 2020 ERROR assistants.rcu.Rcu: oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine::onException: SQLException: ORA-01950: no privileges on tablespace 'USERS' rcu.log:java.sql.SQLSyntaxErrorException: ORA-01950: no privileges on tablespace 'USERS' rcu.log:oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: java.sql.SQLSyntaxErrorException: ORA-01950: no privileges on tablespace 'USERS' rcu.log:Caused by: java.sql.SQLSyntaxErrorException: ORA-01950: no privileges on tablespace 'USERS' rcu.log:Caused by: oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: java.sql.SQLSyntaxErrorException: ORA-01950: no privileges on tablespace 'USERS' rcu.log:Caused by: java.sql.SQLSyntaxErrorException: ORA-01950: no privileges on tablespace 'USERS'
At this point, this is curious, because in 13.3, there were no objects of the SYSMAN in the USERS tablespace. All objects are located in OEM tablespaces with the MGMT prefix. Here is a list of objects in a repository where an Oracle Enterprise Manager 13c Release 3 is up and running.
SQL> select owner,segment_type,tablespace_name,count(*) from dba_segments where owner='SYSMAN' group by owner,segment_type,tablespace_name order by 1,2,3; OWNER SEGMENT_TYPE TABLESPACE_NAME COUNT(*) -------------------- ------------------ ------------------------------ ---------- SYSMAN INDEX MGMT_AD4J_TS 18 SYSMAN INDEX MGMT_ECM_DEPOT_TS 10 SYSMAN INDEX MGMT_TABLESPACE 1701 SYSMAN INDEX PARTITION MGMT_AD4J_TS 192 SYSMAN INDEX PARTITION MGMT_ECM_DEPOT_TS 15 SYSMAN INDEX PARTITION MGMT_TABLESPACE 1515 SYSMAN LOB PARTITION MGMT_ECM_DEPOT_TS 15 SYSMAN LOB PARTITION MGMT_TABLESPACE 203 SYSMAN LOBINDEX MGMT_AD4J_TS 1 SYSMAN LOBINDEX MGMT_ECM_DEPOT_TS 37 SYSMAN LOBINDEX MGMT_TABLESPACE 159 SYSMAN LOBSEGMENT MGMT_AD4J_TS 1 SYSMAN LOBSEGMENT MGMT_ECM_DEPOT_TS 37 SYSMAN LOBSEGMENT MGMT_TABLESPACE 159 SYSMAN NESTED TABLE MGMT_TABLESPACE 7 SYSMAN TABLE MGMT_AD4J_TS 13 SYSMAN TABLE MGMT_ECM_DEPOT_TS 11 SYSMAN TABLE MGMT_TABLESPACE 1053 SYSMAN TABLE PARTITION MGMT_AD4J_TS 192 SYSMAN TABLE PARTITION MGMT_TABLESPACE 622 20 rows selected.
The solution is simple. Just grant the permissions for the USERS tablespace to SYSMAN and retry the installation progress.But, don’t ask me why SYSMAN creates now objects in USERS…
SQL> ALTER USER sysman QUOTA UNLIMITED ON users; User altered.
After a while, you can see new created tables and indexes in tablespace USERS for user SYSMAN.
SQL> select owner,segment_type,tablespace_name,count(*) from dba_segments where owner='SYSMAN' group by owner,segment_type,tablespace_name order by 1,2,3; OWNER SEGMENT_TYPE TABLESPACE_NAME COUNT(*) -------------------- ------------------ ------------------------------ ---------- SYSMAN INDEX MGMT_AD4J_TS 18 SYSMAN INDEX MGMT_ECM_DEPOT_TS 10 SYSMAN INDEX MGMT_TABLESPACE 1753 SYSMAN INDEX USERS 7 SYSMAN INDEX PARTITION MGMT_AD4J_TS 192 SYSMAN INDEX PARTITION MGMT_ECM_DEPOT_TS 15 SYSMAN INDEX PARTITION MGMT_TABLESPACE 1537 SYSMAN LOB PARTITION MGMT_ECM_DEPOT_TS 15 SYSMAN LOB PARTITION MGMT_TABLESPACE 205 SYSMAN LOBINDEX MGMT_AD4J_TS 1 SYSMAN LOBINDEX MGMT_ECM_DEPOT_TS 37 SYSMAN LOBINDEX MGMT_TABLESPACE 181 SYSMAN LOBSEGMENT MGMT_AD4J_TS 1 SYSMAN LOBSEGMENT MGMT_ECM_DEPOT_TS 37 SYSMAN LOBSEGMENT MGMT_TABLESPACE 181 SYSMAN NESTED TABLE MGMT_TABLESPACE 7 SYSMAN TABLE MGMT_AD4J_TS 13 SYSMAN TABLE MGMT_ECM_DEPOT_TS 11 SYSMAN TABLE MGMT_TABLESPACE 1079 SYSMAN TABLE USERS 53 SYSMAN TABLE PARTITION MGMT_AD4J_TS 192 SYSMAN TABLE PARTITION MGMT_TABLESPACE 625 22 rows selected.
The repository upgrade step runs fine now, the whole upgrade process continuous.
Finish
After while, yes we did it, the root script execution is the last task.
Script execution.
# /u01/app/oracle/product/em13cr4/allroot.sh
Starting to execute allroot.sh .........
Starting to execute /u01/app/oracle/product/em13cr4/root.sh ......
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/em13cr4
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
/etc exist
/u01/app/oracle/product/em13cr4
Finished execution of /u01/app/oracle/product/em13cr4/root.sh ......
The Enterprise Manager is now running with Release 4.
About Enterprise Manager
Next steps are
- upgrade the Oracle agents
- uninstall the old OEM software – btw, the installer already detaches the old ORACLE_HOME from Oracle’s central inventory for you and removes the emkey from the repository 🙂
Summary
This is a lab environment, not a huge setup with hundreds of targets. But we can see here two important points for the upgrade process. a) read the manual and disable all adaptive features, even when you have an 19c database and b) user SYSMAN needs permission on the USERS tablespace. This is very unusual and should be corrected from my side. But now, enjoy Oracle Enterprise Manager 13c Release 4!