EM13c Cloud Control

Oracle Enterprise Manager 13c Release 4 Update 5 (13.4.0.5) – Time to Patch the OMS

Availability and Information Release Update 13.4.0.5

Since some days, the Release Update 13.4.0.5 for the Oracle Enterprise Manager 13c Release 4 is available in My Oracle Support. Time to update my lab environment.

Note: This blog post describes how to apply the RU on an Oracle Enterprise Manager 13 Release 4 Oracle Management Server which is running on an Oracle Linux 7.6 machine. Applying the RU for agents is not part of of this blog post. For agent patching please visit readme of Patch 31544641: Oracle Enterprise Manager 13c Release 4 Platform Update 5 (13.4.0.5) for Oracle Management Agent.

The RU for the Oracle Management Server is generic and the compressed patch file has the size of 964.6 MB. Useful links for this Release Update like contained fixes and improvements:

From the Patch Readme:

Oracle Enterprise Manager 13c Release 4 Update 5 for Oracle Management Service contains bug fixes, and can be applied on the following:

  • Oracle Management Service 13c Release 4 (13.4.0.0.0)
  • Any Release Update previously released for Oracle Management Service 13c Release 4 (13.4.0.0.0)

OMSPatcher Prerequisites 

OMSPatcher has to be version 13.9.4.1.0 for Enterprise Manager 13.4.0.0.0 – let’s verify it on my OEM environment:

$ export ORACLE_HOME=/u01/app/oracle/product/em13cr4

$ORACLE_HOME/OMSPatcher/omspatcher version
OMSPatcher Version: 13.9.4.1.0
OPlan Version: 12.2.0.1.16
OsysModel build: Thu Oct 25 18:18:12 PDT 2018

OMSPatcher succeeded.

If your OMSPatcher has an older release, take a look here how to update it: EM 13c: How To Upgrade Enterprise Manager 13.4 Cloud Control OMSPatcher Utility to Version 13.9.4.1.0 (Doc ID 2646080.1)

Transfer and extract Release Update

The RU is extracted in the stage directory on the OEM server:

$ ll
total 987904
drwxr-xr-x 3 oracle oinstall 81 Aug 18 21:26 .
drwxr-xr-x. 13 oracle dba 4096 May 11 13:03 ..
drwxr-x--- 15 oracle oinstall 4096 Aug 13 07:37 31604791
-rw-rw-r-- 1 oracle oinstall 123537 Aug 17 19:54 PatchSearch.xml
-rw-r--r-- 1 oracle oinstall 1011475597 Aug 18 21:25 p31604791_134000_Generic.zip

Pre-Check

Run the pre-check to analyze the existing Oracle Enterprise Manager system.

$ cd /u01/app/oracle/stage/31604791
$ $ORACLE_HOME/OMSPatcher/omspatcher apply -analyze

Extract from the OMSPatch logfile, here in my case called omspatcher_2020-08-18_21-35-49PM_analyze.log.

The following sub-patches are incompatible with components installed in the OMS system:
31511488,31511179,31511147,30851102,31187567,31352743,31510486

These warnings can ignored, these are patches for plugins which I am not using. So the patch will not be applied here, for example:

  • oracle.sysman.emfa.oms.plugin (Enterprise Manager for Fusion Applications)
  • oracle.sysman.bda.oms.plugin (Oracle Big Data Appliance)

OPatch – The inventory shows Platform Update 13.4.0.3 which is my actual release.

$ $ORACLE_HOME/OPatch/opatch lsinventory | grep Platform
Patch description: "Oracle Enterprise Manager 13c Release 4 Platform Update 3 (13.4.0.3) for Oracle Management Service"

About Enterprise Manager

Show the installed Plugins – emcli

Login

$ $ORACLE_HOME/bin/emcli login -username=sysman
Enter password :
Login successful

Sync emcli

$ $ORACLE_HOME/bin/emcli sync
Synchronized successfully

List plugins

$ $ORACLE_HOME/bin/emcli list_plugins_on_server
OMS name is kestenholz.martinberger.local:4889_Management_Service
Plug-in Name Plugin-id Version [revision]

Oracle Cloud Framework oracle.sysman.cfw 13.4.1.0.0
Oracle Database oracle.sysman.db 13.4.1.0.0
Oracle Fusion Middleware oracle.sysman.emas 13.4.1.0.0
Systems Infrastructure oracle.sysman.si 13.4.1.0.0
Oracle Exadata oracle.sysman.xa 13.4.1.0.0

Release Update Apply

1st – Stop the OMS

$ORACLE_HOME/bin/emctl stop oms

2nd – Apply RU with OMSPatcher

I don’t use a credential file, therefore the username (if not using weblogic which is default) and the password for the Weblogic AdminServer has to be provied.

$ cd /u01/app/oracle/stage/31604791
$ $ORACLE_HOME/OMSPatcher/omspatcher apply
OMSPatcher Automation Tool
Copyright (c) 2017, Oracle Corporation. All rights reserved.


OMSPatcher version : 13.9.4.1.0
OUI version : 13.9.4.0.0
Running from : /u01/app/oracle/product/em13cr4
Log file location : /u01/app/oracle/product/em13cr4/cfgtoollogs/omspatcher/opatch2020-08-19_09-49-05AM_1.log

OMSPatcher log file: /u01/app/oracle/product/em13cr4/cfgtoollogs/omspatcher/31604791/omspatcher_2020-08-19_09-49-10AM_analyze.log

Please enter OMS weblogic admin server URL(t3s://kestenholz.martinberger.local:7102):>
Please enter OMS weblogic admin server username(weblogic):>
Please enter OMS weblogic admin server password:>
...
<patching actions>
<patching actions>
<patching actions>
...

Deployment summary:
-------------------

The following artifact(s) have been successfully deployed:

Artifacts Log file
--------- --------
SQL rcu_applypatch_original_patch_2020-08-18_21-54-04PM.log
SQL rcu_applypatch_original_patch_2020-08-18_21-56-10PM.log
SQL rcu_applypatch_original_patch_2020-08-18_21-56-27PM.log
SQL rcu_applypatch_original_patch_2020-08-18_21-56-45PM.log
SQL rcu_applypatch_original_patch_2020-08-18_21-57-03PM.log
MRS-commands emctl_register_commands_2020-08-18_21-57-24PM.log
MRS-commands emctl_register_commands_2020-08-18_21-57-28PM.log
MRS-procedures emctl_register_procedures_2020-08-18_21-57-32PM.log
MRS-swlib emctl_register_swlib_2020-08-18_21-57-39PM.log
MRS-swlib emctl_register_swlib_2020-08-18_21-57-45PM.log
MRS-targetType emctl_register_targetType_2020-08-18_21-57-53PM.log
MRS-targetType emctl_register_targetType_2020-08-18_21-58-02PM.log
MRS-storeTargetType emctl_register_storeTargetType_2020-08-18_22-00-48PM.log
MRS-storeTargetType emctl_register_storeTargetType_2020-08-18_22-00-54PM.log
MRS-default_collection emctl_register_default_collection_2020-08-18_22-01-01PM.log
MRS-default_collection emctl_register_default_collection_2020-08-18_22-01-08PM.log
MRS-omsPropertyDef emctl_register_omsPropertyDef_2020-08-18_22-01-35PM.log
MRS-omsPropertyDef emctl_register_omsPropertyDef_2020-08-18_22-01-40PM.log
MRS-jobTypes emctl_register_jobTypes_2020-08-18_22-01-45PM.log
MRS-jobTypes emctl_register_jobTypes_2020-08-18_22-01-50PM.log
MRS-jobTypes emctl_register_jobTypes_2020-08-18_22-01-55PM.log
MRS-systemStencil emctl_register_systemStencil_2020-08-18_22-02-03PM.log
MRS-systemStencil emctl_register_systemStencil_2020-08-18_22-02-08PM.log
MRS-gccompliance emctl_register_gccompliance_2020-08-18_22-02-12PM.log
MRS-discovery emctl_register_discovery_2020-08-18_22-02-28PM.log
MRS-CredstoreMetadata emctl_register_CredstoreMetadata_2020-08-18_22-02-33PM.log


--------------------------------------------------------------------------------
The following warnings have occurred during OPatch execution:
1) Could not apply the patch "31511488" because the "oracle.sysman.vt.oms.plugin with version 13.4.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
2) Could not apply the patch "31511179" because the "oracle.sysman.emct.oms.plugin with version 13.4.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
3) Could not apply the patch "31511147" because the "oracle.sysman.am.oms.plugin with version 13.4.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
4) Could not apply the patch "30851102" because the "oracle.sysman.vi.oms.plugin with version 13.4.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
5) Could not apply the patch "31187567" because the "oracle.sysman.bda.oms.plugin with version 13.4.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
6) Could not apply the patch "31352743" because the "oracle.sysman.ssa.oms.plugin with version 13.4.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
7) Could not apply the patch "31510486" because the "oracle.sysman.emfa.oms.plugin with version 13.4.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
--------------------------------------------------------------------------------
OMSPatcher Session completed with warnings.
Log file location: /u01/app/oracle/product/em13cr4/cfgtoollogs/omspatcher/31604791/omspatcher_2020-08-18_21-41-32PM_deploy.log

OMSPatcher completed with warnings.

Warnings can be ignored as long they depend on plugins which are not installed.

OPatch – The inventory shows Platform Update 13.4.0.5 the RU is applied.

$ $ORACLE_HOME/OPatch/opatch lsinventory | grep Platform
Patch description: "Oracle Enterprise Manager 13c Release 4 Platform Update 5 (13.4.0.5) for Oracle Management Service"

3rd – Start the OMS again

$ $ORACLE_HOME/bin/emctl start oms

About Enterprise Manager

Summary

Oracle Enterprise Manager 13c Release 4 Update 5 (13.4.0.5) – easy to apply, works!

Oracle Enterprise Manager EM13c – My Oracle Support is back!

Since a few weeks, the online connection in Oracle Enterprise Manager EM13c to My Oracle Support was broken. This resulted in errors like Error occurred when sending request to My Oracle Support, for example when you tried to set the My Oracle Support credentials in a fresh EM13c setup. Existing configurations were not able to get newest patch information anymore. Oracle has documented it in the note Recent Changes to MOS will Disable Enterprise Manager Online Connections (Doc ID 2664002.1)

Screenshot when trying to set MOS credentials in a new EM13c RU4 setup:

Now there is a fix available for all current EM 13 releases, RU included. For more details see Patch Now Available For Recent Changes to MOS That Disabled Enterprise Manager Online Connections (Doc ID 2678494.1). Here is the installation of the patch for an Oracle Enterprise Manager EM13c Release 4 RU2 without using a property file (optional). The patch is transferred to target OMS server and extracted in directory /u01/app/oracle/stage/31233849.

oracle@kestenholz:/u01/app/oracle/stage/ [em13cr4] ll
total 424
drwxr-xr-x 3 oracle oinstall 92 Jul 12 23:00 .
drwxr-xr-x. 13 oracle dba 4096 May 11 13:03 ..
drwxr-x--- 4 oracle oinstall 76 Jun 22 22:10 31233849
-rw-rw-r-- 1 oracle oinstall 3092 Jul 6 17:24 PatchSearch.xml
-rw-r--r-- 1 oracle oinstall 425127 Jul 12 23:00 p31233849_13400200531Bundle_Generic.zip

 

Stop OMS

oracle@kestenholz:/u01/app/oracle/stage/ [em13cr4] emctl stop oms

Validate Patch Apply

oracle@kestenholz:/u01/app/oracle/stage/ [em13cr4] cd /u01/app/oracle/stage/31233849

oracle@kestenholz:/u01/app/oracle/stage/31233849/ [em13cr4] $ORACLE_HOME/OMSPatcher/omspatcher apply -analyze
OMSPatcher Automation Tool
Copyright (c) 2017, Oracle Corporation. All rights reserved.


OMSPatcher version : 13.9.4.1.0
OUI version : 13.9.4.0.0
Running from : /u01/app/oracle/product/em13cr4
Log file location : /u01/app/oracle/product/em13cr4/cfgtoollogs/omspatcher/opatch2020-07-12_23-01-08PM_1.log

OMSPatcher log file: /u01/app/oracle/product/em13cr4/cfgtoollogs/omspatcher/31233849/omspatcher_2020-07-12_23-01-13PM_analyze.log

Please enter OMS weblogic admin server URL(t3s://kestenholz.martinberger.local:7102):>
Please enter OMS weblogic admin server username(weblogic):>
Please enter OMS weblogic admin server password:>

Configuration Validation: Success


Running apply prerequisite checks for sub-patch(es) "31233849" and Oracle Home "/u01/app/oracle/product/em13cr4"...
Sub-patch(es) "31233849" are successfully analyzed for Oracle Home "/u01/app/oracle/product/em13cr4"


Complete Summary
================


All log file names referenced below can be accessed from the directory "/u01/app/oracle/product/em13cr4/cfgtoollogs/omspatcher/2020-07-12_23-01-08PM_SystemPatch_31233849_1"

Prerequisites analysis summary:
-------------------------------

The following sub-patch(es) are applicable:

Featureset Sub-patches Log file
---------- ----------- --------
oracle.sysman.top.oms 31233849 31233849_opatch2020-07-12_23-01-13PM_1.log

Log file location: /u01/app/oracle/product/em13cr4/cfgtoollogs/omspatcher/31233849/omspatcher_2020-07-12_23-01-13PM_analyze.log

OMSPatcher succeeded.

Patch Apply

oracle@kestenholz:/u01/app/oracle/stage/31233849/ [em13cr4] $ORACLE_HOME/OMSPatcher/omspatcher apply
OMSPatcher Automation Tool
Copyright (c) 2017, Oracle Corporation. All rights reserved.


OMSPatcher version : 13.9.4.1.0
OUI version : 13.9.4.0.0
Running from : /u01/app/oracle/product/em13cr4
Log file location : /u01/app/oracle/product/em13cr4/cfgtoollogs/omspatcher/opatch2020-07-12_23-03-23PM_1.log

OMSPatcher log file: /u01/app/oracle/product/em13cr4/cfgtoollogs/omspatcher/31233849/omspatcher_2020-07-12_23-03-27PM_deploy.log

Please enter OMS weblogic admin server URL(t3s://kestenholz.martinberger.local:7102):>
Please enter OMS weblogic admin server username(weblogic):>
Please enter OMS weblogic admin server password:>

Configuration Validation: Success


Running apply prerequisite checks for sub-patch(es) "31233849" and Oracle Home "/u01/app/oracle/product/em13cr4"...
Sub-patch(es) "31233849" are successfully analyzed for Oracle Home "/u01/app/oracle/product/em13cr4"

To continue, OMSPatcher will do the following:
[Patch and deploy artifacts] : Apply sub-patch(es) [ 31233849 ]
Apply RCU artifact with patch "/u01/app/oracle/product/em13cr4/.omspatcher_storage/31233849_Jun_22_2020_22_08_35/original_patch";
Register MRS artifact "omsPropertyDef"


Do you want to proceed? [y|n]
y
User Responded with: Y

Applying sub-patch(es) "31233849"
Please monitor log file: /u01/app/oracle/product/em13cr4/cfgtoollogs/opatch/opatch2020-07-12_23-03-27PM_1.log


Updating repository with RCU reference file "/u01/app/oracle/product/em13cr4/.omspatcher_storage/31233849_Jun_22_2020_22_08_35/original_patch"

Registering service "omsPropertyDef" with register file "/u01/app/oracle/product/em13cr4/sysman/metadata/omsProperties/definition/MosOMSPropDefinition.xml" for plugin id as "core"...
Please monitor log file: /u01/app/oracle/product/em13cr4/cfgtoollogs/omspatcher/2020-07-12_23-03-23PM_SystemPatch_31233849_1/emctl_register_omsPropertyDef_2020-07-12_23-06-34PM.log


Complete Summary
================


All log file names referenced below can be accessed from the directory "/u01/app/oracle/product/em13cr4/cfgtoollogs/omspatcher/2020-07-12_23-03-23PM_SystemPatch_31233849_1"

Patching summary:
-----------------

Binaries of the following sub-patch(es) have been applied successfully:

Featureset Sub-patches Log file
---------- ----------- --------
oracle.sysman.top.oms_13.4.0.0.0 31233849 31233849_opatch2020-07-12_23-03-27PM_1.log

Deployment summary:
-------------------

The following artifact(s) have been successfully deployed:

Artifacts Log file
--------- --------
SQL rcu_applypatch_original_patch_2020-07-12_23-06-16PM.log
MRS-omsPropertyDef emctl_register_omsPropertyDef_2020-07-12_23-06-34PM.log


Log file location: /u01/app/oracle/product/em13cr4/cfgtoollogs/omspatcher/31233849/omspatcher_2020-07-12_23-03-27PM_deploy.log

OMSPatcher succeeded.

 

Start OMS

oracle@kestenholz:/u01/app/oracle/stage/31233849/ [em13cr4] emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation. All rights reserved.
Starting Oracle Management Server...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
JVMD Engine is Up

 

New try to set My Oracle Support Credentials – MOS is back!

Et voilà

MOS is back, take care if you use a proxy or a firewall – a new URL has to be whitelisted:  https://oauth-e.oracle.com !

Oracle Enterprise Manager 13c Release 4 – Time to Upgrade – First Experiences

The roll-out of the newest Oracle Enterprise Manager 13 Release 4 is a few days old, about time to try out the upgrade process in my ESXi lab environment.

First: This blog post about the OEM upgrade process is based on my own experience.

Software

https://www.oracle.com/enterprise-manager/downloads/cloud-control-downloads.html

Documents

https://docs.oracle.com/en/enterprise-manager/cloud-control/enterprise-manager-cloud-control/13.4/emupg/index.html

The Environment

  • Oracle Enterprise Manager 13.3 running on Oracle Linux 7.6 – OEM Patchlevel January 2020 – Non-HA
  • Oracle Enterprise Edition Repository 19.6.0 Single Tenant Database 
  • Oracle Restart / ASM 19.6.0
  • Additional Oracle Linux Server with 19.3.0 Container Databases
  • All targets up and running
  • My Oracle Support connected
  • Software staged directory in /u01/app/oracle/stage/em13cr4 and permission changed to execute bin file (chmod u+x em13400_linux64.bin).
oracle@kestenholz:/u01/app/oracle/stage/em13cr4/ [oms1303] ll
total 12429656
drwxr-xr-x 2 oracle oinstall       4096 Feb  2 12:13 .
drwxr-xr-x 3 oracle oinstall         21 Feb  2 20:13 ..
-rw-r--r-- 1 oracle oinstall 2109004737 Feb  2 12:11 em13400_linux64-2.zip
-rw-r--r-- 1 oracle oinstall 2047749474 Feb  2 12:11 em13400_linux64-3.zip
-rw-r--r-- 1 oracle oinstall 2117312528 Feb  2 12:12 em13400_linux64-4.zip
-rw-r--r-- 1 oracle oinstall  808014516 Feb  2 12:12 em13400_linux64-5.zip
-rw-r--r-- 1 oracle oinstall 2143125187 Feb  2 12:13 em13400_linux64-6.zip
-rw-r--r-- 1 oracle oinstall 1879163226 Feb  2 12:14 em13400_linux64-7.zip
-rwxr-x--- 1 oracle oinstall 1623577684 Feb  2 12:10 em13400_linux64.bin

Prerequisites

It’s important to verify the changed prerequisites – Chapter 3 – Prerequisites for Upgrading to Enterprise Manager Cloud Control 13c Release 4 – I had do disable all adaptive features in the repository pluggable database:

alter system set "_optimizer_nlj_hj_adaptive_join"= FALSE scope=both
sid='*';
alter system set "_optimizer_strans_adaptive_pruning" = FALSE
scope=both sid='*';
alter system set "_px_adaptive_dist_method" = OFF scope=both sid='*';
alter system set "_sql_plan_directive_mgmt_control" = 0 scope=both
sid='*';
alter system set "_optimizer_dsdir_usage_control" = 0 scope=both
sid='*';
alter system set "_optimizer_use_feedback" = FALSE scope=both sid='*';
alter system set "_optimizer_gather_feedback" = FALSE scope=both
sid='*';
alter system set "_optimizer_performance_feedback" = OFF scope=both
sid='*';

Restart the database after the changed settings.From my view this sentence here is wrong is the documentation (my database has version 19.6.0):

If your Management Repository is using Oracle Database 12.2 or higher, none of these parameters need to be set. 

Why? If these parameters are not set, the installer refuses working.

EMKEY Copy Requirements

The Key was copied to the repository. It well be removed after the successful upgrade.

oracle@kestenholz: [oms1303] $ORACLE_HOME/bin/emctl config emkey -copy_to_repos
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".

Run Installer

Start the Oracle Universal Installer

The OMS is down now (emctl stop oms -all). Start the installer.

oracle@kestenholz:/u01/app/oracle/stage/em13cr4/ [oms1303] ./em13400_linux64.bin

My Oracle Support Details

I am a registered user in My Oracle Support, I get enough information every week… I don’t enable this checkbox.

Software Updates

At the moment, there are no updates available. Maybe in future there will be any patches available for auto apply during the installation/upgrade process.

Installation Type

Upgrade an existing Enterprise Manager system, this one is my existing installation.

 

Installation Details

Enter a new Middleware Home Location.

Database Connection Details

Enter the SYS and SYSMAN password of the running repository. DDMP (Deferred Data Migration) Jobs are enabled. According the documentation, these jobs are running in the background when the OEM is starting up during the upgrade process to convert old data to the new format. If you have a huge amount of data in your earlier release, then the upgrade can take longer. You can run this job – if disabled – later as Post Upgrade Tasks.

Warnings

I will change to SHA communication later.

The repository has 19.6.0, all required patches are included.

I confirm that the JVMD engine is stopped, this has happened by stopping the OMS.

I had to stop the agent which was running on the management server.

Fix Parameter Settings by the Installer

For my environment, I let the installer fix this settings.

Plug-in Upgrade

Here we see the already installed plugins, they will be upgraded too.

Select Plug-ins

I don’t ant to install additional plug-ins.

Extend WebLogic Server Domain

Enter the password for the weblogic user and define the OMS Instance Base Location.

Enterprise Manager Shared Location Details

My Oracle Enterprise Manager doesn’t run in a high availability setup, I don’t need any shared location.

Port Configuration Details

I use the settings from the existing installation.

Review

Let’s start the upgrade!

Repository Upgrade failed

The upgrade process fails at the step where the repository has to be upgraded.

View Log – ORA-01950

When I scroll up the provided log from the installer, I see this error message here:

The schema manager logfile located in the 13.4 subdirectory $ORACLE_HOME/sysman/log/schemamanager confirms this error.

oracle@kestenholz:/u01/app/oracle/product/em13cr4/sysman/log/schemamanager/m_020220_0947_PM/m_020220_0947_PM.PREUPGRADE/ [oms1303] grep 1950 *
emschema.log.PREUPGRADE:SEVERE: ERROR:ORA-01950: no privileges on tablespace 'USERS'
grep: RCU2020-02-02_21-47_829509113: Is a directory
rcu.log:Sun Feb 2 21:47:41.519 CET 2020 ERROR assistants.rcu.Rcu: oracle.sysman.assistants.rcu.backend.action.AbstractAction::handleNonIgnorableError: Received Non-Ignorable Error: ORA-01950: no privileges on tablespace 'USERS'
rcu.log:Sun Feb 2 21:47:41.520 CET 2020 ERROR assistants.rcu.Rcu: oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine::onException: SQLException: ORA-01950: no privileges on tablespace 'USERS'
rcu.log:java.sql.SQLSyntaxErrorException: ORA-01950: no privileges on tablespace 'USERS'
rcu.log:oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: java.sql.SQLSyntaxErrorException: ORA-01950: no privileges on tablespace 'USERS'
rcu.log:Caused by: java.sql.SQLSyntaxErrorException: ORA-01950: no privileges on tablespace 'USERS'
rcu.log:Caused by: oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: java.sql.SQLSyntaxErrorException: ORA-01950: no privileges on tablespace 'USERS'
rcu.log:Caused by: java.sql.SQLSyntaxErrorException: ORA-01950: no privileges on tablespace 'USERS'

At this point, this is curious, because in 13.3, there were no objects of the SYSMAN in the USERS tablespace. All objects are located in OEM tablespaces with the MGMT prefix. Here is a list of objects in a repository where an Oracle Enterprise Manager 13c Release 3 is up and running.

SQL> select owner,segment_type,tablespace_name,count(*) from dba_segments where owner='SYSMAN' group by owner,segment_type,tablespace_name order by 1,2,3;

OWNER                SEGMENT_TYPE       TABLESPACE_NAME                  COUNT(*)
-------------------- ------------------ ------------------------------ ----------
SYSMAN               INDEX              MGMT_AD4J_TS                           18
SYSMAN               INDEX              MGMT_ECM_DEPOT_TS                      10
SYSMAN               INDEX              MGMT_TABLESPACE                      1701
SYSMAN               INDEX PARTITION    MGMT_AD4J_TS                          192
SYSMAN               INDEX PARTITION    MGMT_ECM_DEPOT_TS                      15
SYSMAN               INDEX PARTITION    MGMT_TABLESPACE                      1515
SYSMAN               LOB PARTITION      MGMT_ECM_DEPOT_TS                      15
SYSMAN               LOB PARTITION      MGMT_TABLESPACE                       203
SYSMAN               LOBINDEX           MGMT_AD4J_TS                            1
SYSMAN               LOBINDEX           MGMT_ECM_DEPOT_TS                      37
SYSMAN               LOBINDEX           MGMT_TABLESPACE                       159
SYSMAN               LOBSEGMENT         MGMT_AD4J_TS                            1
SYSMAN               LOBSEGMENT         MGMT_ECM_DEPOT_TS                      37
SYSMAN               LOBSEGMENT         MGMT_TABLESPACE                       159
SYSMAN               NESTED TABLE       MGMT_TABLESPACE                         7
SYSMAN               TABLE              MGMT_AD4J_TS                           13
SYSMAN               TABLE              MGMT_ECM_DEPOT_TS                      11
SYSMAN               TABLE              MGMT_TABLESPACE                      1053
SYSMAN               TABLE PARTITION    MGMT_AD4J_TS                          192
SYSMAN               TABLE PARTITION    MGMT_TABLESPACE                       622

20 rows selected.

The solution is simple. Just grant the permissions for the USERS tablespace to SYSMAN and retry the installation progress.But, don’t ask me why SYSMAN creates now objects in USERS…

SQL> ALTER USER sysman QUOTA UNLIMITED ON users;

User altered.

After a while, you can see new created tables and indexes in tablespace USERS for user SYSMAN.

SQL> select owner,segment_type,tablespace_name,count(*) from dba_segments where owner='SYSMAN' group by owner,segment_type,tablespace_name order by 1,2,3;

OWNER                SEGMENT_TYPE       TABLESPACE_NAME                  COUNT(*)
-------------------- ------------------ ------------------------------ ----------
SYSMAN               INDEX              MGMT_AD4J_TS                           18
SYSMAN               INDEX              MGMT_ECM_DEPOT_TS                      10
SYSMAN               INDEX              MGMT_TABLESPACE                      1753
SYSMAN               INDEX              USERS                                   7
SYSMAN               INDEX PARTITION    MGMT_AD4J_TS                          192
SYSMAN               INDEX PARTITION    MGMT_ECM_DEPOT_TS                      15
SYSMAN               INDEX PARTITION    MGMT_TABLESPACE                      1537
SYSMAN               LOB PARTITION      MGMT_ECM_DEPOT_TS                      15
SYSMAN               LOB PARTITION      MGMT_TABLESPACE                       205
SYSMAN               LOBINDEX           MGMT_AD4J_TS                            1
SYSMAN               LOBINDEX           MGMT_ECM_DEPOT_TS                      37
SYSMAN               LOBINDEX           MGMT_TABLESPACE                       181
SYSMAN               LOBSEGMENT         MGMT_AD4J_TS                            1
SYSMAN               LOBSEGMENT         MGMT_ECM_DEPOT_TS                      37
SYSMAN               LOBSEGMENT         MGMT_TABLESPACE                       181
SYSMAN               NESTED TABLE       MGMT_TABLESPACE                         7
SYSMAN               TABLE              MGMT_AD4J_TS                           13
SYSMAN               TABLE              MGMT_ECM_DEPOT_TS                      11
SYSMAN               TABLE              MGMT_TABLESPACE                      1079
SYSMAN               TABLE              USERS                                  53
SYSMAN               TABLE PARTITION    MGMT_AD4J_TS                          192
SYSMAN               TABLE PARTITION    MGMT_TABLESPACE                       625

22 rows selected.

The repository upgrade step runs fine now, the whole upgrade process continuous.

Finish

After while, yes we did it, the root script execution is the last task. 

Script execution.

# /u01/app/oracle/product/em13cr4/allroot.sh

Starting to execute allroot.sh .........

Starting to execute /u01/app/oracle/product/em13cr4/root.sh ......
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/em13cr4

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
/etc exist
/u01/app/oracle/product/em13cr4
Finished execution of  /u01/app/oracle/product/em13cr4/root.sh ......

The Enterprise Manager is now running with Release 4.

About Enterprise Manager

Next steps are

  • upgrade the Oracle agents
  • uninstall the old OEM software –  btw, the installer already detaches the old ORACLE_HOME from Oracle’s central inventory for you and removes the emkey from the repository 🙂

Summary

This is a lab environment, not a huge setup with hundreds of targets. But we can see here two important points for the upgrade process. a) read the manual and disable all adaptive features, even when you have an 19c database and b) user SYSMAN needs permission on the USERS tablespace. This is very unusual and should be corrected from my side. But now, enjoy Oracle Enterprise Manager 13c Release 4!

OPatch 13.9 in EM13c – Say Goodbye to Unzip, Copy & Paste

Yesterday I wanted to apply a brand new patch to customer’s Enterprise Manager 13cR2 OMS running on Linux. First I updated the OMSPatcher as described here: How to upgrade the 13.1 Cloud Control OMSPatcher to latest version of OMSPatcher (Doc ID 2135028.1). This update was easy. Download, transfer, unzip and copy the OMSPatcher files to the Enterprise Manager ORACLE_HOME directory.

Then the patch apply results in an error. OMSPatcher is based on OPatch, he needs an update too.

We need a new OPatch Version

This was error message for patch apply with omspatcher when I executed the apply command.

[Dec 1, 2016 10:17:38 AM]    Prerequisite check "CheckMinimumOPatchVersion" failed.
                             The details are:

                             The OPatch being used has version 13.8.0.0.0 while the following patch(es) require higher versions:
                             Patch 24914115 requires OPatch version 13.9.1.0.0.
                             Please download latest OPatch from My Oracle Support.

My actual EM13cR2 OPatch version was 13.8.0.0.0.

oracle@kestenholz:/u00/app/oracle/product/em13cr2/OPatch/ [oms13cr2] ./opatch version
OPatch Version: 13.8.0.0.0

OPatch succeeded.

Get the new Version

In My Oracle Support I found the newest version of OPatch, Download.

opatch_01

Transfer and extract

The next step was to extract and to transfer the package to the target server.And then I was wondering that not like in versions before an OPatch directory was created, now there is a JAR file inside the package. Copy & Paste of an OPatch directory is no more possible anymore.

oracle@kestenholz:/u00/app/oracle/stage/OPatch/ [oms13cr2] unzip p6880880_139000_Generic.zip
Archive:  p6880880_139000_Generic.zip
   creating: 6880880/
  inflating: 6880880/README.txt
  inflating: 6880880/opatch_generic.jar
  inflating: 6880880/version.txt

Installation Routine – README.txt

Let’s take a look in the README.txt – there is an installation manual.

oracle@kestenholz:/u00/app/oracle/stage/OPatch/6880880/ [oms13cr2] cat README.txt
Installation
----------------------------------
- Backup your <ORACLE_HOME>

- Unzip this patch into your staging directory PATCH_HOME

- Install the software via:
    java -jar <PATCH_HOME>/6880880/opatch_generic.jar -silent oracle_home=ORACLE_HOME

- To validate the installation:
    cd <ORACLE_HOME>/OPatch
    opatch version
    opatch lspatches

Installation – 1st run – where is my JDK ?

oracle@kestenholz:~/ [oms13cr2] java -jar /u00/app/oracle/stage/6880880/opatch_generic.jar -silent oracle_home=/u00/app/oracle/roduct/em13cr2
Launcher log file is /tmp/OraInstall2016-12-02_10-00-10AM/launcher2016-12-02_10-00-10AM.log.
Extracting the installer . . . . Done
This installer must be executed using a Java Development Kit (JDK)
but /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-3.b17.el7.x86_64/jre is not a valid JDK Java Home.
The log is located here: /tmp/OraInstall2016-12-02_10-00-10AM/launcher2016-12-02_10-00-10AM.log.

Installation – 2nd run – JRE is not good enough

I tried to use the existing JDK from my Oracle 12.1 installation

oracle@kestenholz:/u00/app/oracle/stage/6880880/ [oms13cr2] export JAVA_HOME=/u00/app/oracle/product/12.1.0.2/dbhome_1/jdk/bin
oracle@kestenholz:/u00/app/oracle/stage/6880880/ [oms13cr2] $JAVA_HOME/java -jar /u00/app/oracle/stage/6880880/opatch_generic.jar -silent oracle_home=/u00/app/oracle/product/em13cr2
Exception in thread "main" java.lang.UnsupportedClassVersionError: com/oracle/cie/nextgen/launcher/Launcher : Unsupported major.minor version 51.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:637)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: com.oracle.cie.nextgen.launcher.Launcher. Program will exit.

Installation – 3rd run – JDK 1.7 rocks

Now I have downloaded and configured JDK 1.7. Another try, and now it works. Here is a short summary of the output. In the background now the Oracle Universal Installer is started.

Thanks to Gokhan Atil – http://www.gokhanatil.com – for the second possibility, you can use the JDK from the OMS too.

oracle@kestenholz:~/ [oms13cr2] export JAVA_HOME=/u00/app/oracle/product/jdk1.7.0_79/bin
oracle@kestenholz:~/ [oms13cr2] $JAVA_HOME/java -jar /u00/app/oracle/stage/6880880/opatch_generic.jar -silent oracle_home=/u00/app/oracle/product/em13cr2
Launcher log file is /tmp/OraInstall2016-12-02_10-05-14AM/launcher2016-12-02_10-05-14AM.log.
Extracting the installer . . . . Done
Checking if CPU speed is above 300 MHz.   Actual 2793.546 MHz    Passed
Checking swap space: must be greater than 512 MB.   Actual 2047 MB    Passed
Checking if this platform requires a 64-bit JVM.   Actual 64    Passed (64-bit not required)
Checking temp space: must be greater than 300 MB.   Actual 37290 MB    Passed


Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2016-12-02_10-05-14AM
Installation Summary


Disk Space : Required 6 MB, Available 37,253 MB
Feature Sets to Install:
        Next Generation Install Core 13.9.1.0.0
        OPatch 13.9.1.0.0
        OPatch Auto OPlan 13.9.1.0.0
Session log file is /tmp/OraInstall2016-12-02_10-05-14AM/install2016-12-02_10-05-14AM.log

Loading products list. Please wait.
 1%
 40%

Loading products. Please wait.
 43%
 46%
 ...
 99%
Updating Libraries
Starting Installations
 1%
 2%
 3%
 ...
 91%
 92%

Install pending
Installation in progress
 Component : oracle.swd.opatch 13.9.1.0.0
Copying files for 'oracle.swd.opatch 13.9.1.0.0 '
 Component : oracle.glcm.osys.core 13.9.1.0.0
...
 Feature Set : oracle.glcm.opatchauto.core.actions.classpath
Post Feature installing 'oracle.glcm.opatchauto.core.actions.classpath'
Post feature install complete
String substitutions pending
...
Setting up 'oracle.glcm.opatchauto.core 13.9.1.0.0 '
Setup successful
Save inventory pending
Saving inventory
 93%
Saving inventory complete
 94%
Configuration complete
Logs successfully copied to /u00/app/oraInventory/logs.

Job done

Now the version was ok, the patch could be applied.

oracle@kestenholz:/u00/app/oracle/product/em13cr2/OPatch/ [oms13cr2] ./opatch version
OPatch Version: 13.9.1.0.0

OPatch succeeded.

Additional Information: If the installer does not find your inventory location (was happened to me on an AIX system), just execute the install command with the parameter -invPtrLoc.

java -jar /u00/app/oracle/stage/6880880/opatch_generic.jar -silent oracle_home=/u00/app/oracle/product/oms13cr2 -invPtrLoc /etc/oraInst.loc

The OMSPatcher can be updated like before, but the other important component in the patch process OPatch has has to be installed. From my point of view this is a step back, I don’t know what was the idea behind to change it. Now you have to be sure, that there is an existing JDK 1.7 available on your system. This means that applying a patch on the express way is not possible anymore, you have to fulfill the prerequisite before you have an actual OPatch. And at the moment, there is no other documentation than the README.txt available, the hint with the inventory parameter came from My Oracle Support after I opened a SR.

Say good-bye to Unzip, Copy & Paste.

Enterprise Manager 13cR2 – How to apply Patch 23094292: WLS PATCH SET UPDATE 12.1.3.0.160719

My fresh EM13cR2 installation showed me that there is a patch for the Weblogic environment available. This patch was released in July 2016: Patch Patch 23094292: WLS PATCH SET UPDATE 12.1.3.0.160719. This patchset has included 157 fixes and is a generic one. The patch is marked in My Oracle Support as recommended. OPatch has no to be updated. This patch is not an online patch, you have to shut down your running EM13cR2 server.

This blog post describes the apply of the Weblogic patch set update in an Enterprise Manager 13cR2 environment running on a Oracle Linux server.

patch_01

Prepare Patch Set Update on EM13c Server

The patch file has to be extracted. I have copied to file to my EM13cR2 server stage directory /u00/app/oracle/stage.

[oracle@kestenholz ~]$ cd /u00/app/oracle/stage/
[oracle@kestenholz stage]$ ll
total 32560
-rw-r--r--. 1 oracle oinstall 33340052 Oct 17 12:51 p23094292_121300_Generic.zip
[oracle@kestenholz stage]$ unzip p23094292_121300_Generic.zip

Set ORACLE_HOME

Set the ORACLE_HOME variable to the directory where the Oracle Enterprise Manager 13cR2 is located. In my example the EM13cR2 is installed in directory /u00/app/oracle/product/em13cr2.

[oracle@kestenholz stage]$ export ORACLE_HOME=/u00/app/oracle/product/em13cr2

Stop running Oracle Enterprise Manager 13c

[oracle@kestenholz stage]$ $ORACLE_HOME/bin/emctl stop oms -all

Apply Patch 23094292: WLS PATCH SET UPDATE 12.1.3.0.160719

Go to the extracted patch set directory:

[oracle@kestenholz ~]$ cd /u00/app/oracle/stage/23094292

Apply the patch:

[oracle@kestenholz 23094292]$ $ORACLE_HOME/OPatch/opatch apply
Oracle Interim Patch Installer version 13.8.0.0.0
Copyright (c) 2016, Oracle Corporation.  All rights reserved.


Oracle Home       : /u00/app/oracle/product/em13cr2
Central Inventory : /u00/app/oraInventory
   from           : /u00/app/oracle/product/em13cr2/oraInst.loc
OPatch version    : 13.8.0.0.0
OUI version       : 13.8.0.0.0
Log file location : /u00/app/oracle/product/em13cr2/cfgtoollogs/opatch/23094292_Oct_17_2016_13_02_45/apply2016-10-17_13-02-42PM_1.log


OPatch detects the Middleware Home as "/u00/app/oracle/product/em13cr2"

Verifying environment and performing prerequisite checks...

Conflicts/Supersets for each patch are:

Patch : 23094292

        Bug Superset of 21252292
        Super set bugs are:
        21252292

        Bug Superset of 21243471
        Super set bugs are:
        20613957, 19883023, 19703527

        Bug Superset of 20758863
        Super set bugs are:
        20758863

        Bug Superset of 19953516
        Super set bugs are:
        19953516

        Bug Superset of 19879223
        Super set bugs are:
        19879223

        Bug Superset of 19730967
        Super set bugs are:
        19730967

        Bug Superset of 18836900
        Super set bugs are:
        18836900


Patches [   21252292   21243471   20758863   19953516   19879223   19730967   18836900 ] will be rolled back.

OPatch continues with these patches:   23094292

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u00/app/oracle/product/em13cr2')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '23094292' to OH '/u00/app/oracle/product/em13cr2'
Rolling back interim patch '21252292' from OH '/u00/app/oracle/product/em13cr2'

Patching component oracle.wls.libraries, 12.1.3.0.0...

Patching component oracle.wls.libraries, 12.1.3.0.0...
RollbackSession removing interim patch '21252292' from inventory
Rolling back interim patch '21243471' from OH '/u00/app/oracle/product/em13cr2'

Patching component oracle.wls.libraries, 12.1.3.0.0...

Patching component oracle.wls.libraries, 12.1.3.0.0...
RollbackSession removing interim patch '21243471' from inventory
Rolling back interim patch '20758863' from OH '/u00/app/oracle/product/em13cr2'

Patching component oracle.wls.libraries, 12.1.3.0.0...

Patching component oracle.wls.libraries, 12.1.3.0.0...
RollbackSession removing interim patch '20758863' from inventory
Rolling back interim patch '19953516' from OH '/u00/app/oracle/product/em13cr2'

Patching component oracle.wls.workshop.code.completion.support, 12.1.3.0.0...

Patching component oracle.wls.workshop.code.completion.support, 12.1.3.0.0...

Patching component oracle.wls.libraries, 12.1.3.0.0...

Patching component oracle.wls.libraries, 12.1.3.0.0...

Patching component oracle.wls.clients, 12.1.3.0.0...

Patching component oracle.wls.clients, 12.1.3.0.0...
RollbackSession removing interim patch '19953516' from inventory
Rolling back interim patch '19879223' from OH '/u00/app/oracle/product/em13cr2'

Patching component oracle.wls.libraries, 12.1.3.0.0...

Patching component oracle.wls.libraries, 12.1.3.0.0...
RollbackSession removing interim patch '19879223' from inventory
Rolling back interim patch '19730967' from OH '/u00/app/oracle/product/em13cr2'

Patching component oracle.wls.libraries, 12.1.3.0.0...
RollbackSession removing interim patch '19730967' from inventory
Rolling back interim patch '18836900' from OH '/u00/app/oracle/product/em13cr2'

Patching component oracle.wls.libraries, 12.1.3.0.0...
RollbackSession removing interim patch '18836900' from inventory


OPatch back to application of the patch '23094292' after auto-rollback.


Patching component oracle.wls.workshop.code.completion.support, 12.1.3.0.0...

Patching component oracle.wls.workshop.code.completion.support, 12.1.3.0.0...

Patching component oracle.wls.shared.with.cam, 12.1.3.0.0...

Patching component oracle.wls.shared.with.cam, 12.1.3.0.0...

Patching component oracle.wls.libraries.mod, 12.1.3.0.0...

Patching component oracle.wls.libraries.mod, 12.1.3.0.0...

Patching component oracle.wls.admin.console.en, 12.1.3.0.0...

Patching component oracle.wls.admin.console.en, 12.1.3.0.0...

Patching component oracle.wls.core.app.server, 12.1.3.0.0...

Patching component oracle.wls.core.app.server, 12.1.3.0.0...

Patching component oracle.webservices.wls, 12.1.3.0.0...

Patching component oracle.webservices.wls, 12.1.3.0.0...

Patching component oracle.wls.clients, 12.1.3.0.0...

Patching component oracle.wls.clients, 12.1.3.0.0...

Patching component oracle.wls.server.shared.with.core.engine, 12.1.3.0.0...

Patching component oracle.wls.server.shared.with.core.engine, 12.1.3.0.0...

Patching component oracle.wls.libraries, 12.1.3.0.0...

Patching component oracle.wls.libraries, 12.1.3.0.0...
Patch 23094292 successfully applied.
OPatch Session completed with warnings.
Log file location: /u00/app/oracle/product/em13cr2/cfgtoollogs/opatch/23094292_Oct_17_2016_13_02_45/apply2016-10-17_13-02-42PM_1.log

OPatch completed with warnings.

You can ignore the warning message, it’s because OPatch has to rollback a previous installed patch.

Start Oracle Enterprise Manager 13c

[oracle@kestenholz 23094292]$ $ORACLE_HOME/bin/emctl start oms

Summary

Like the patches which were released for Weblogic / Enterprise Manager before, the installation was very smooth. There were no problems and the EM13cR2 started well after the patch apply.