EM12c Cloud Control

Enterprise Manager 13c – Let’s use the Hybrid Agent for Amazon EC2 and Azure Instances

I like the concept behind the Oracle Enterprise Manager Hybrid Cloud Architecture to connect my on-premise OMS with targets in the Oracle cloud. The Agent communicates via SSH tunnel to target servers, no other ports than SSH 22 are open against the world wide web. An I was interested to find out, if the installation of such an agent works for other cloud providers than Oracle too.

Create a Oracle Linux Instance in Amazon AWS

I have created a small Oracle Linux instance in Amazon AWS and inserted the public IP into the /etc/hosts file of the Oracle Management Server. Why I have used an Oracle Linux? According the documentation, at the moment only Oracle Linux x86-64 is supported to use this hybrid feature.

On the Amazon instance I installed the 12c prerequisite package (yum install oracle-rdbms-server-12cR1-preinstall)  to be sure that libraries etc. are available and the user oracle is created. And finally I added the public key in the authorized_keys file of the user oracle so that connects via SSH without a password are possible.

hybrid_01

Hint: Test your passwordless SSH connection with a tool like Putty or MobaXterm.

Login Credential

For the login via SSH tunnel, in Enterprise Manager 13c a named credential has to be created with the SSH keys which were used by the Amazon instance. This credential us used later in the agent deployment process. For further information how to create such a credential, please take a look into the Hybrid Coud documentation https://docs.oracle.com/cd/E24628_01/doc.121/e24473/hybrid-cloud.htm#BABJACHI.

hybrid_02

 

 

 

 

 

 

 

 

 

 

Hybrid Agent Deployment – First Run

After setting the DNS information and SSH configuration, it’s time to start a Hybrid Agent deployment. EM13c – Setup – Add Target – Install Agent on Host. The most important thing is that the checkbox for the Hybrid Cloud Agent is enabled at the bottom of the browser window.

hybrid_03

 

 

 

 

 

 

 

 

 

Host Name does not map to an Oracle Public Cloud Virtual Host – Investigation

The deployment has started. But in the prerequisites phase the remote validation fails with this message: The provided host name does not map to an Oracle Public Cloud virtual host. You can deploy Hybrid Cloud Agents only on Oracle Public Cloud virtual hosts.

hybrid_04

 

 

It looks like the prerequisites check is verifying the hostname. In the deployment logfile from the Oracle Management Server, I found these lines:

Oracle is doing a simple hostname -d command to verify if the host is running in the Oracle Cloud. I have verified the hostname -d command on Oracle Cloud instance, and the output there is different:

But my Amazon instance has this output here:

Lets fake the Amazon instance hostname and try the deployment again. I added this line into the Amazon instance /etc/hosts with the new domainname.

Now the hostname -d command shows me the new name according the Oracle cloud instances.

Hybrid Agent Deployment – Second Run

EM13c – Retry, using same inputs. And the error is gone, the agent is installed successfully. After running the scripts

  • /u00/app/oracle/product/agent13c/agent_13.1.0.0.0/root.sh
  • /home/oracle/oraInventory/orainstRoot.sh

as user root, the Amazon instance is added as new host.

hybrid_05

Summary

With a simple change for the hostname -d command, you can install the Oracle Hybrid Agent on targets outside the Oracle cloud. BTW, this works for local instances too. All other ports than SSH 22 are closed. And that’s an important thing when you work with cloud products.

Oracle Enterprise Manager 13c – KILL SESSION for Application Administrators – Part 1

Basically to execute a ALTER SYSTEM KILL SESSION command you have to be a) a DBA or b) you need the ALTER SYSTEM privilege. Granting the ALTER SYSTEM privilege to a Non-DBA has big risks. This user is now able to change a lot of parameters like memory parameters, NLS settings etc.

In one of my projects, a small team of well known application administrators is having a read-only account in Enterprise Manager 12c to verify the performance, see the user sessions and many more of their subset of databases. And sometimes, they have to kill a hanging Oracle session. Until now they called the DBA: “Please do it for me”. Sure, we can build a small PL/SQL procedure on every database and give them the executions rights so they can kill a session in their terminal theirself. But this is not very user friendly.

Here is an approach to manage the small path between security and manageability. I am aware that this is – like we say in Switzerland – a “Kompromiss”. But in fact we have implemented this solution in a production environment two months ago without any negative impacts.

Note: All the steps which are show below in Enterprise Manager 13c can be executed in 12c too.

The Concept

  • we create a new database user in the target databases
  • we create a new role with ALTER SYSTEM privilege in the target databases
  • we enable auditing for ALTER SYSTEM commands in the target databases
  • we create a new Enterprise Manager role for the application administrators
  • we create a new named credential with the new user and grant it to the application administrators
  • we build an Enterprise Manager report which shows us the ALTER SYSTEM actions based on a metric extension

The New Database User

This user has to be created in every target database.

The New Database Role

This role has to be created in every target database.

Grant role to the user:

Enable Auditing for ALTER SYSTEM Commands

For ALTER SESSION and ALTER SYSTEM:

Verify the enabled audit settings:

Verify the audit parameterin the target database. If audit_trail is not set to EXTENDED, the SQL command which was executed is not recorded. How it works with Unified Auditing will be verified in a later blog post.

In the example below you can see the difference in the column SQL_TEXT.

Enterprise Manager Role

Setup – Security – Role – Create

em13c_role1

Set name and description – Next

em13c_role2

Next

em13c_role3

Activate the checkbox for the privilige Connect to any viewable target and scroll down

em13c_role4

Add database targets and set the Manage Targets Privilege Grants

  • For EM12c use: View
  • For EM13c use: Manage Database Sessions

Next

em13c_role5b
Next

em13c_role6

Select the user to grant the role – Next

em13c_role5b

Finish

em13c_role8

The role is now created and be granted to a user.

em13c_role9

Enterprise Manager Named Credential

The application adninistrators don’t have to know the password for the created user with the ALTER privileges. We create a named credential and give the admins the permission to use it.

Setup – Security – Named Credentials – Create

Set Credential Name, Authenticated Target Type, Credential Type and set Scope to Global. The Credential Properties are according to our new created user APPL_ADMIN.

Scroll down to set Access Control

em13c_nc1

Add Grant

em13c_nc2

Search for the user, in my case it is APPL_BERGER

Select

em13c_nc3

Feel free to test ist against a target which contains the APPL_ADMIN user.

Save

em13c_nc5

Test

Now we test the configuration. The role APPL_ADMIN was granted to my user APPL_BERGER. On the target database TVD12 user SCOTT has locked some data.

em13c_test1

On the target site we go to the Blocking Sessions page

em13c_test2

The Named Credential is already filled in.

Login

em13c_test3

Select the session – Kill Session

em13c_test4

Confirm the action to kill the selected session immediate – Yes

em13c_test5

Session has been killed.

em13c_test6

Verification

On the target database, an audit record was generated. Login as user SYS and execute this query. You can see the TIME, the USERNAME, the EM13c USERNAME in column CLIENT_ID and the SQL statement which was executed in background.

Summary – Part 1

As I said in the introduction, giving some other users than DBAs the ALTER SYSTEM privilege is risky. But when the DBAs and application adminstrators are working as a team, then this can be a possible solution to make their daily business easier.

In the next blog post I will show how you can create a Enterprise Manager report based on a Metric Extension to produce daily reports of the ALTER SYSTEM actions.

EM12c Agent – java.lang.OutOfMemoryError: Java heap space

After an AIX 7.1 server reboot, there was one agent which did not started. The command emctl start agent resulted in a java.lang.OutOfMemoryError: Java heap space message. In the agent subdirectory some dumpfiles were created:

First I took a look in the agent logfiles to gather more information.

/u00/app/oracle/product/agent12c/agent_inst/sysman/log/emdctlj.log

There are a log of MOS notes available for the search term peer not authentificated, but none of them were helpful.

/u00/app/oracle/product/agent12c/agent_inst/sysman/log/emagent.nohup

Here we see that the EM12c agent had a JVM memory problem at startup. Let’s try out the emctl clearstate agent command.

emctl clearstate agent

From the Oracle documentation about the clearstate command – http://docs.oracle.com/cd/E29597_01/doc.1111/e24473/emctl.htm#r21c1-t27

emctl clearstate Clears the state directory contents. The files that are located under $ORACLE_HOME/sysman/emd/state will be deleted if this command is run. The state files are the files which are ready for the agent to convert them into corresponding xml files.

The emctl clearstate agent command produced the same error:

After some research in My Oracle Support for OutOfMemoryError: Java heap space, I found this note:

Duplicate 1952593.1 – EM12c: emctl start agent Fails With ‘ Target Interaction Manager failed at Startup java.lang.OutOfMemoryError: Java heap space’ reported in gcagent_errors.log (Doc ID 1902124.1)

From the MOS note:

  1. Kill any leftover process -> not for me, the agent was no started at all
  2. Move old files from  /agent_inst/sysman/emd/state/* to a new directory -> sounds very interesting
  3. Execute clearstate command -> let’s try it out

The directory u00/app/oracle/product/agent12c/agent_inst/sysman/emd/state/

There was a lot of stuff in this directory – time to move:

emctl clearstate agent – 2nd run

Now the clearstate command has worked without errors.

emctl start agent

The start was successful – fine.

Summary

The cleanup of the state directory solved the problem. But why does a emctl clearstate agent command not clean up the state directory as described in the documentation? Aaah, and do’t forget to clean up the moved files. They are not needed anymore.