Monthly Archive: October 2015

EM12c Agent – java.lang.OutOfMemoryError: Java heap space

After an AIX 7.1 server reboot, there was one agent which did not started. The command emctl start agent resulted in a java.lang.OutOfMemoryError: Java heap space message. In the agent subdirectory some dumpfiles were created:

First I took a look in the agent logfiles to gather more information.


There are a log of MOS notes available for the search term peer not authentificated, but none of them were helpful.


Here we see that the EM12c agent had a JVM memory problem at startup. Let’s try out the emctl clearstate agent command.

emctl clearstate agent

From the Oracle documentation about the clearstate command –

emctl clearstate Clears the state directory contents. The files that are located under $ORACLE_HOME/sysman/emd/state will be deleted if this command is run. The state files are the files which are ready for the agent to convert them into corresponding xml files.

The emctl clearstate agent command produced the same error:

After some research in My Oracle Support for OutOfMemoryError: Java heap space, I found this note:

Duplicate 1952593.1 – EM12c: emctl start agent Fails With ‘ Target Interaction Manager failed at Startup java.lang.OutOfMemoryError: Java heap space’ reported in gcagent_errors.log (Doc ID 1902124.1)

From the MOS note:

  1. Kill any leftover process -> not for me, the agent was no started at all
  2. Move old files from  /agent_inst/sysman/emd/state/* to a new directory -> sounds very interesting
  3. Execute clearstate command -> let’s try it out

The directory u00/app/oracle/product/agent12c/agent_inst/sysman/emd/state/

There was a lot of stuff in this directory – time to move:

emctl clearstate agent – 2nd run

Now the clearstate command has worked without errors.

emctl start agent

The start was successful – fine.


The cleanup of the state directory solved the problem. But why does a emctl clearstate agent command not clean up the state directory as described in the documentation? Aaah, and do’t forget to clean up the moved files. They are not needed anymore. – Create a Storage Container for a Backup in the Cloud

Why to use the Oracle Backup  Cloud storage?

During the instance creation process – the full deployed instance, not for the virtual machine – you can select if you want to store your database local or in the cloud – or both. The benefit to store a instance backup by RMAN in the cloud? Storage redundancy, easy access, full RMAN integration and with a service price of $33 / TB / Month it’s a real storage option.

Here is the step where you can select the backup storage option.


Information Gathering

Before you can use this storage, you have to configure a storage container first. This can be done via REST command for example by a tool like curl. In this example I used curl for Windows. To create a storage container you have to know an authentification token (X-Auth-Token) and the URL (X-Storage-Url)  where the storage is presented. The storage name is composed by the word Storage and your Identity Domain name. For example my domain is called chtrivadis4703, the name of the storage user is Storage-chtrivadis4703.

The GET command:

There are some more lines with connection output text, I have removed then so it’s better readable:

Create a Storage Container

Now we know the X-Auth-Token and the X-Storage-Url, so we can create a new storage container called Backup.

For the verification I used Cloud Berry Explorer for OpenStack Storage ( – the container Backup was created successfully.


Configure the Instance to use the Storage

Fill in the name of the cloud created storage container here and finalize the instance creation process.


If the credentials or the storage container name is wrong, you will get an error message:


Enjoy the new Cloud Storage

The new instance will be automatically configured to use this cloud storage as backup destination. But this container is not only for a backup of the cloud instances, you can use the Oracle Database Cloud Backup Module to backup your on-premise instances into the Oracle cloud. But this will be content in one of my next posts…