EM12c Agent – java.lang.OutOfMemoryError: Java heap space

After an AIX 7.1 server reboot, there was one agent which did not started. The command emctl start agent resulted in a java.lang.OutOfMemoryError: Java heap space message. In the agent subdirectory some dumpfiles were created:

First I took a look in the agent logfiles to gather more information.

/u00/app/oracle/product/agent12c/agent_inst/sysman/log/emdctlj.log

There are a log of MOS notes available for the search term peer not authentificated, but none of them were helpful.

/u00/app/oracle/product/agent12c/agent_inst/sysman/log/emagent.nohup

Here we see that the EM12c agent had a JVM memory problem at startup. Let’s try out the emctl clearstate agent command.

emctl clearstate agent

From the Oracle documentation about the clearstate command – http://docs.oracle.com/cd/E29597_01/doc.1111/e24473/emctl.htm#r21c1-t27

emctl clearstate Clears the state directory contents. The files that are located under $ORACLE_HOME/sysman/emd/state will be deleted if this command is run. The state files are the files which are ready for the agent to convert them into corresponding xml files.

The emctl clearstate agent command produced the same error:

After some research in My Oracle Support for OutOfMemoryError: Java heap space, I found this note:

Duplicate 1952593.1 – EM12c: emctl start agent Fails With ‚ Target Interaction Manager failed at Startup java.lang.OutOfMemoryError: Java heap space‘ reported in gcagent_errors.log (Doc ID 1902124.1)

From the MOS note:

  1. Kill any leftover process -> not for me, the agent was no started at all
  2. Move old files from  /agent_inst/sysman/emd/state/* to a new directory -> sounds very interesting
  3. Execute clearstate command -> let’s try it out

The directory u00/app/oracle/product/agent12c/agent_inst/sysman/emd/state/

There was a lot of stuff in this directory – time to move:

emctl clearstate agent – 2nd run

Now the clearstate command has worked without errors.

emctl start agent

The start was successful – fine.

Summary

The cleanup of the state directory solved the problem. But why does a emctl clearstate agent command not clean up the state directory as described in the documentation? Aaah, and do’t forget to clean up the moved files. They are not needed anymore.