Fixing "status pending" in OEM 12C after an agent re-install

I was having trouble with an OEM 12C host agent that wasn't behaving, so I thought the
easiest thing to do would be to uninstall and re-install. Little did I know..

First thing I learnt was that you don't just remove the 12C agent home. No, you need to go to the agent home oui bin directory and run the installer (yes, run the installer to de-install it):

/u01/app/oracle/product/12.1.0.1/core/12.1.0.1.0/oui/bin/runInstaller

Make sure you have an X-Windows server running.

Click on "Installed Products", then select all the "oracle_sysman*" items and de-install them first.

If you try and de-install the agent12c1 or sbin12c1 items you'll get an error. Once all the "oracle_sysman*" ones have been de-installed do the sbin12c1 and then the agent12c.

Now you can rm -rf the agent directory.

Go to the OEM 12C console and re-install the agent in the usual manner (setup, add target, add target manually).

Assuming all goes well, you should see the agent in the OEM console, but it will probably be showing "status pending" with the clock icon. If it isn't, congratulations, stop reading this and go and do something else.

If it is, then you will need to resynchronise the agent and unblock it from OEM.

You will also see that an "emctl status agent" on the host will show no files have been uploaded to the OEM, and that an "emctl pingOMS" will show a "blocked" message. Any attempts to do "emctl upload" will result in a timeout.

The Oracle documentation says you need to resynchronize and unblock the agent but I couldn't find any screenshots that show where to do this, eventually I poked around enough and found it.

Go to the OEM console and select "setup, agents". Click on the "Misconfiguration Alerts and Blocked Agents" tab, you should see the blocked agent there. Note, in V12.0.0.3, the menu has changed to
Setup , Manage Cloud Control and then Agents.



It took me ages to find out how to do the resynchronize - the main agent page has lots of clickable buttions, but none that say "Resynchronize"



Click on the agent to select it and the main host page will open. There is a small "Agent" icon at the top left of the page. Click on that and the drop-down menu will show a "Resynchronization" item. Click on that and "Continue" on the next page. This will submit a job which you can select to check if it works.

 


If it works, it will unblock the agent and it should no longer show up as "status pending".


This is the Oracle documentation : http://docs.oracle.com/cd/E25178_01/doc.1111/e24473/ha_outages.htm#BABHJCED

Management Agent Reinstall Using the Same Port

A Management Agent is monitoring multiple targets. The Agent installation is lost.

1. De-install the Agent Oracle Home using the Oracle Universal Installer.
   
   Note:

   This step is necessary in order to clean up the inventory.
2. Install a new Management Agent or use the Management Agent clone option to reinstall the Management Agent though Enterprise Manager. Specify the same port that was used by the crashed Agent. The location of the install must be same as the previous install.
   The OMS detects that the Management Agent has been re-installed and blocks the Management Agent.
   
3. Initiate Management Agent Resynchronization from the Management Agent homepage.
   All targets in the Management Repository are pushed to the new Management Agent. The Agent is instructed to clear backlogged files and then do a clearstate. The Agent is then unblocked.
   
4. Reconfigure User-defined Metrics if the location of User-defined Metric scripts have changed.
   
5. Verify that the Management Agent is operational and all target configurations have been restored using the following emctl commands:
   

   emctl status agent 
   emctl upload agent 
   

   There should be no errors and no XML files in the backlog.