Fixing "status pending" in OEM 12C after an agent re-install

I was having trouble with an OEM 12C host agent that wasn't behaving, so I thought the
easiest thing to do would be to uninstall and re-install. Little did I know..

First thing I learnt was that you don't just remove the 12C agent home. No, you need to go to the agent home oui bin directory and run the installer (yes, run the installer to de-install it):

/u01/app/oracle/product/12.1.0.1/core/12.1.0.1.0/oui/bin/runInstaller

Make sure you have an X-Windows server running.

Click on "Installed Products", then select all the "oracle_sysman*" items and de-install them first.

If you try and de-install the agent12c1 or sbin12c1 items you'll get an error. Once all the "oracle_sysman*" ones have been de-installed do the sbin12c1 and then the agent12c.

Now you can rm -rf the agent directory.

Go to the OEM 12C console and re-install the agent in the usual manner (setup, add target, add target manually).

Assuming all goes well, you should see the agent in the OEM console, but it will probably be showing "status pending" with the clock icon. If it isn't, congratulations, stop reading this and go and do something else.

If it is, then you will need to resynchronise the agent and unblock it from OEM.

You will also see that an "emctl status agent" on the host will show no files have been uploaded to the OEM, and that an "emctl pingOMS" will show a "blocked" message. Any attempts to do "emctl upload" will result in a timeout.

The Oracle documentation says you need to resynchronize and unblock the agent but I couldn't find any screenshots that show where to do this, eventually I poked around enough and found it.

Go to the OEM console and select "setup, agents". Click on the "Misconfiguration Alerts and Blocked Agents" tab, you should see the blocked agent there. Note, in V12.0.0.3, the menu has changed to
Setup , Manage Cloud Control and then Agents.



It took me ages to find out how to do the resynchronize - the main agent page has lots of clickable buttions, but none that say "Resynchronize"



Click on the agent to select it and the main host page will open. There is a small "Agent" icon at the top left of the page. Click on that and the drop-down menu will show a "Resynchronization" item. Click on that and "Continue" on the next page. This will submit a job which you can select to check if it works.

 


If it works, it will unblock the agent and it should no longer show up as "status pending".


This is the Oracle documentation : http://docs.oracle.com/cd/E25178_01/doc.1111/e24473/ha_outages.htm#BABHJCED

Management Agent Reinstall Using the Same Port

A Management Agent is monitoring multiple targets. The Agent installation is lost.

1. De-install the Agent Oracle Home using the Oracle Universal Installer.
   
   Note:

   This step is necessary in order to clean up the inventory.
2. Install a new Management Agent or use the Management Agent clone option to reinstall the Management Agent though Enterprise Manager. Specify the same port that was used by the crashed Agent. The location of the install must be same as the previous install.
   The OMS detects that the Management Agent has been re-installed and blocks the Management Agent.
   
3. Initiate Management Agent Resynchronization from the Management Agent homepage.
   All targets in the Management Repository are pushed to the new Management Agent. The Agent is instructed to clear backlogged files and then do a clearstate. The Agent is then unblocked.
   
4. Reconfigure User-defined Metrics if the location of User-defined Metric scripts have changed.
   
5. Verify that the Management Agent is operational and all target configurations have been restored using the following emctl commands:
   

   emctl status agent 
   emctl upload agent 
   

   There should be no errors and no XML files in the backlog.
   

A quick dip into Oracle 12C containers and pluggable databases

I've been playing around with 12C in the evening at home (what a sad, sad life I lead). I had a look through the documentation : http://docs.oracle.com/cd/E16655_01/server.121/e17633/cdbovrvw.htm#CEGEEIJG

This is well worth reading, it explains what all this container and pluggable stuff is all about.

I don't like the way Oracle appears to be at war with DBAs - from Oracle 10g which was billed as "the self-managing database" (Ha!), to this comment in the doco:

"Figure 17-2 depicts the databases in Figure 17-1 after consolidation onto one computer. The DBA team is reduced from five to three, with one CDB administrator managing the CDB while two PDB administrators split management of the PDBs."

Anyway, on with the tale.

When you create the database, you need to specify that it's an old-fashioned one (a non-CDB), or a new-fangled Container Database.

This is either with the "enable pluggable database" clause in the "create database" statement, or by checking the appropriate box in the dbca.

I've not done much investigation, but created a CDB and then a pluggable database within that.

I used the DBCA and checked the box to create it as a container database, and called it ANDYPLG (I should really have called this ANDYCDB but I didn't).

Once created, I then restarted the DBCA and selected 'Manage Pluggable Databases" and then "Create a Pluggable Database" - called it ANDYPLG1.

That was it. I now had an Oracle 12C container database that had a pluggable database inside it.

I didn't get a chance to do too much with it, but found a few things:

The pluggable database has no OS processes.

You can see the databases using the v$services view:

SQL> select name from v$services;

NAME
----------------------------------------------------------------
andyplg1
ANDYPLGXDB
ANDYPLG
SYS$BACKGROUND
SYS$USERS

You switch from the container to the pluggable database with the alter session command:

SQL> alter session set container=andyplg1;

Session altered.

SQL> select name from v$services;

NAME
----------------------------------------------------------------
andyplg1

I then created a new container from the one I'd created earlier - this is one of the big advantages with pluggable databases, the ability to clone them very easily. However, I did notice that the source database has to be read-only:

SQL> create pluggable database andyplg2 from andyplg1;
create pluggable database andyplg2 from andyplg1
*
ERROR at line 1:
ORA-65081: database or pluggable database is not open in read only mode

SQL> alter pluggable database andyplg1 close;

Pluggable database altered.

SQL> alter pluggable database andyplg1 open read only;

Pluggable database altered.

SQL> create pluggable database andyplg2 from andyplg1;

Pluggable database created.

SQL> alter pluggable database andyplg1 close;

Pluggable database altered.

SQL> alter pluggable database andyplg1 open read write;

Pluggable database altered.



SQL> select name from v$services;

NAME
----------------------------------------------------------------
andyplg2
andyplg1
ANDYPLGXDB
ANDYPLG
SYS$BACKGROUND
SYS$USERS

Voila! The andyplg2 was created as a clone of andyplg1.

My next investigations will be around how much control we have around creating the pluggable databases - for instance, look at the datafile names of andyplg1:

/u04/oradata/ANDYPLG/E06734C67F8740C5E043161816ACD7D0/datafile/o1_mf_system_8x1b
hcnw_.dbf

Not very nice.

I'll also look into how to manage resources withing the pluggable databases.