Experiments in Geekery

There's a short story behind my new job but that'll wait for another day... I wanted to write about my experiences installing Oracle Configuration Manager (OCM) to monitor Oracle databases in an Oracle Enterprise Manager 12c (OEM 12c) environment.  It's deceptively simple, but I've wasted more than a day trying to figure out what's going on. The first place you start if you're like me, is finding out that OCM exists!  It's a nice idea; a small bit of software that sits in your Oracle installation, monitors the configuration of the DB, the host, the listeners, anything relevant, and uploads it to Oracle's servers so that if you ever need to raise a service request - it's much easier and the tech support people have more information to help you with. As a concept that is fabulous, but on a practical level, installing OCM on 20 different servers, in 2 locations on each one because it needs installing separately for each product, is not going to ha...

A relatively new OEM12c and Oracle 11.2.0.3 installation, single server. The VM was taken down for a memory upgrade, hurrah!  Turns out that 4GB really isn't quite enough for OEM and the database on the same box to work together nicely.  I mean, it worked OK, but didn't work quickly and often struggled when busy. The server came back up, and the gcstartup script fired, starting OEM.  At this point we're not using clusterware (don't ask me why, I don't know. I'm very used to it and consider it "the norm", but new job, new challenges, right?) and don't have an init script to start up the database on boot, so OMS couldn't talk to the repository.  I started up the database through SQL*Plus thinking "yep, that'll do it". But apparently not.  The log (/u1/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out) was  flooded with "Network error: can't connect to database" type errors. ...

I've recently started a new job as an Oracle DBA!  Long story for another day. As part of taking over control the new database, I spent some time monitoring the authentication failures in the audit log. On this particular (production!) database, this is what I was seeing: User Name OS User Name Date Time DBSNMP oracle 27-May-14 15.14.56 DBSNMP oracle 27-May-14 15.13.56 DBSNMP oracle 27-May-14 15.12.55 DBSNMP oracle 27-May-14 15.11.55 DBSNMP oracle 27-May-14 15.10.55 DBSNMP oracle 27-May-14 15.09.55 And on and on and on.  Every minute a failed connection.  OK maybe not a problem given that the DBSNMP account has been set up by the previous DBA with UNLIMITED connection attempts, but definitely an indication that something is wrong. I went through the following troubleshooting: Determined where the connections were coming from (localhost) Checked if the...