Repeated DBSNMP Authentication Failures

I've recently started a new job as an Oracle DBA!  Long story for another day.

As part of taking over control the new database, I spent some time monitoring the authentication failures in the audit log.

On this particular (production!) database, this is what I was seeing:
User Name OS User Name Date Time
DBSNMP oracle 27-May-14 15.14.56
DBSNMP oracle 27-May-14 15.13.56
DBSNMP oracle 27-May-14 15.12.55
DBSNMP oracle 27-May-14 15.11.55
DBSNMP oracle 27-May-14 15.10.55
DBSNMP oracle 27-May-14 15.09.55

And on and on and on.  Every minute a failed connection.  OK maybe not a problem given that the DBSNMP account has been set up by the previous DBA with UNLIMITED connection attempts, but definitely an indication that something is wrong.

I went through the following troubleshooting:
  • Determined where the connections were coming from (localhost)
  • Checked if there were any cron jobs running at these times (no)
  • Checked if OEM/DB Control was running (yes)
  • Checked whether OEM/DB Control was configured with the correct username/password (it was)
  • Removed the password from targets.xml in $DBCONTROL_HOME/sysman/config/ and reconfigured it through the interface, testing as it went on
  • Checked if there were any jobs in the database or OEM configured incorrectly (none)
  • Scanned through every other process (using ps -ef | less) to see if something else was running that looked out of place
  • Stopped dbconsole to see if the connection attempts stopped ($ORACLE_HOME/bin/emctl stop dbconsole) - they didn't
It was at this point, after restarting dbconsole, that I noticed that there were two emagent processes running which didn't seem right, and on closer inspection one of them had been running since 2013, despite my restart of dbconsole:

PRIMARY-/u1/app/oracle/product/11.2.0/dbhome_1/bin> ps aux | grep emagent
oracle   24731  0.0  0.0 929580  6580 ?        Sl    2013 147:46 /u1/app/oracle/product/11.2.0/dbhome_1/bin/emagent
oracle   26818  0.1  0.2 1327364 43000 pts/3   Sl   14:39   0:03 /u1/app/oracle/product/11.2.0/dbhome_1/bin/emagent
oracle   36234  0.0  0.0 103248   864 pts/3    S+   15:15   0:00 grep emagent
Looking at the output from emctl status agent, I could see which agent process should be running...
PRIMARY-/u1/app/oracle/product/11.2.0/dbhome_1/bin> ./emctl status agent
Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 10.2.0.4.4
[snip]
Agent Process ID  : 26818
Parent Process ID : 26766
I killed the other process (24731) and voila, the connection attempts stopped.

Comments

Post a Comment

Popular posts from this blog

APEX, SERT, and EPG

APEX, SERT, and EPG Application Express, Security Evaluation and Recommendation Tool, and the Embedded PL/SQL Gateway My organisation has only just started seriously working with APEX and have recently launched our first internal production app (that was a challenge... I was told that this app needed to be in production in 2 days and nobody had bothered to check with me what was needed to make this happen).  I've learned a lot about APEX in the past few weeks and I've got a lot more to go. I recently came across SERT ( https://github.com/OraOpenSource/apex-sert ) and it sounds like a really nifty tool.  Today I found some time to try it out and see if our dev team might find it useful.  I decided to install it in our dev database.  This uses the EPG rather than ORDS. The install didn't go so well.  The first install failed near the end because I think I used a lowercase 'y' for a response somewhere and it got confused, bombing out with an ORA-06502...
Read more

Data Guard with Transparent Application Failover (TAF)

One of the problems with Data Guard is that even after switching/failing over to the standby database, all clients need to be reconfigured to communicate with the new primary. The simplest way to achieve this (if using a centralised tnsnames.ora ), is to modify the file and  add the line " (FAILOVER=YES) ",  and either multiple ( DESCRIPTION=...) or (ADDRESS=...)  lines, one for each of the standby systems, in the order that the client should attempt connections . This tells the client to try each address in turn until one succeeds. Doing so provides basic failover functionality within Oracle Net, and will probably function correctly most of the time.  Though be aware - because both primary and standby can share the same instance name (e.g. the same SID, therefore the same connection string), there is potential for this simple scheme to go wrong. An example in an active data guard environment (where the standby is open read only): a temporary network issue cau...
Read more

Error upgrading 11.2.0.3 to 11.2.0.4: ORA-00942 & ORA-06512

During an upgrade of an 11.2.0.3 DB to 11.2.0.4 using DBUA, this error popped up: Errors - Oracle Workspace Manager . ORA-00942: table or view does not exist . ORA-06512: at "WMSYS.OWM_MIG_PKG", line 1575 . ORA-06512: at "WMSYS.OWM_MIG_PKG", line 1592 . ORA-06512: at line 1 . VALID 11.2.0.1.0 00:00:39 This error is referenced in MOS note  1051991.1 and can be ignored: If DBA_REGISTRY shows the status of (in this case) Workspace Manager is VALID, then this error is safe to ignore. Click Ignore and the upgrade will continue.
Read more