Repeated DBSNMP Authentication Failures

I've recently started a new job as an Oracle DBA!  Long story for another day.

As part of taking over control the new database, I spent some time monitoring the authentication failures in the audit log.

On this particular (production!) database, this is what I was seeing:
User Name OS User Name Date Time
DBSNMP oracle 27-May-14 15.14.56
DBSNMP oracle 27-May-14 15.13.56
DBSNMP oracle 27-May-14 15.12.55
DBSNMP oracle 27-May-14 15.11.55
DBSNMP oracle 27-May-14 15.10.55
DBSNMP oracle 27-May-14 15.09.55

And on and on and on.  Every minute a failed connection.  OK maybe not a problem given that the DBSNMP account has been set up by the previous DBA with UNLIMITED connection attempts, but definitely an indication that something is wrong.

I went through the following troubleshooting:
  • Determined where the connections were coming from (localhost)
  • Checked if there were any cron jobs running at these times (no)
  • Checked if OEM/DB Control was running (yes)
  • Checked whether OEM/DB Control was configured with the correct username/password (it was)
  • Removed the password from targets.xml in $DBCONTROL_HOME/sysman/config/ and reconfigured it through the interface, testing as it went on
  • Checked if there were any jobs in the database or OEM configured incorrectly (none)
  • Scanned through every other process (using ps -ef | less) to see if something else was running that looked out of place
  • Stopped dbconsole to see if the connection attempts stopped ($ORACLE_HOME/bin/emctl stop dbconsole) - they didn't
It was at this point, after restarting dbconsole, that I noticed that there were two emagent processes running which didn't seem right, and on closer inspection one of them had been running since 2013, despite my restart of dbconsole:

PRIMARY-/u1/app/oracle/product/11.2.0/dbhome_1/bin> ps aux | grep emagent
oracle   24731  0.0  0.0 929580  6580 ?        Sl    2013 147:46 /u1/app/oracle/product/11.2.0/dbhome_1/bin/emagent
oracle   26818  0.1  0.2 1327364 43000 pts/3   Sl   14:39   0:03 /u1/app/oracle/product/11.2.0/dbhome_1/bin/emagent
oracle   36234  0.0  0.0 103248   864 pts/3    S+   15:15   0:00 grep emagent
Looking at the output from emctl status agent, I could see which agent process should be running...
PRIMARY-/u1/app/oracle/product/11.2.0/dbhome_1/bin> ./emctl status agent
Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 10.2.0.4.4
[snip]
Agent Process ID  : 26818
Parent Process ID : 26766
I killed the other process (24731) and voila, the connection attempts stopped.

Comments

Post a Comment

Popular posts from this blog

Data Guard with Transparent Application Failover (TAF)

One of the problems with Data Guard is that even after switching/failing over to the standby database, all clients need to be reconfigured to communicate with the new primary. The simplest way to achieve this (if using a centralised tnsnames.ora ), is to modify the file and  add the line " (FAILOVER=YES) ",  and either multiple ( DESCRIPTION=...) or (ADDRESS=...)  lines, one for each of the standby systems, in the order that the client should attempt connections . This tells the client to try each address in turn until one succeeds. Doing so provides basic failover functionality within Oracle Net, and will probably function correctly most of the time.  Though be aware - because both primary and standby can share the same instance name (e.g. the same SID, therefore the same connection string), there is potential for this simple scheme to go wrong. An example in an active data guard environment (where the standby is open read only): a temporary network issue causes all c
Read more

RMAN-05531 During RMAN Duplicate from Active Data Guard Standby

[oracle@target dbs]$ rman auxiliary sys@dup target sys@source [...] RMAN> duplicate target database to DUP from active database db_file_name_convert '/u2/data/PROD','/u2/data/DUP' LOG_FILE_NAME_CONVERT '/u2/data/PROD/onlinelog','/u2/data/DUP' nofilenamecheck; Starting Duplicate Db at 25-FEB-15 using target database control file instead of recovery catalog allocated channel: ORA_AUX_DISK_1 channel ORA_AUX_DISK_1: SID=171 device type=DISK RMAN-00571: =================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ======= RMAN-00571: =================================================== RMAN-03002: failure of Duplicate Db command at 02/25/2015 16:04:44 RMAN-05501: aborting duplication of target database RMAN-05531: a mounted database cannot be duplicated while datafiles are fuzzy Using Oracle 11.2.0.3 (anything below 11.2.0.3.6), this is the error if an attempt is made to duplicate from an activ
Read more

Data pump - "ORA-39786: Number of columns does not match between export and import databases"

Today I was asked if it was possible to import data for a specific table from a data pump export file, which was taken from when a table had a different structure to what it has now. e.g. when the export was taken, there were 10 columns.  Now there are 11. I thought that was simple, and I was sure I'd heard of this being done before.  Simply import using the "CONTENT=DATA_ONLY" parameter.  Right? Apparently not: Centos-/home/oracle/demo> impdp DEMO_USER/yyyyyyy directory=INBOUND dumpfile=demo_data.dmp logfile=impdpdemo.log content=data_only tables=xxxxxxxx Import: Release 10.2.0.4.0 - 64bit Production on Tuesday, 08 July, 2014 14:32:20 Copyright (c) 2003, 2007, Oracle.  All rights reserved. Connected to: Oracle Database 10g Release 10.2.0.4.0 - 64bit Production Master table "DEMO_USER"."SYS_IMPORT_TABLE_01" successfully loaded/unloaded Starting "DEMO_USER"."SYS_IMPORT_TABLE_01":  xxxxxxxx/******** directory=INBOUND dumpfile
Read more