Recovering a dataguard standby after downtime

Recently, we had a planned weekend of standby downtime on a production system from Friday afternoon to Monday morning.  This was all fine, and when the database came back up I expected it to just do it's FAL thing (Fetch Archive Log) and catch up automatically.

Except I forgot 2 important things:
  1. Backups on this system are taken on the primary
  2. Archive log deletion policy on the primary is set to "BACKED UP 1 TIMES TO DISK"
Thus when I came to start the redo apply on Monday morning, I found the archive logs had been deleted from the FRA on the primary and the standby couldn't recover automatically.

That was when I started to feel that sinking feeling you get when you suddenly realise that you could have avoided this.

I found out from the alert log which archive logs were needed.  This shows in the alert log as something like this:
FAL[client]: Failed to request gap sequence 
GAP - thread 1 sequence 27658-27658
  DBID 123456789 branch 823623081
FAL[client]: All defined FAL servers have been attempted.
I used RMAN (i.e.: 'list backup of archivelog until time '14-SEP-2015') to find out which backup pieces these archive logs were in.  I was missing Friday evening through to Saturday afternoon in the primary FRA, all of these would be in Friday & Saturday night's backups.

At this point, I did two things:
  1. I raised a severity 1 SR with Oracle support.  This was not really because I wanted help with the recovery but because I felt it was important to cover my arse
  2. I copied the backup pieces containing the relevant archivelogs to the standby
I could have used incremental backups to roll the database forward to a point where it could recover itself, however decided to stick to what I knew would work.

At the RMAN prompt:
RMAN> catalog start with '/path/to/backup/piece/with/archive/logs';
[...]
RMAN> recover database;
At this point, I got the call from Oracle support.  I explained I'd started the recovery and things seemed to be going OK and they was happy to leave the support call open until recovery had completed.

Recovery finished with the usual error about further recovery needed.  I opened the database read-only and started the managed recovery process ('ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;')

Things were, finally, back to normal.

Comments

Post a Comment

Popular posts from this blog

Data Guard with Transparent Application Failover (TAF)

One of the problems with Data Guard is that even after switching/failing over to the standby database, all clients need to be reconfigured to communicate with the new primary. The simplest way to achieve this (if using a centralised tnsnames.ora ), is to modify the file and  add the line " (FAILOVER=YES) ",  and either multiple ( DESCRIPTION=...) or (ADDRESS=...)  lines, one for each of the standby systems, in the order that the client should attempt connections . This tells the client to try each address in turn until one succeeds. Doing so provides basic failover functionality within Oracle Net, and will probably function correctly most of the time.  Though be aware - because both primary and standby can share the same instance name (e.g. the same SID, therefore the same connection string), there is potential for this simple scheme to go wrong. An example in an active data guard environment (where the standby is open read only): a temporary network issue causes all c
Read more

RMAN-05531 During RMAN Duplicate from Active Data Guard Standby

[oracle@target dbs]$ rman auxiliary sys@dup target sys@source [...] RMAN> duplicate target database to DUP from active database db_file_name_convert '/u2/data/PROD','/u2/data/DUP' LOG_FILE_NAME_CONVERT '/u2/data/PROD/onlinelog','/u2/data/DUP' nofilenamecheck; Starting Duplicate Db at 25-FEB-15 using target database control file instead of recovery catalog allocated channel: ORA_AUX_DISK_1 channel ORA_AUX_DISK_1: SID=171 device type=DISK RMAN-00571: =================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ======= RMAN-00571: =================================================== RMAN-03002: failure of Duplicate Db command at 02/25/2015 16:04:44 RMAN-05501: aborting duplication of target database RMAN-05531: a mounted database cannot be duplicated while datafiles are fuzzy Using Oracle 11.2.0.3 (anything below 11.2.0.3.6), this is the error if an attempt is made to duplicate from an activ
Read more

Data pump - "ORA-39786: Number of columns does not match between export and import databases"

Today I was asked if it was possible to import data for a specific table from a data pump export file, which was taken from when a table had a different structure to what it has now. e.g. when the export was taken, there were 10 columns.  Now there are 11. I thought that was simple, and I was sure I'd heard of this being done before.  Simply import using the "CONTENT=DATA_ONLY" parameter.  Right? Apparently not: Centos-/home/oracle/demo> impdp DEMO_USER/yyyyyyy directory=INBOUND dumpfile=demo_data.dmp logfile=impdpdemo.log content=data_only tables=xxxxxxxx Import: Release 10.2.0.4.0 - 64bit Production on Tuesday, 08 July, 2014 14:32:20 Copyright (c) 2003, 2007, Oracle.  All rights reserved. Connected to: Oracle Database 10g Release 10.2.0.4.0 - 64bit Production Master table "DEMO_USER"."SYS_IMPORT_TABLE_01" successfully loaded/unloaded Starting "DEMO_USER"."SYS_IMPORT_TABLE_01":  xxxxxxxx/******** directory=INBOUND dumpfile
Read more