Redo logs on Physical Standby

Redo logs on Physical Standby Online redo logs on a physical standby database is not really required. For real-time apply , standby redo logs are a must on standby site.

Standby redo logs are optional at the source site, but always advisable to be prepared for switchover or failovers.

If DB_CREATE_ONLINE_LOG_DEST_1 is not set and
DB_CREATE_FILE_DEST and DB_RECOVERY_FILE_DEST is specified,
the redo logs are created multiplexed each under DB_CREATE_FILE_DEST and DB_RECOVERY_FILE_DEST

If you need to move the redo log to another diskgroup or another filesystem on standby database, these are the steps

--cancel the recovery
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

--change the Standby File Management
ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT='MANUAL';

--Clear the redo logs
ALTER DATABASE CLEAR LOGFILE GROUP 2;

--Drop the redo logs
ALTER DATABASE DROP LOGFILE GROUP 2;

--Add the new logs at the correct locations
ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 2 ('+REDO_DG1','+REDO_DG2') size 1024M;

-- If you need to add standby redo logs, do so...
ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 5 ('+REDO_DG','+REDO_DG2') size 1024M;

Put the standby database back into action

ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT='AUTO';

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;

EMACS and SQLPLUS

I have recently discovered EMACS thanks to the unfaultering effort by my co-brother Roy. I have only one word for EMACS- fan-"bleedy"-tastic.
Although not very intuitive, it is immensely powerful and extremely great to improve productivity.

I will break this blog into three sections for easier reading.

Section 1 - How does this fit into my work?

---

Remember the days spent with a SQL in a notepad, and server has no scripts. The day drags on with you running Ctrl+C, Ctrl+V , then right click into the Putty Console.


Now here is how it happens in Emacs.

So start the SQL-Plus buffer in Emacs. This should be inbuilt inside the 24.2 version onwards.

M-x sql-oracle
It should prompt for a userid, password and host name. Hostnames is the TNS names entry in your local tnsnames.ora file.

If you get the below error, like I did
ORA-12557: TNS:protocol adapter not loadable

Set ORACLE_HOME and PATH to same Oracle home(I have two Oracle Homes in my laptop) .Restart EMacs
Try M-x sql-oracle, again
This time hopefully you should get connected.
You are on the command prompt so you try any of the SQLs.

Now the interesting bit of working efficiently.
C-X, C-f

Then enter a existing or new SQL file name.

Your screen should split with the SQL file and the SQLPlus buffer in another.

Use C- to set the mark and move the keys to select the whole SQL

To send the selected SQL into SQLPlus buffer use the C-C C-r

Section 2 - Errors I encountered

---

Some of the errors I encountered were --

If you get the below error,

No SQL process started

You can try below

M-x sql-set-sqli-buffer
select *SQL* ( should be the default)

That lead me to another error

There is not suitable SQLi buffer

To fix this error, note the bottom of the opened SQL file. The status bar should say SQL[Oracle].

If it says SQL[ANSI], it needs to be changed by doing

M-x sql-set-product
key in "oracle"

Now the "No SQL process started" error can be fixed by using the M-x sql-set-sqli-buffer like above.

Now use C-X C-r to send the SQL from the file to the SQL buffer.

Section 3 - Couple of other hacks.

---

You must have noticed that pressing the backspace deletes the "SQL>" prompt. To fix this , add this to your .emacs file

(setq comint-prompt-regexp "^SQL>")

(custom-set-variables
'(comint-prompt-read-only t)
'(comint-use-prompt-regexp t))
(custom-set-faces)

---

PS:

This blog was written in EMACS

Roy's awesome blog on technology can be found here

DataGuard ReadOnly to Apply and back

Dataguard and Reporting Before the days of Active Dataguard, were the days of read-only reporting off the physical standby. It pains to see data-centers with production like hardware and unused physical standby databases doing nothing but receive redo content once every few hours. Active Dataguard is a boon in that way.

In older versions, reporting environments typically received all the archive logs through the day and then, applied them overnight. The database is available through the day as read only , for querying and taken offline once a day for applying the logs and bringing back online to read-only mode.

Below is a shell script to do exactly that.

Its self explanatory

#!/bin/ksh
################################################################
################################################################
########## AUTHOR          : Harris
########## DATE            : 13 Aug 2012
########## VERSION         : 1.0
########## DESCRIPTION     :This script is used to stop physical
#### standby in readonly mode, apply the logs and bring back
##### online
################################################################
# VERSION |    DATE    |   Author   | DETAIL OF CHANGES
#---------------------------------------------------------------
# V0.1    | 08/13/2012 |Harris Baskaran |Initial Script
################################################################
################################################################
set -x
ORATAB=/var/opt/oracle/oratab
##Entries to update##########
export ORACLE_SID=STANDBYTNS
export CONNECTSTR=dg/dg@PRODTNS
export ENVIRON=DB service name
#####Review before editing parameters below this

export NLS_DATE_FORMAT='Mon DD YYYY HH24:MI:SS'
export ORACLE_HOME=`grep -i "$ORACLE_SID:" $ORATAB | grep -v "^#" | grep -v "^\*"| grep ":" | awk -F":" '{print $2}'`
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/ctx/lib:$ORACLE_HOME/lib:$LD_LIBRARY_PATH

touch error.log
touch shutdown_report.log
touch apply_report.log
touch check_report.log
touch switch2ro.log
touch handshake.log


## Email functionality

mail_it_to_all()
{
#Email the report
SEND_DATE=`date`
RECIPIENTS=dba@company.com
MESSAGE_TITLE="Reporting DB ${ORACLE_SID} on ${ENVIRON} ${SEND_DATE}"
/usr/bin/mailx -s "${MESSAGE_TITLE}" ${RECIPIENTS}  << EOF
${SEND_DATE}
`cat error.log`
`cat shutdown_report.log`
`cat apply_report.log`
`cat check_report.log`
`cat switch2ro.log`
`cat handshake.log`
EOF
}


##Find Sequence of logs shipped
log_shipping_check()
{
sqlplus -s -l / as sysdba  << LOGSHIP
whenever sqlerror exit sql.sqlcode;
whenever oserror exit FAILURE
set heading off
SELECT SEQUENCE# FROM V\$ARCHIVED_LOG WHERE FIRST_TIME = (SELECT MAX(FIRST_TIME) FROM V\$ARCHIVED_LOG GROUP BY THREAD#);
exit
LOGSHIP
}

## function to get Production Sequence
runsqlprod()
{
sqlplus -s -l ${CONNECTSTR}  << PRDCHK
whenever sqlerror exit sql.sqlcode;
whenever oserror exit FAILURE
set heading off
SELECT distinct SEQUENCE# FROM V\$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V\$ARCHIVED_LOG GROUP BY THREAD#);
exit
PRDCHK
}

## function to get Standby Sequence
runsqlstdby()
{
sqlplus -s -l / as sysdba  << STDCHK
whenever sqlerror exit sql.sqlcode;
whenever oserror exit FAILURE
set heading off
select max(sequence#) from V\$ARCHIVED_LOG where applied='YES';
exit
STDCHK
}

shutdown_db()
{
sqlplus -s -l / as sysdba <<E1
whenever sqlerror exit sql.sqlcode;
whenever oserror exit FAILURE
spool shutdown_report.log
select name, open_mode from V\$database;
select username, machine from v\$session where status='ACTIVE';
shutdown immediate;
spool off
exit
E1
}

apply_logs()
{
sqlplus -s -l / as sysdba <<E2
whenever sqlerror exit sql.sqlcode;
whenever oserror exit FAILURE
spool apply_report.log
startup nomount;
alter database mount standby database;
recover managed standby database using current logfile disconnect from session;
spool off
exit
E2
}


#### custom handshake function with whatever reporting job you run
handshake()
{
sqlplus -s -l / as sysdba <<E4
whenever sqlerror exit sql.sqlcode;
whenever oserror exit FAILURE
##Replace with code here
##@handshake.sql
exit
E4
}

switch_to_ro()
{
sqlplus -s -l / as sysdba <<E2
whenever sqlerror exit sql.sqlcode;
whenever oserror exit FAILURE
spool switch2ro.log
recover managed standby database cancel;
alter database open read only;
spool off
exit
E2
}



##Find out if the logs are shipped already, if not no point continuing

PRODSEQ=`runsqlprod`
SHIPPED=`log_shipping_check`

if [ ${SHIPPED} -lt ${PRODSEQ} ]
then
echo "The "${ENVIRON}" Reporting and Source Logs are not shipped through the day. Need DBA intervention" >> error.log
echo "Production sequence is "${PRODSEQ} >> error.log
echo "Standby Sequence shipped is "${SHIPPED} >> error.log
mail_it_to_all()
exit 1
fi




####Shutdown the reporting to allow log application

shutdown_db

if grep -c 'ORA-' shutdown_report.log
then
echo ${ENVIRON}"-Error in bringing down reporting database..."
mail_it_to_all
exit 1
else
echo ${ENVIRON}"-Reporting Shutdown..."
fi


####Apply the archive logs and bring to current

apply_logs

if grep -c 'ORA-' apply_report.log
then
echo ${ENVIRON}"-Error in applying the logs on reporting database..."
mail_it_to_all
exit 1
else
echo ${ENVIRON}"-Reporting Logs Applying..."
fi




####Ensure it is synched with Production



### Sleep 15 minutes 10 times until Standby catches up with Prod site
### Increase freq as required


for i in 1 2 3 4 5 6 7 8 9 10
do
STDBSEQ=`runsqlstdby`
if [ ${STDBSEQ} -lt ${PRODSEQ} ]
then
sleep 60
else
break
fi
done


## If catch up still doesnt happen, send email and die!
if [ ${STDBSEQ} -lt ${PRODSEQ} ]
then
echo "The ${ENVIRON} Reporting and Source could not be synched. Need DBA intervention" >> error.log
echo "Production sequence is ${PRODSEQ} " >> error.log
echo "Standby Sequence is ${STDBSEQ} " >> error.log
mail_it_to_all()
exit 1
EEOF
fi

## Bring to read only

switch_to_ro

####Handshake with Application

#handshake()

if grep -c 'ORA-' handshake.log
then
echo ${ENVIRON}"-Error in handshaking with reporting module..."
mail_it_to_all
exit 1
else
echo ${ENVIRON}"-Reporting handshake done..."
fi

mail_it_to_all
mkdir `date "+%m%d%y"`
/usr/bin/mv *.log `date "+%m%d%y"`\/.

SCN headroom and CVE-2012-0082

CVE-2012-0082 is a security flaw in Oracle Databases; which manifests in some scenarios which can significantly increase the rate SCN increases. InfoWorld also has documented the problem in details - InfoWorld(http://www.infoworld.com/d/security/fundamental-oracle-flaw-revealed-184163-0)
Oracle’s CPU2012Jan addresses this problem. Oracle Document 1374524.1

To understand the problem it is necessary to define SCN headroom.
The SCN value is an ever increasing value for a database. The threshold for this SCN value is also ever increasing number too. The max limit of SCN is defined by the number of seconds past the year 1988 X 16k, hence as time passes this value also keeps increasing. This is the soft-limit. There is also a hard limit of 218 trillion.

It is also necessary to understand that when a DB link operation between two databases, Oracle automatically selects the higher SCN out of the two databases and syncs them between the two databases.

--On first database
SQL> select name from V$database;

NAME
---------
Database1

SQL> select dbms_flashback.get_system_change_number from dual;

GET_SYSTEM_CHANGE_NUMBER
------------------------
398829972857 -- note the SCN number



--On second database
SQL> select name from V$database;

NAME
---------
Database2

SQL> select dbms_flashback.get_system_change_number from dual;

GET_SYSTEM_CHANGE_NUMBER
------------------------
391409782850


SQL> select sysdate from dual@Database1.WORLD; -- DB link query

SYSDATE
---------
19-JAN-12



SQL> select dbms_flashback.get_system_change_number from dual;

GET_SYSTEM_CHANGE_NUMBER
------------------------
398829973230 -- notice the significant increase for synchronizing the SCNs

InfoWorld also documents that Oracle Hot backups can significantly increase the scn generation. To find the headroom at any given time use the below query.

select
version,
to_char(SYSDATE,'YYYY/MM/DD HH24:MI:SS') CURR_TIME,
((((
((to_number(to_char(sysdate,'YYYY'))-1988)*12*31*24*60*60) + ((to_number(to_char(sysdate,'MM'))-1)*31*24*60*60) +
(((to_number(to_char(sysdate,'DD'))-1))*24*60*60) + (to_number(to_char(sysdate,'HH24'))*60*60) +
(to_number(to_char(sysdate,'MI'))*60) + (to_number(to_char(sysdate,'SS')))) * (16*1024))
- dbms_flashback.get_system_change_number)/ (16*1024*60*60*24)) scn_headroom
from v$instance;

See Oracle document : ID 1393363.1
Reading through Oracle’ patch if the headroom is less than 62, urgent attention is required. i.e. Apply patch immediately and contact Oracle.
To ensure that the headroom is not being reached rapidly, run the below query.

select max(first_change#)-min(first_change#), to_char(first_time,'YYYY-MM-DD') from V$log_history
group by to_char(first_time,'YYYY-MM-DD')
order by 2 desc;

Sometimes there may be a significant jump in the SCN which can be associated with a DBlink activity. Hot backups can also significantly increase the SCN generation rate as detailed by InfoWorld
If you notice that the SCN rate is very high there is plenty to worry about. If the high SCN rate database is interconnected via DBlink with other databases it can bring down a lot of databases together.
Along with the patch, Oracle have also added a hidden parameter _external_scn_rejection_threshold_hours which needs to be set to 24. There is not much details about this hidden parameter , but it might have something to do with the distributed transaction SCN synch.
As always please ensure you have read and understood the Oracle documentation before applying.

Quiescing a Database

There are times when a DBA needs to perform work on the database that requires no other ACTIVE sessions. This is very common is development databases, where a DBA tries to run a package or a modify a table when the application is till running.
There are also frequent occurances where too much IT support folks time is spent in bringing lot of application servers down to perform a deployment or some database maintainence. That is where Quiescing comes in.
To put a database(run on one instance and it affects all the instances of the RAC database).

ALTER SYSTEM QUIESCE RESTRICTED;

Quiescing puts the database in mode that does not allow any further connections to become ACTIVE state. It would however wait for all ACTIVE connections to become INACTIVE or terminate. Gladly Oracle provides a query to find sessions which are stopping it from enter into QUIESCE mode.

select bq.sid, ss.user, ss.osuser, ss.type, ss.program
from v$blocking_quiesce bq, v$session ss
where bq.sid = ss.sid;

Once in QUIESCE mode, DBA connections are the only ones that are allowed to be in ACTIVE state. The DBA can then easily lock any object to perform his tasks.

Once done to release the QUIESCE;
the DBA runs

ALTER SYSTEM UNQUIESCE;

What ever you do DBAs, for heaven sake just dont bounce that db unnecessarily.

Adaptive Cursor Sharing

Oracle 11g has introduced the new mode of cursor sharing to handle bind peeking.
Bind-peeking is the occurrence of shared cursors leading to non-optimal explain plans.
For eg.

A SQL "select name from emp where dept=:B". This SQL is executed from a package which is called multiple times for different values.
On the first execution, the cursor are cached in the shared pool. The first execution is

'select name from emp where dept=5'

and returns 3 rows. This execution plan involves a index which is optimal give that only 3 out of 253 rows are returned.

On the second execution, the cached cursor gets reused.

'select name from emp where dept=10'

But returns 250 rows out of 253 rows in the table. But sadly the same execution plan gets called because the cursor is shared.

In 11g, Oracle uses an adaptive cursor sharing where Oracle creates child cursors for the matching cursors in the shared pool. This then checks for any optimizations that can be done to plan apart from the parent cursor.
Hence when upgrading its necessary to have some extra memory allocations for the shared pool. In Oracle 11g, cursor_sharing should only be used in two modes EXACT and FORCE.

EXACT mode, this is the default in Oracle 11g. Oracle creates child cursors for SQLs which differ in bind variables. This is desirable approach. The application is making best use of bind variables and Oracle is making sure there is no bind peeking.

In FORCE mode, Oracle forces reuse of the cursors. This is best for legacy applications bind variables are not effectively used.
In this mode Oracle, forces SQLs with similar literal and binds to reuse the same cursors. It does not generate child cursors for different bind variables.

The SIMILAR mode although available in Oracle 11g is not recommended as it forces Oracle to replace literals with binds and allow similar child cursors. Oracle also says it heavily impacts the number of child cursor creation and upsets the optimizer operations.

Although there is not explicit child cursor limit(can be controlled with _ parameter), they can fill up your shared pool quiet rapidly with one bad SQL. Oracle reckons that excessive child cursors may be more detrimental to performance than using separate cursors. We noticed that SIMILAR floods the shared pool and leads to some bugs like Bug 11657468 - Excessive mutex waits with adaptive cursor sharing .

Thanks to Senthil for assisting in the research

IOT and Upgrade

IOTs created when the DB is lesser than 9.2 compatible and then later upgraded to 10g and above, they become corrupt. We noticed this problem in a different area; in Advanced Queues.
Queue tables by default have some underlying attached to them.

Oracle documents them as
AQ$_"queue_table"_I (the dequeue IOT)
AQ$_"queue_table"_T (the time-management IOT)
AQ$_"queue_table"_S (the subscriber table)
AQ$_"queue_table"_H (the history IOT)

After an upgrade we noticed that the queues just would not function until they were dropped and recreated.
This seemed a little ruthless so we then exported the queue tables along with these IOTS.
Then dropped the queues and tables in the upgraded system and imported them.
Easy fix :)