Tuesday, September 13, 2011

Adding a node to a 11gr2 cluster

Pre-checks


cluvfy stage -pre nodeadd -n NODEc -verbose
cluvfy comp peer -refnode NODEb -n NODEc -verbose
cluvfy stage -post hwos -n NODEc -verbose



Only if the above return successful , we can proceed with the next steps. In the above, space comparisions mismatches can be ignored.
You may get SEVERE error:Java NullPointerException if firewall is turned on. Remember to turn
Remember to set XDISPLAY before you start

First the clusterware has to be replicated over to the new node.


oracle@NODEa:/apps/product/11gr2/grid/oui/bin ./addNode.sh "CLUSTER_NEW_NODES={NODEc}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={NODEc-vip}"

Performing pre-checks for node addition

Checking node reachability...
Node reachability check passed from node "NODEa"


Checking user equivalence...
User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Check: Node connectivity for interface "bge0"
Node connectivity passed for interface "bge0"

Node connectivity check passed


Checking CRS integrity...

CRS integrity check passed

Checking shared resources...

Checking CRS home location...
The location "/apps/product/11gr2/grid" is not shared but is present/creatable on all nodes
Shared resources check for node addition passed


Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Check: Node connectivity for interface "bge0"
Node connectivity passed for interface "bge0"

Check: Node connectivity for interface "bge1"
Node connectivity passed for interface "bge1"

Check: Node connectivity for interface "ce1"
Node connectivity passed for interface "ce1"

Node connectivity check passed

Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "NODEc:/var/tmp/"
Free disk space check passed for "NODEa:/var/tmp/"
Check for multiple users with UID value 2001 passed
User existence check passed for "oracle"
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "project.max-sem-ids"
Kernel parameter check passed for "process.max-sem-nsems"
Kernel parameter check passed for "project.max-shm-memory"
Kernel parameter check passed for "project.max-shm-ids"
Kernel parameter check passed for "tcp_smallest_anon_port"
Kernel parameter check passed for "tcp_largest_anon_port"
Kernel parameter check passed for "udp_smallest_anon_port"
Kernel parameter check passed for "udp_largest_anon_port"
Package existence check passed for "SUNWarc-...( sparc)"
Package existence check passed for "SUNWbtool-...( sparc)"
Package existence check passed for "SUNWhea-...( sparc)"
Package existence check passed for "SUNWlibm-...( sparc)"
Package existence check passed for "SUNWlibms-...( sparc)"
Package existence check passed for "SUNWsprot-...( sparc)"
Package existence check passed for "SUNWtoo-...( sparc)"
Package existence check passed for "SUNWi1of-...( sparc)"
Package existence check passed for "SUNWi1cs-...( sparc)"
Package existence check passed for "SUNWi15cs-...( sparc)"
Package existence check passed for "SUNWxwfnt-...( sparc)"
Package existence check passed for "SUNWlibC-...( sparc)"
Package existence check passed for "SUNWcsl-...( sparc)"
Operating system patch check passed for "Patch 119963-14"
Operating system patch check passed for "Patch 120753-06"
Operating system patch check passed for "Patch 139574-03"
Operating system patch check passed for "Patch 141414-02"
Check for multiple users with UID value 0 passed
Current group ID check passed

Checking OCR integrity...

OCR integrity check passed

Checking Oracle Cluster Voting Disk configuration...

Oracle Cluster Voting Disk configuration check passed
Time zone consistency check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
NTP Configuration file check passed

Checking daemon liveness...
Liveness check passed for "xntpd"
Check for NTP daemon or service alive passed on all nodes

NTP daemon slewing option check passed

NTP daemon's boot time configuration check for slewing option passed

NTP common Time Server Check started...
Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started...
Clock time offset check passed

Clock synchronization check using Network Time Protocol(NTP) passed


User "oracle" is not part of "root" group. Check passed
Checking consistency of file "/etc/resolv.conf" across nodes

File "/etc/resolv.conf" does not have both domain and search entries defined
domain entry in file "/etc/resolv.conf" is consistent across nodes
search entry in file "/etc/resolv.conf" is consistent across nodes
All nodes have one search entry defined in file "/etc/resolv.conf"
The DNS response time for an unreachable node is within acceptable limit on all nodes

File "/etc/resolv.conf" is consistent across nodes

Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.

Pre-check for node addition was successful.
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 13193 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
Oracle Universal Installer, Version 11.2.0.2.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.


Performing tests to see whether nodes NODEb,NODEc are available
............................................................... 100% Done.

...
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /apps/product/11gr2/grid
New Nodes
Space Requirements
New Nodes
NODEc
/: Required 11.39GB : Available 33.37GB
Installed Products
Product Names
Oracle Grid Infrastructure 11.2.0.2.0
Sun JDK 1.5.0.24.08
Installer SDK Component 11.2.0.2.0
Oracle One-Off Patch Installer 11.2.0.0.2
Oracle Universal Installer 11.2.0.2.0
Oracle USM Deconfiguration 11.2.0.2.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Enterprise Manager Common Core Files 10.2.0.4.3
Oracle DBCA Deconfiguration 11.2.0.2.0
Oracle RAC Deconfiguration 11.2.0.2.0
Oracle Quality of Service Management (Server) 11.2.0.2.0
Installation Plugin Files 11.2.0.2.0
Universal Storage Manager Files 11.2.0.2.0
Oracle Text Required Support Files 11.2.0.2.0
Automatic Storage Management Assistant 11.2.0.2.0
Oracle Database 11g Multimedia Files 11.2.0.2.0
Oracle Multimedia Java Advanced Imaging 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
Oracle Multimedia Locator RDBMS Files 11.2.0.2.0
Oracle Core Required Support Files 11.2.0.2.0
Bali Share 1.1.18.0.0
Oracle Database Deconfiguration 11.2.0.2.0
Oracle Quality of Service Management (Client) 11.2.0.2.0
Expat libraries 2.0.1.0.1
Oracle Containers for Java 11.2.0.2.0
Perl Modules 5.10.0.0.1
Secure Socket Layer 11.2.0.2.0
Oracle JDBC/OCI Instant Client 11.2.0.2.0
Oracle Multimedia Client Option 11.2.0.2.0
LDAP Required Support Files 11.2.0.2.0
Character Set Migration Utility 11.2.0.2.0
Perl Interpreter 5.10.0.0.1
PL/SQL Embedded Gateway 11.2.0.2.0
OLAP SQL Scripts 11.2.0.2.0
Database SQL Scripts 11.2.0.2.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
SSL Required Support Files for InstantClient 11.2.0.2.0
SQL*Plus Files for Instant Client 11.2.0.2.0
Oracle Net Required Support Files 11.2.0.2.0
Oracle Database User Interface 2.2.13.0.0
RDBMS Required Support Files for Instant Client 11.2.0.2.0
RDBMS Required Support Files Runtime 11.2.0.2.0
XML Parser for Java 11.2.0.2.0
Oracle Security Developer Tools 11.2.0.2.0
Oracle Wallet Manager 11.2.0.2.0
Enterprise Manager plugin Common Files 11.2.0.2.0
Platform Required Support Files 11.2.0.2.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
RDBMS Required Support Files 11.2.0.2.0
Oracle Ice Browser 5.2.3.6.0
Oracle Help For Java 4.2.9.0.0
Enterprise Manager Common Files 10.2.0.4.3
Deinstallation Tool 11.2.0.2.0
Oracle Java Client 11.2.0.2.0
Cluster Verification Utility Files 11.2.0.2.0
Oracle Notification Service (eONS) 11.2.0.2.0
Oracle LDAP administration 11.2.0.2.0
Cluster Verification Utility Common Files 11.2.0.2.0
Oracle Clusterware RDBMS Files 11.2.0.2.0
Oracle Locale Builder 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
Buildtools Common Files 11.2.0.2.0
Oracle RAC Required Support Files-HAS 11.2.0.2.0
SQL*Plus Required Support Files 11.2.0.2.0
XDK Required Support Files 11.2.0.2.0
Agent Required Support Files 10.2.0.4.3
Parser Generator Required Support Files 11.2.0.2.0
Precompiler Required Support Files 11.2.0.2.0
Installation Common Files 11.2.0.2.0
Required Support Files 11.2.0.2.0
Oracle JDBC/THIN Interfaces 11.2.0.2.0
Oracle Multimedia Locator 11.2.0.2.0
Oracle Multimedia 11.2.0.2.0
HAS Common Files 11.2.0.2.0
Assistant Common Files 11.2.0.2.0
PL/SQL 11.2.0.2.0
HAS Files for DB 11.2.0.2.0
Oracle Recovery Manager 11.2.0.2.0
Oracle Database Utilities 11.2.0.2.0
Oracle Notification Service 11.2.0.2.0
SQL*Plus 11.2.0.2.0
Oracle Netca Client 11.2.0.2.0
Oracle Net 11.2.0.2.0
Oracle JVM 11.2.0.2.0
Oracle Internet Directory Client 11.2.0.2.0
Oracle Net Listener 11.2.0.2.0
Cluster Ready Services Files 11.2.0.2.0
Oracle Database 11g 11.2.0.2.0
-----------------------------------------------------------------------------


Instantiating scripts for add node (Monday, 5 September 2011 11:46:03 o'clock BST)
. 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Monday, 5 September 2011 11:46:15 o'clock BST)
............................................................................................... 96% Done.
Home copied to new nodes

Saving inventory on nodes (Monday, 5 September 2011 11:55:39 o'clock BST)
. 100% Done.
Save inventory complete
WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system.
To register the new inventory please run the script at '/apps/oracle/home/oraInventory/orainstRoot.sh' with root privileges on nodes 'NODEc'.
If you do not register the inventory, you may not be able to update or patch the products you installed.
The following configuration scripts need to be executed as the "root" user in each cluster node.
/apps/oracle/home/oraInventory/orainstRoot.sh #On nodes NODEc
/apps/product/11gr2/grid/root.sh #On nodes NODEc
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node

The Cluster Node Addition of /apps/product/11gr2/grid was successful.
Please check '/tmp/silentInstall.log' for more details.





Next the RDBMS home needs to be migrated over.


oracle@NODEa:/apps/oracle/product/11gr2/db/oui/bin ./addNode.sh "CLUSTER_NEW_NODES={NODEc}"

Performing pre-checks for node addition

Checking node reachability...
Node reachability check passed from node "NODEa"


Checking user equivalence...
User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Check: Node connectivity for interface "bge0"
Node connectivity passed for interface "bge0"

Node connectivity check passed


Checking CRS integrity...

CRS integrity check passed

Checking shared resources...

Checking CRS home location...
The location "/apps/product/11gr2/grid" is not shared but is present/creatable on all nodes
Shared resources check for node addition passed


Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Check: Node connectivity for interface "bge0"
Node connectivity passed for interface "bge0"

Check: Node connectivity for interface "bge1"
Node connectivity passed for interface "bge1"

Check: Node connectivity for interface "ce1"
Node connectivity passed for interface "ce1"

Node connectivity check passed

Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "NODEc:/var/tmp/"
Free disk space check passed for "NODEa:/var/tmp/"
Check for multiple users with UID value 2001 passed
User existence check passed for "oracle"
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "project.max-sem-ids"
Kernel parameter check passed for "process.max-sem-nsems"
Kernel parameter check passed for "project.max-shm-memory"
Kernel parameter check passed for "project.max-shm-ids"
Kernel parameter check passed for "tcp_smallest_anon_port"
Kernel parameter check passed for "tcp_largest_anon_port"
Kernel parameter check passed for "udp_smallest_anon_port"
Kernel parameter check passed for "udp_largest_anon_port"
Package existence check passed for "SUNWarc-...( sparc)"
Package existence check passed for "SUNWbtool-...( sparc)"
Package existence check passed for "SUNWhea-...( sparc)"
Package existence check passed for "SUNWlibm-...( sparc)"
Package existence check passed for "SUNWlibms-...( sparc)"
Package existence check passed for "SUNWsprot-...( sparc)"
Package existence check passed for "SUNWtoo-...( sparc)"
Package existence check passed for "SUNWi1of-...( sparc)"
Package existence check passed for "SUNWi1cs-...( sparc)"
Package existence check passed for "SUNWi15cs-...( sparc)"
Package existence check passed for "SUNWxwfnt-...( sparc)"
Package existence check passed for "SUNWlibC-...( sparc)"
Package existence check passed for "SUNWcsl-...( sparc)"
Operating system patch check passed for "Patch 119963-14"
Operating system patch check passed for "Patch 120753-06"
Operating system patch check passed for "Patch 139574-03"
Operating system patch check passed for "Patch 141414-02"
Check for multiple users with UID value 0 passed
Current group ID check passed

Checking OCR integrity...

OCR integrity check passed

Checking Oracle Cluster Voting Disk configuration...

Oracle Cluster Voting Disk configuration check passed
Time zone consistency check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
NTP Configuration file check passed

Checking daemon liveness...
Liveness check passed for "xntpd"
Check for NTP daemon or service alive passed on all nodes

NTP daemon slewing option check passed

NTP daemon's boot time configuration check for slewing option passed

NTP common Time Server Check started...
Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started...
Clock time offset check passed

Clock synchronization check using Network Time Protocol(NTP) passed


User "oracle" is not part of "root" group. Check passed
Checking consistency of file "/etc/resolv.conf" across nodes

File "/etc/resolv.conf" does not have both domain and search entries defined
domain entry in file "/etc/resolv.conf" is consistent across nodes
search entry in file "/etc/resolv.conf" is consistent across nodes
All nodes have one search entry defined in file "/etc/resolv.conf"
The DNS response time for an unreachable node is within acceptable limit on all nodes

File "/etc/resolv.conf" is consistent across nodes


Pre-check for node addition was successful.
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 13192 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
Oracle Universal Installer, Version 11.2.0.2.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.


Performing tests to see whether nodes NODEb,NODEc are available
............................................................... 100% Done.

....
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /apps/oracle/product/11gr2/db
New Nodes
Space Requirements
New Nodes
NODEc
/: Required 6.10GB : Available 24.03GB
Installed Products
Product Names
Oracle Database 11g 11.2.0.2.0
Sun JDK 1.5.0.24.08
Installer SDK Component 11.2.0.2.0
Oracle One-Off Patch Installer 11.2.0.0.2
Oracle Universal Installer 11.2.0.2.0
Oracle USM Deconfiguration 11.2.0.2.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Oracle DBCA Deconfiguration 11.2.0.2.0
Oracle RAC Deconfiguration 11.2.0.2.0
Oracle Database Deconfiguration 11.2.0.2.0
Oracle Configuration Manager Client 10.3.2.1.0
Oracle Configuration Manager 10.3.3.1.1
Oracle ODBC Driverfor Instant Client 11.2.0.2.0
LDAP Required Support Files 11.2.0.2.0
SSL Required Support Files for InstantClient 11.2.0.2.0
Bali Share 1.1.18.0.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
Oracle Real Application Testing 11.2.0.2.0
Oracle Database Vault J2EE Application 11.2.0.2.0
Oracle Label Security 11.2.0.2.0
Oracle Data Mining RDBMS Files 11.2.0.2.0
Oracle OLAP RDBMS Files 11.2.0.2.0
Oracle OLAP API 11.2.0.2.0
Platform Required Support Files 11.2.0.2.0
Oracle Database Vault option 11.2.0.2.0
Oracle RAC Required Support Files-HAS 11.2.0.2.0
SQL*Plus Required Support Files 11.2.0.2.0
Oracle Display Fonts 9.0.2.0.0
Oracle Ice Browser 5.2.3.6.0
Oracle JDBC Server Support Package 11.2.0.2.0
Oracle SQL Developer 11.2.0.2.0
Oracle Application Express 11.2.0.2.0
XDK Required Support Files 11.2.0.2.0
RDBMS Required Support Files for Instant Client 11.2.0.2.0
SQLJ Runtime 11.2.0.2.0
Database Workspace Manager 11.2.0.2.0
RDBMS Required Support Files Runtime 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
Exadata Storage Server 11.2.0.1.0
Provisioning Advisor Framework 10.2.0.4.3
Enterprise Manager Database Plugin -- Repository Support 11.2.0.2.0
Enterprise Manager Repository Core Files 10.2.0.4.3
Enterprise Manager Database Plugin -- Agent Support 11.2.0.2.0
Enterprise Manager Grid Control Core Files 10.2.0.4.3
Enterprise Manager Common Core Files 10.2.0.4.3
Enterprise Manager Agent Core Files 10.2.0.4.3
RDBMS Required Support Files 11.2.0.2.0
regexp 2.1.9.0.0
Agent Required Support Files 10.2.0.4.3
Oracle 11g Warehouse Builder Required Files 11.2.0.2.0
Oracle Notification Service (eONS) 11.2.0.2.0
Oracle Text Required Support Files 11.2.0.2.0
Parser Generator Required Support Files 11.2.0.2.0
Oracle Database 11g Multimedia Files 11.2.0.2.0
Oracle Multimedia Java Advanced Imaging 11.2.0.2.0
Oracle Multimedia Annotator 11.2.0.2.0
Oracle JDBC/OCI Instant Client 11.2.0.2.0
Oracle Multimedia Locator RDBMS Files 11.2.0.2.0
Precompiler Required Support Files 11.2.0.2.0
Oracle Core Required Support Files 11.2.0.2.0
Sample Schema Data 11.2.0.2.0
Oracle Starter Database 11.2.0.2.0
Oracle Message Gateway Common Files 11.2.0.2.0
Oracle XML Query 11.2.0.2.0
XML Parser for Oracle JVM 11.2.0.2.0
Oracle Help For Java 4.2.9.0.0
Installation Plugin Files 11.2.0.2.0
Enterprise Manager Common Files 10.2.0.4.3
Expat libraries 2.0.1.0.1
Deinstallation Tool 11.2.0.2.0
Oracle Quality of Service Management (Client) 11.2.0.2.0
Perl Modules 5.10.0.0.1
JAccelerator (COMPANION) 11.2.0.2.0
Oracle Containers for Java 11.2.0.2.0
Perl Interpreter 5.10.0.0.1
Oracle Net Required Support Files 11.2.0.2.0
Secure Socket Layer 11.2.0.2.0
Oracle Universal Connection Pool 11.2.0.2.0
Oracle JDBC/THIN Interfaces 11.2.0.2.0
Oracle Multimedia Client Option 11.2.0.2.0
Oracle Java Client 11.2.0.2.0
Character Set Migration Utility 11.2.0.2.0
Oracle Code Editor 1.2.1.0.0I
PL/SQL Embedded Gateway 11.2.0.2.0
OLAP SQL Scripts 11.2.0.2.0
Database SQL Scripts 11.2.0.2.0
Oracle Locale Builder 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
SQL*Plus Files for Instant Client 11.2.0.2.0
Required Support Files 11.2.0.2.0
Oracle Database User Interface 2.2.13.0.0
Oracle ODBC Driver 11.2.0.2.0
Oracle Notification Service 11.2.0.2.0
XML Parser for Java 11.2.0.2.0
Oracle Security Developer Tools 11.2.0.2.0
Oracle Wallet Manager 11.2.0.2.0
Cluster Verification Utility Common Files 11.2.0.2.0
Oracle Clusterware RDBMS Files 11.2.0.2.0
Oracle UIX 2.2.24.6.0
Enterprise Manager plugin Common Files 11.2.0.2.0
HAS Common Files 11.2.0.2.0
Precompiler Common Files 11.2.0.2.0
Installation Common Files 11.2.0.2.0
Oracle Help for the Web 2.0.14.0.0
Oracle LDAP administration 11.2.0.2.0
Buildtools Common Files 11.2.0.2.0
Assistant Common Files 11.2.0.2.0
Oracle Recovery Manager 11.2.0.2.0
PL/SQL 11.2.0.2.0
Generic Connectivity Common Files 11.2.0.2.0
Oracle Database Gateway for ODBC 11.2.0.2.0
Oracle Programmer 11.2.0.2.0
Oracle Database Utilities 11.2.0.2.0
Enterprise Manager Agent 10.2.0.4.3
SQL*Plus 11.2.0.2.0
Oracle Netca Client 11.2.0.2.0
Oracle Multimedia Locator 11.2.0.2.0
Oracle Call Interface (OCI) 11.2.0.2.0
Oracle Multimedia 11.2.0.2.0
Oracle Net 11.2.0.2.0
Oracle XML Development Kit 11.2.0.2.0
Database Configuration and Upgrade Assistants 11.2.0.2.0
Oracle JVM 11.2.0.2.0
Oracle Advanced Security 11.2.0.2.0
Oracle Internet Directory Client 11.2.0.2.0
Oracle Enterprise Manager Console DB 11.2.0.2.0
HAS Files for DB 11.2.0.2.0
Oracle Net Listener 11.2.0.2.0
Oracle Text 11.2.0.2.0
Oracle Net Services 11.2.0.2.0
Oracle Database 11g 11.2.0.2.0
Oracle OLAP 11.2.0.2.0
Oracle Spatial 11.2.0.2.0
Oracle Partitioning 11.2.0.2.0
Enterprise Edition Options 11.2.0.2.0
-----------------------------------------------------------------------------


Instantiating scripts for add node (Monday, 5 September 2011 12:09:04 o'clock BST)
. 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Monday, 5 September 2011 12:09:19 o'clock BST)
............................................................................................... 96% Done.
Home copied to new nodes

Saving inventory on nodes (Monday, 5 September 2011 12:20:17 o'clock BST)
. 100% Done.
Save inventory complete
WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system.
To register the new inventory please run the script at '/apps/oracle/home/oraInventory/orainstRoot.sh' with root privileges on nodes 'NODEc'.
If you do not register the inventory, you may not be able to update or patch the products you installed.
The following configuration scripts need to be executed as the "root" user in each cluster node.
/apps/oracle/home/oraInventory/orainstRoot.sh #On nodes NODEc
/apps/oracle/product/11gr2/db/root.sh #On nodes NODEc
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node

The Cluster Node Addition of /apps/oracle/product/11gr2/db was successful.
Please check '/tmp/silentInstall.log' for more details.
you have mail in /var/mail//oracle




Go to the DB HOME first and run root.sh and root user


root@NODEc # ./root.sh
Running Oracle 11g root script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /apps/oracle/product/11gr2/db

Enter the full pathname of the local bin directory: [/usr/local/bin]:
Creating /usr/local/bin directory...
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...


Creating /var/opt/oracle/oratab file...
Entries will be added to the /var/opt/oracle/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.



Then run the root.sh under the GRID HOME as root user. This actually starts up ASM and the clusterware and does the node addition(takes a little bit longer)


Entries will be added to the /var/opt/oracle/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /apps/product/11gr2/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
OLR initialization - successful
Adding daemon to inittab
ACFS-9200: Supported
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node NODEc, number 3, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded




Ensure ASM is running and all the diskgroup and disks are visible and mounted.

Finally add the instance to the cluster.


oracle@NODEc:/apps/oracle/product/11gr2/db/dbs srvctl add instance -d dbtest -i dbtest2 -n NODEc
oracle@NODEc:/apps/oracle/product/11gr2/db/dbs srvctl status database -d dbtest
Instance dbtest3 is running on node NODEa
Instance dbtest4 is running on node NODEb
Instance dbtest2 is not running on node NODEc
oracle@NODEc:/apps/oracle/product/11gr2/db/dbs srvctl start instance -d dbtest -i dbtest2

Friday, September 9, 2011

Oracle Clusterware restore

ASM and Clusterware are interleaved. ASM cannot startup without the clusterware running first and the clusterware can store all its voting disk and OCR in ASM
This is a bit of a complex intergration. I was in a scenario where I lost all my disks in ASM. When I say lost, we were trying a proof of concept. :)

When I tried to start ASM as a standalone with a pfile it complained that it couldnt communicate with the CSS
And I couldnt start the CRS without the voting disk and OCR, which I had now lost.
I was able to see all the LUNs on the server under /dev/rdsk

Below is how to recover.


Start CRS in exclusive NOCRS mode. This should be run only on one node of the cluster.


oracle@NODEA:/var/opt/oracle sudo crsctl start crs -excl -nocrs
Password:
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'NODEA'
CRS-2676: Start of 'ora.mdnsd' on 'NODEA' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'NODEA'
CRS-2676: Start of 'ora.gpnpd' on 'NODEA' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'NODEA'
CRS-2672: Attempting to start 'ora.gipcd' on 'NODEA'
CRS-2676: Start of 'ora.cssdmonitor' on 'NODEA' succeeded
CRS-2676: Start of 'ora.gipcd' on 'NODEA' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'NODEA'
CRS-2672: Attempting to start 'ora.diskmon' on 'NODEA'
CRS-2676: Start of 'ora.diskmon' on 'NODEA' succeeded
CRS-2676: Start of 'ora.cssd' on 'NODEA' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'NODEA'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'NODEA'
CRS-2672: Attempting to start 'ora.ctssd' on 'NODEA'
CRS-2676: Start of 'ora.ctssd' on 'NODEA' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'NODEA' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'NODEA' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'NODEA'
CRS-2676: Start of 'ora.asm' on 'NODEA' succeeded



Re-create all the diskgroups.


SQL> CREATE DISKGROUP ORCLCLU HIGH REDUNDANCY
2 FAILGROUP fg1 DISK '/dev/rdsk/c6t001738000CE8002Ad0s0'
3 FAILGROUP fg2 DISK '/dev/rdsk/c6t001738000CE8002Bd0s0'
4 FAILGROUP fg3 DISK '/dev/rdsk/c6t001738000CE8002Cd0s0'
5 FAILGROUP fg4 DISK '/dev/rdsk/c6t001738000CE8002Dd0s0'
6 QUORUM FAILGROUP fg5 DISK '/dev/rdsk/c6t001738000CE8002Ed0s0'
7 ATTRIBUTE 'compatible.asm' = '11.2.0.0.0';

Diskgroup created.

create diskgroup DATA_FAST external redundancy disk '/dev/rdsk/c6t001738000CE80026d0s0' attribute 'COMPATIBLE.ASM' = '11.2';

create diskgroup ARCHFLASH external redundancy disk '/dev/rdsk/c6t001738000CE80029d0s0' attribute 'COMPATIBLE.ASM' = '11.2';

create diskgroup DATA_RW external redundancy disk '/dev/rdsk/c6t001738000CE80028d0s0' attribute 'COMPATIBLE.ASM' = '11.2';
............




Run queries on V$asm_disk and V$asm_diskgroup to check all disks and diskgroups are available.

Time to restore OCR and voting disks.
OCR is first.
Find out where the OCR backups are located.


oracle@NODEA:/var/opt/oracle sudo ocrconfig -showbackup
Password:
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy

NODEA 2011/09/05 17:36:47 /apps/product/11gr2/grid/cdata/test-clu/backup00.ocr

NODEA 2011/09/05 13:36:46 /apps/product/11gr2/grid/cdata/test-clu/backup01.ocr

NODEA 2011/09/05 09:36:44 /apps/product/11gr2/grid/cdata/test-clu/backup02.ocr

NODEA 2011/09/04 01:36:33 /apps/product/11gr2/grid/cdata/test-clu/day.ocr

NODEA 2011/08/23 01:08:30 /apps/product/11gr2/grid/cdata/test-clu/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available

oracle@NODEA:/var/opt/oracle sudo ocrconfig -restore /apps/product/11gr2/grid/cdata/test-clu/backup00.ocr
oracle@NODEA:/var/opt/oracle



Now the voting disk. In 11g Voting disks are autobacked up with the OCR.


oracle@NODEA:/var/opt/oracle sudo crsctl replace votedisk +ORCLCLU
Successful addition of voting disk 142da6336d704f7fbf0fbebd482506d2.
Successful addition of voting disk 2477e787c9dd4f72bfa50bc3e88fc8d1.
Successful addition of voting disk 72c3a35d987a4f2ebf2e27c97ee69946.
Successful addition of voting disk 6c3a9fd3b0b84f06bfed66c9b9604f1b.
Successful addition of voting disk 56db0703046b4f22bf614ec7ab1bd716.
Successfully replaced voting disk group with +ORCLCLU.
CRS-4266: Voting file(s) successfully replaced



Create a SPFILE for ASM from its local PFILE. Place it on the shared storage.


sqlplus / as sysasm

create spfile='+DATA_RW' from pfile;



Clusterware restore done. Time to stop the exclusive running clusterware and make it work for all nodes

Stop the clusterware. -f forces the stop


oracle@NODEA:/apps/product/11gr2/grid/dbs sudo crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'NODEA'
CRS-2673: Attempting to stop 'ora.ctssd' on 'NODEA'
CRS-2673: Attempting to stop 'ora.asm' on 'NODEA'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'NODEA'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'NODEA'
CRS-2677: Stop of 'ora.asm' on 'NODEA' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'NODEA'
CRS-2677: Stop of 'ora.drivers.acfs' on 'NODEA' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'NODEA' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'NODEA' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'NODEA' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'NODEA'
CRS-2677: Stop of 'ora.cssd' on 'NODEA' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'NODEA'
CRS-2677: Stop of 'ora.gipcd' on 'NODEA' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'NODEA'
CRS-2677: Stop of 'ora.gpnpd' on 'NODEA' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'NODEA' has completed
CRS-4133: Oracle High Availability Services has been stopped.



Start the clusterware.


oracle@NODEA:/apps/product/11gr2/grid/dbs sudo crsctl start crs
CRS-4123: Oracle High Availability Services has been started.



finally check the status of the cluster on all nodes.


oracle@NODEB:/apps/product/11gr2/grid/log/NODEB/evmd sudo crsctl check cluster -all
Password:
**************************************************************
NODEA:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
NODEB:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************



My first try in restarting clusterware did not help as the server had some hanging cluster threads. To clear them, I had to restart the server and let the init scripts start clusterware cleanly.

Use crsctl stat res -t to check the status of all cluster resources.

Thursday, September 8, 2011

oraInventory

The Oracle inventory is a important meta-data respository, which most DBAs ignore. Lose it and you will run into problems patching and keeping the clusterware sane.
CLUVY heavily depends on the oraInventory; without which you will get the olsnodes missing error.

To check the oraInventory health simple commands as below are used


$ORACLE_HOME/OPatch/opatch lsinventory



If you do endup losing the inventory or corrupting it, it can be rebuild online without taking the DB or the clusterware down.
It is recommended to detach and attach the home back and the oraInventory should be preferably kept local on each node. I would also recommend DBAs tar and save the oraInventory once in a while.
In the below example GI is the Grid home with cluterware and ASM and ORACLE_HOME the RDBMS home

The below needs to be run on each node.


export ORACLE_HOME=/apps/oracle/product/11g/db
export GI_HOME=/apps/oracle/product/11g/grid

$GI_HOME/oui/bin/runInstaller -detachHome -silent -local ORACLE_HOME=$ORACLE_HOME

$GI_HOME/oui/bin/runInstaller -silent -local -ignoreSysPrereqs -attachHome ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME='11gR2DB1' LOCAL_NODE= CLUSTER_NODES=,

$GI_HOME/oui/bin/runInstaller -silent -local -ignoreSysPrereqs -attachHome ORACLE_HOME=$GI_HOME ORACLE_HOME_NAME='11gR2GI1' LOCAL_NODE=racnode1 CLUSTER_NODES=racnode1,racnode2 CRS=true



Note GI home is the only one which should have CRS=TRUE set.

The Oracle Home names can be found under


/oraInventory/ContentsXML/inventory.xml

where is the located? Open up /var/opt/oracle/oraInst.loc
#Wed Sep 07 15:59:40 BST 2011
inst_group=oinstall
inventory_loc=/apps/oracle/home/oraInventory

Tuesday, September 6, 2011

Disk controllers and ASM

What happens when you dont have similar hardware to setup cluster? They definately have to be same architecture.
We recently tried adding a node to a 2 node Oracle 11gr2 cluster. The new node was of a different hardware configuration although both were of the same architecture(Sun SPARC).

ASM is the storage for all the components(datafiles, voting disks, ocr files, spfiles). The storage paths on Node 1 and Node 2 were shown as


/dev/rdsk/c6t001738000CE8002Fd0s0



c6 - denotes the controller 6

When we introduced the storage to the new node it showed up as


/dev/rdsk/c0t001738000CE8002Fd0s0



c0 - denotes contoller 0

This means that the OS auto-assigns the storage paths under different controller. MPxIO was used to multipath and that set the controllers to access these storage paths.
How can we make ASM, recognize these different paths ?
two options
a) Create symbolic links for the devices so that they all look same on every node
b) Use ASM_DISKSTRING parameter to denote the search string in each node. ASM_DISKSTRING can be different on each node of the cluster. This parameter is dynamic and can be changed on the fly. Run 'select name, path from V$asm_disk;'. This forces ASM to search in the path again.

Remember to chown the disks to oracle:oinstall, otherwise ASM will not detect them.

How does Oracle find that the disk is part of ASM?
In Solaris, it is necessary to mark the disks. This is done by ASMLib in Linux. To mark the disk , it necessary to skip cylinders 0,1 and 2. cylinder 0 contains the VTOC and cylinder 2 stores the size of the disk. ASM data will start writing from cylinder 3.

ASM will not read the disks if this cylinder setup is not done properly.




Part Tag Flag Cylinders Size Blocks
0 unassigned wm 3 - 51197 100.00GB (51198/0/0) 209707008
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 - 51197 100.00GB (51198/0/0) 209707008
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0



The Solaris format tool can be used to achieve this.

If you have already used cylinders 0,1,2 to fill the ASM data; the disks need to be rebuilt. Backup all the data; drop the diskgroups and recreate them. Restore data after rebuilding the disks.

How do you destroy and rebuild all the disks including clusterware?

Thats coming up in the next blog :)

Monday, September 5, 2011

DB Health Check:: 11gR2 New Feature

A relatively unknown new feature with 11g is the Health Monitor Checks.

This feature has 27 checks out of which 6 of them are not internal. The internal checks are managed by the RDBMS software
The non internal checks can be used by DBAs to check and fix any problems.
the non-internal checks are

DB Structure Integrity Check
Data Block Integrity Check
Redo Integrity Check
Transaction Integrity Check
Undo Segment Integrity Check
Dictionary Integrity Check

Some of these checks needed Oracle Support guidence before like
Data Dictionary Consistency checks ID 456468.1. It is very useful to make it part of the software package now.


DBA should make this part of their checks every few months. Some of these checks are resource intensive and it is recommended to run during low load periods.

To run a check use


BEGIN
DBMS_HM.RUN_CHECK('Dictionary Integrity Check', to_char(sysdate,'DDMONYY')||'_dict_check');
END;
/


BEGIN
DBMS_HM.RUN_CHECK('DB Structure Integrity Check', to_char(sysdate,'DDMONYY')||'_dbstruct');
END;
/




These can also be added to the scheduler.

ADRCI can be used then to view the reports.


adrci> show hm_run

ADR Home = /apps/oracle/diag/rdbms/XXXX/XXXX4:
*************************************************************************

**********************************************************
HM RUN RECORD 1
**********************************************************
RUN_ID 1
RUN_NAME 01SEP11_check
CHECK_NAME Dictionary Integrity Check
NAME_ID 24
MODE 0
START_TIME 2011-09-01 09:25:46.844074 +01:00
RESUME_TIME
END_TIME 2011-09-01 09:35:15.977181 +01:00
MODIFIED_TIME 2011-09-01 09:37:45.385591 +01:00
TIMEOUT 0
FLAGS 0
STATUS 5
SRC_INCIDENT_ID 0
NUM_INCIDENTS 0
ERR_NUMBER 0
REPORT_FILE /apps/oracle/diag/rdbms/XXXX/XXXX4/hm/HMREPORT_01SEP11_check.hm
1 rows fetched




adrci> show report hm_run 01SEP11_check


This then should return the errors if any as



Dictionary Inconsistency
6386
FAILURE
OPEN
CRITICAL
0
2011-09-01 09:35:11.868297 +01:00
SQL dictionary health check: syn$.owner fk 95 on object SYN$ failed
Damaged rowid is AAAAA9AABAAAC5BAER - description: Synonymn ORDDCM_DOC_REGS is referenced