Disk controllers and ASM

What happens when you dont have similar hardware to setup cluster? They definately have to be same architecture.
We recently tried adding a node to a 2 node Oracle 11gr2 cluster. The new node was of a different hardware configuration although both were of the same architecture(Sun SPARC).

ASM is the storage for all the components(datafiles, voting disks, ocr files, spfiles). The storage paths on Node 1 and Node 2 were shown as

/dev/rdsk/c6t001738000CE8002Fd0s0

c6 - denotes the controller 6

When we introduced the storage to the new node it showed up as

/dev/rdsk/c0t001738000CE8002Fd0s0

c0 - denotes contoller 0

This means that the OS auto-assigns the storage paths under different controller. MPxIO was used to multipath and that set the controllers to access these storage paths.
How can we make ASM, recognize these different paths ?
two options
a) Create symbolic links for the devices so that they all look same on every node
b) Use ASM_DISKSTRING parameter to denote the search string in each node. ASM_DISKSTRING can be different on each node of the cluster. This parameter is dynamic and can be changed on the fly. Run 'select name, path from V$asm_disk;'. This forces ASM to search in the path again.

Remember to chown the disks to oracle:oinstall, otherwise ASM will not detect them.

How does Oracle find that the disk is part of ASM?
In Solaris, it is necessary to mark the disks. This is done by ASMLib in Linux. To mark the disk , it necessary to skip cylinders 0,1 and 2. cylinder 0 contains the VTOC and cylinder 2 stores the size of the disk. ASM data will start writing from cylinder 3.

ASM will not read the disks if this cylinder setup is not done properly.

Part Tag Flag Cylinders Size Blocks
0 unassigned wm 3 - 51197 100.00GB (51198/0/0) 209707008
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 - 51197 100.00GB (51198/0/0) 209707008
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0

The Solaris format tool can be used to achieve this.

If you have already used cylinders 0,1,2 to fill the ASM data; the disks need to be rebuilt. Backup all the data; drop the diskgroups and recreate them. Restore data after rebuilding the disks.

How do you destroy and rebuild all the disks including clusterware?

Thats coming up in the next blog :)