Tuesday, June 26, 2012

ORA-27303: additional information: startup egid = 501 (oinstall), current egid = 506 (asmadmin)

Following error message was seen trying to start the cluster database.
srvctl start database -d clusdb
PRCR-1079 : Failed to start resource ora.clusdb.db
CRS-5017: The resource action "ora.clusdb.db start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/clusdb/spfileclusdb.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/clusdb/spfileclusdb.ora
ORA-27140: attach to post/wait facility failed
ORA-27300: OS system dependent operation:invalid_egid failed with status: 1
ORA-27301: OS failure message: Operation not permitted
ORA-27302: failure occurred at: skgpwinit6
ORA-27303: additional information: startup egid = 501 (oinstall), current egid = 506 (asmadmin)
Looking at the oracle binary permission showed
cd $ORACLE_HOME/bin
ls -l oracle
-rwxr-s--x 1 oracle asmadmin 232437369 Apr 30 14:01 oracle

cd $GI_HOME/bin
ls -l oracle
-rwxr-x--x 1 oracle oinstall 204010819 Apr 30 13:57 oracle
The correct permissions of these files are
cd $ORACLE_HOME/bin
ls -l oracle
-rwsr-s--x. 1 oracle asmadmin 220009315 Jun 12 16:59 oracle

cd $GI_HOME/bin
ls -l oracle
-rwsr-s--x. 1 grid oinstall 204009973 Jun 12 14:14 oracle
If role separation is not used
cd $GI_HOME/bin
ls -l oracle
-rwsr-s--x 1 oracle oinstall 204010819 Apr 30 13:57 oracle
Permissions could be changed by setting the correct setuid bit
chmod 6751 $ORACLE_HOME/bin/oracle
ls -l $ORACLE_HOME/bin/oracle
-rwsr-s--x 1 oracle asmadmin 232437369 Apr 30 14:01 oracle

chmod 6751 $GI_HOME/bin/oracle
ls -l oracle
-rwsr-s--x 1 oracle oinstall 204010819 Apr 30 13:57 oracle
Or with the Oracle supplied setasmgidwrap executable. setasmgidwrap is in the GI_HOME but should be executed only against the ORACLE_HOME's oracle binary and not against the GI_HOME's oralce binary
$CRS_HOME/bin/setasmgidwrap o=$ORACLE_HOME/bin/oracle
If GI_HOME's oracle binary's permission has been changed then this must be set to correct permission manually as above.

Useful Metalink notes
11gR2 GRID User Cannot Connect To ASM Instance [ID 1382648.1]
Pre 11.2 Database Issues in 11gR2 Grid Infrastructure Environment [ID 948456.1]
How to change ownership and group memberships of Grid home in 11.2 Standalone [ID 1171674.1]
ORA-15183 Unable to Create Database on Server using 11.2 ASM and Grid Infrastructure [ID 1054033.1]
Database Creation on 11.2 Grid Infracture with Role Separation ( ORA-15025, KFSG-00312, ORA-15081 ) [ID 1084186.1]


Friday, June 22, 2012

Upgrading ASMLib and OS in 11gR1 RAC Environment

Upgrading operating system (OS) requires several consideration and tasks.
1. If ASMLib is used then ASMLib libraries must be upgraded as part of the OS upgrade.
2. After the upgrade Oracle/clusterware binaries should be relinked.
In this case the two nodes 11gR1 (11.1.0.7) RAC environment running on RHEL 5 (2.6.18-194.el5) will be upgraded to RHEL 5 (2.6.18-308.el5).
As per (743649.1) Oralce support rolling upgrade of OS when both OS are certified on the database that's running. This is supported only for the duration of the upgrade. But according to (1391807.1) using multiple versions of ASMLib across the cluster may not be compatible. To avoid these incompatible issues and to minimize the total downtime one node could be upgraded first and then shutting down the other node before the upgraded one with new version of ASMLib is started. This way only one node will be active in the cluster when two nodes are having different ASMLib versions. In this case all nodes are upgraded at the same time incurring total system outage.

1. Find out if ASMLibs are available for the kernel version to which the system is being upgraded to. If not move asmlib devices to block devices before the upgrade (also true if upgrading to RHEL 6 where to get asmlibs require ULN account) or see if (462618.1) could be of any help.

2. Make a note of the ownership and permission of the following files in $CRS_HOME/bin directory using the script below. Some files might be missing this is not a problem.
#!/bin/sh

if [ "$CRS_HOME" != "" ]; then
                echo "crs home exist"

        for i in clntsh.map clntst_1.lis clntst_2.lis clntst.lis libclntsh.so libclntsh.so.10.1 libclntst10.so libclntsh.so.11.1 libclntst11.so clntst
        do
                if [ -f $CRS_HOME/lib/$i ]; then
                        echo "file exists"
                        ls -l $CRS_HOME/lib/$i

                        if [ -f $CRS_HOME/lib32/$i ]; then
                                echo "file in lib32"
                                ls -l $CRS_HOME/lib32/$i
                        else
                                echo "file not exists"
                        fi
                else
                        if [ -f $CRS_HOME/lib32/$i ]; then
                                echo "file in lib32"
                                ls -l $CRS_HOME/lib32/$i
                        else
                                echo "file not exists"
                        fi
                fi
        done
else
                echo "Please set CRS_HOME and run again"
fi

./cluster_file_check.sh
crs home exist
file exists
-rwxr--r-- 1 root root 4765562 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/clntsh.map
file in lib32
-rw-r--r-- 1 root root 3985432 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/clntsh.map
file not exists
file not exists
file not exists
file exists
lrwxrwxrwx 1 root root 17 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so -> libclntsh.so.11.1
file in lib32
lrwxrwxrwx 1 root root 17 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so -> libclntsh.so.11.1
file exists
lrwxrwxrwx 1 oracle oinstall 51 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.10.1 -> /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so
file in lib32
lrwxrwxrwx 1 oracle oinstall 53 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so.10.1 -> /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so
file not exists
file exists
-rwxr-xr-x 1 root root 48316147 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.11.1
file in lib32
-rwxr-xr-x 1 root root 37081093 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so.11.1
file not exists
file not exists
3. Make a note of the asm disk's scsi_ids , this setup is a test setup using Virtualbox. The scsi_ids will be compared after the OS upgrade and must be the same as before the upgrade.
# blkid|grep sd.*oracleasm|while read a b;do echo -n $a$b" scsi_id=";(echo $a|tr -d [:digit:]|tr -d [:]|cut -d"/" -f3|xargs -i scsi_id -g -s /block/{})done;
/dev/sdc1:LABEL="DATA" TYPE="oracleasm" scsi_id=SATA     VBOX HARDDISK  VB35fd6dbb-14d87b49
/dev/sdd1:LABEL="FLASH" TYPE="oracleasm" scsi_id=SATA     VBOX HARDDISK  VB4b88ed44-eff71eb1
4. Shutdown the cluster stack across the cluster. If one node at a time is upgraded then do this only on one node. In this case since all nodes are upgraded at the same time entire cluster is brought down
crs_stop -all
5. Stop the crs and disbale auto start of the crs. Do this on all ndoes.
# crsctl stop crs
Stopping resources.
This could take several minutes.
Successfully stopped Oracle Clusterware resources
Stopping Cluster Synchronization Services.
Shutting down the Cluster Synchronization Services daemon.
Shutdown request successfully issued.

# crsctl disable crs
Oracle Clusterware is disabled for start-up after a reboot.
6. Comment out the crs spwaning processes entries in /etc/inittab. Do this on all nodes
#h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null
#h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null
#h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null
Run init on the systme and verify that no crs related processes are running.
/sbin/init q

ps -ef|grep css
ps -ef|grep crs
ps -ef|grep evm
ps -ef|grep init
7. Stop oracleasm running and disable and prevent it loading on reboot. Do this on all nodes.
# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# /etc/init.d/oracleasm disable
Writing Oracle ASM library driver configuration: done
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# /sbin/chkconfig --list | grep oracleasm
oracleasm       0:off   1:off   2:on    3:on    4:on    5:on    6:off
# /sbin/chkconfig oracleasm off
# /sbin/chkconfig --list | grep oracleasm
oracleasm       0:off   1:off   2:off   3:off   4:off   5:off   6:off
8. Make a backup of /etc/sysconfig/oracleasm-_dev_oracleasm on all nodes.
# cp /etc/sysconfig/oracleasm-_dev_oracleasm /etc/sysconfig/oracleasm-_dev_oracleasm.bak
At this point the system is ready for the OS upgrade.

9. Carry out the OS ugprade. All Oracle Home locations (ORACLE_HOME,ASM_HOME and CRS_HOME) are to be same before and after the upgrade





10. Install the new version of oracleasm and remove the old version. Old version could only be removed after installing the new version due to dependencies.
rpm -e oracleasm-2.6.18-194.el5-2.0.5-1.el5
error: Failed dependencies:
        oracleasm >= 1.0.4 is needed by (installed) oracleasmlib-2.0.4-1.el5.x86_64

# rpm -ivh oracleasm-2.6.18-308.el5-2.0.5-1.el5.x86_64.rpm
warning: oracleasm-2.6.18-308.el5-2.0.5-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...                ########################################### [100%]
   1:oracleasm-2.6.18-308.el########################################### [100%]

# rpm -e oracleasm-2.6.18-194.el5-2.0.5-1.el5

# rpm -qa | grep oracleasm
oracleasmlib-2.0.4-1.el5
oracleasm-2.6.18-308.el5-2.0.5-1.el5
oracleasm-support-2.1.3-1.el5
11. Configure oracleasm and make sure content on oracleasm-_dev_oracleasm.bak and oracleasm-_dev_oracleasm are the same.
/etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting  without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [oracle]:
Default group to own the driver interface [dba]:
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
12. Enable oracleasm and set it up to start on reboot.
/etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

chkconfig --list | grep oracleasm
oracleasm       0:off   1:off   2:off   3:off   4:off   5:off   6:off

# chkconfig oracleasm on

# chkconfig --list | grep oracleasm
oracleasm       0:off   1:off   2:on    3:on    4:on    5:on    6:off
13. Verify scsi_id are same as before the upgrade
# blkid|grep sd.*oracleasm|while read a b;do echo -n $a$b" scsi_id=";(echo $a|tr -d [:digit:]|tr -d [:]|cut -d"/" -f3|xargs -i scsi_id -g -s /block/{})done;
/dev/sdc1:LABEL="DATA" TYPE="oracleasm" scsi_id=SATA     VBOX HARDDISK  VB35fd6dbb-14d87b49
/dev/sdd1:LABEL="FLASH" TYPE="oracleasm" scsi_id=SATA     VBOX HARDDISK  VB4b88ed44-eff71eb1
With this the ASMLib upgrade is complete. Next is to relink the Oracle binaries.

14. As oracle user relink both ASM_HOME and ORACLE_HOME. Verify which relink is in the path variable using
which relink
Relink ORACLE_HOME
$ cd $ORACLE_HOME/bin
$ relink all
Reset the PATH and ORACLE_HOME before running relink for ASM_HOME
export PATH=$ASM_HOME/bin:$PATH
export ORACLE_HOME=$ASM_HOME
Verify the relink binary from ASM_HOME is in path
which relink
Run the relink of ASM_HOME
$ cd $ASM_HOME/bin
$ relink all
15. As per (743649.1 step 10) clusterware client shared libraries should be relinked as the clusterware software owner. But running this as software owner result in permission issue
export ORACLE_HOME=$CRS_HOME (important to set ORACLE_HOME variable to CRS_HOME location)
cd $ORACLE_HOME/bin

./genclntsh
/bin/rm: cannot remove `/opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.11.1': Permission denied
/bin/rm: cannot remove `/opt/crs/oracle/product/11.1.0/crs/lib/clntsh.map': Permission denied
./genclntsh: line 309: /opt/crs/oracle/product/11.1.0/crs/lib/clntsh.map: Permission denied
genclntsh: Failed to link libclntsh.so.11.1
Permission issue persist even after setting the permission to oracle:oinstall. Running ./genclntsh as root results in files get created with new timestamp permission same as before (metalink note could be wrong!??). When genclntsh runs without any issue no message is shown and return to shell prompt. After run check the file permissions are same as before (step 2 above).
./cluster_file_check.sh
crs home exist
file exists
-rw-r--r-- 1 root root 4764652 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib/clntsh.map
file in lib32
-rw-r--r-- 1 root root 3984490 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib32/clntsh.map
file not exists
file not exists
file not exists
file exists
lrwxrwxrwx 1 root root 17 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so -> libclntsh.so.11.1
file in lib32
lrwxrwxrwx 1 root root 17 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so -> libclntsh.so.11.1
file exists
lrwxrwxrwx 1 oracle oinstall 51 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.10.1 -> /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so
file in lib32
lrwxrwxrwx 1 oracle oinstall 53 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so.10.1 -> /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so
file not exists
file exists
-rwxr-xr-x 1 root root 48316147 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.11.1
file in lib32
-rwxr-xr-x 1 root root 37081093 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so.11.1
file not exists
file not exists
16. Enable crs and remove the comments on /etc/inittab and start crs
# crsctl enable crs
Oracle Clusterware is enabled for start-up after a reboot.

h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null
h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null
h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null
17. Start crs with crsctl or with OS init
crsctl start crs
Attempting to start Oracle Clusterware stack
The CRS stack will be started shortly
or
/sbin/init q
This conclude the ASMLib and OS upgrade.

Useful Metalink Notes
How to Relink Oracle Database Software on UNIX [ID 131321.1]
Is It Necessary To Relink Oracle Following OS Upgrade? [ID 444595.1]
Will an Operating System Upgrade Affect Oracle Clusterware? [ID 743649.1]
How To Upgrade ASMLib Kernel Driver as Part of Kernel Upgrade? [ID 1391807.1]
Do You Need to Relink Oracle Clusterware When Upgrading the Operating System? [ID 743649.1]
Is It Required To Relink The Oracle Binaries When Booting With The Old Kernel? [ID 461138.1]
Cannot Find Exact Kernel Version Match For ASMLib (Workaround using oracleasm_debug_link tool) [ID 462618.1]
RAC: Frequently Asked Questions [ID 220970.1] (Is a relink required for the clusterware home after an OS upgrade?)

Useful White Paper
Best Practices for Optimizing Availability During Planned Maintenance Using Oracle Clusterware and Oracle Real Application Clusters. Oracle Maximum Availability Architecture White Paper September 2007

Related Posts
Upgrading RHEL 6 OS in a 11gR2 RAC Environment
Upgrading OS in 11gR2 RAC Environment (RHEL 5)

Moving ASMLib disk to block devices (Non-ASMLib) in 11gR1 RAC

This post list steps specific to moving ASMLib disk to block devices in two nodes 11gR1 RAC environment. There's an earlier post which shows steps involved with moving ASMLib to block devices in a 11gR2 standalone setup. Most of the steps are similar as the earlier post but in the RAC environment the moving from ASMLib to block device could be done in a rolling fashion reducing down time.

1. Create a pfile from the asm spfile. This is more for protection (to start ASM if there's a mistake in asm_string and can't get the ASM up with spfile) and is not part of the moving process.
SQL> create pfile='/home/oracle/asmpfile.ora' from spfile;
File created.

more asmpfile.ora
+ASM1.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from environment
+ASM2.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from environment
+ASM1.asm_diskgroups='DATA','FLASH'
+ASM2.asm_diskgroups='DATA','FLASH'
*.cluster_database=true
*.diagnostic_dest='/opt/app/oracle'
+ASM2.instance_number=2
+ASM1.instance_number=1
*.instance_type='asm'
*.large_pool_size=12M
2. Current ASM disks
SQL> select name,label,path from v$asm_disk;

NAME       LABEL      PATH
---------- ---------- ----------
DATA       DATA       ORCL:DATA
FLASH      FLASH      ORCL:FLASH
3. It is possible to dismount diskgroup in one node and test the procedure before applying the changes permanently.
SQL> alter diskgroup flash dismount
From ASM alert log
SQL> alter diskgroup flash dismount
NOTE: cache dismounting group 2/0x73784638 (FLASH)
Fri Jun 22 13:29:49 2012
kjbdomdet send to node 1
detach from dom 2, sending detach message to node 1
Fri Jun 22 13:29:49 2012
NOTE: detached from domain 2
NOTE: cache dismounted group 2/0x73784638 (FLASH)
kfdp_dismount(): 9
Fri Jun 22 13:29:49 2012
kfdp_dismountBg(): 9
NOTE: De-assigning number (2,0) from disk (ORCL:FLASH)
SUCCESS: diskgroup FLASH was dismounted
SUCCESS: alter diskgroup flash dismount
Status of ASM disks accross cluster which shows FLASH disgroup in node 1 dismounted
SQL> select inst_id,name,state from gv$asm_diskgroup; 

   INST_ID NAME      STATE
    ------- --------- -----------
         2 DATA      MOUNTED
         2 FLASH     MOUNTED
         1 DATA      MOUNTED
         1 FLASH     DISMOUNTED
4. Query the ASMLib disk and find out the block device used for creating ASMLib disk
# /etc/init.d/oracleasm querydisk -d FLASH
Disk "FLASH" is a valid ASM disk on device [8, 49]

ls -l /dev/sd*
...
brw-r----- 1 root   disk     8, 32 Jun 20 17:16 /dev/sdc
brw-r----- 1 root   disk     8, 33 Jun 20 17:17 /dev/sdc1
brw-r----- 1 root   disk     8, 48 Jun 20 17:16 /dev/sdd
brw-r----- 1 root   disk     8, 49 Jun 20 17:17 /dev/sdd1
Change the ownership of the block device
chown oracle:dba /dev/sdd1

ls -l /dev/sd*
...
brw-r----- 1 root   disk     8, 32 Jun 20 17:16 /dev/sdc
brw-r----- 1 root   disk     8, 33 Jun 20 17:17 /dev/sdc1
brw-r----- 1 root   disk     8, 48 Jun 20 17:16 /dev/sdd
brw-r----- 1 oracle dba      8, 49 Jun 20 17:17 /dev/sdd1
5. Change the asm_diskstring to include the block device and mount the flash diskgroup
SQL> alter system set asm_diskstring ='ORCL:DATA','/dev/sdd1' scope=memory;
System altered.
Mount the FLASH diskgroup and verify block device is used for ASM diskgroup
SQL> alter diskgroup flash mount

NOTE: cache registered group FLASH number=2 incarn=0x73784644
NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x73784644
NOTE: Assigning number (2,0) to disk (/dev/sdd1)
kfdp_query(FLASH): 11
Fri Jun 22 13:35:46 2012
kfdp_queryBg(): 11
NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdd1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 2/0x73784644 (FLASH)
Fri Jun 22 13:35:46 2012
kjbdomatt send to node 1
From the data views
SQL> select inst_id,name,label,path from gv$asm_disk;

   INST_ID NAME       LABEL      PATH
---------- ---------- ---------- ----------
         1 DATA       DATA       ORCL:DATA
         1 FLASH                 /dev/sdd1
         2 DATA       DATA       ORCL:DATA
         2 FLASH      FLASH      ORCL:FLASH
6. Do the same with the other diskgroup. Query the disk to find out the block device and change the ownership of the block device so oracle process could write to it directly
/etc/init.d/oracleasm querydisk -d data
Disk "DATA" is a valid ASM disk on device [8, 33]

# chown oracle:dba /dev/sdc1

ls /dev/sd*
...
brw-r----- 1 root   disk     8, 32 Jun 20 17:16 /dev/sdc
brw-r----- 1 oracle dba      8, 33 Jun 20 17:17 /dev/sdc1
brw-r----- 1 root   disk     8, 48 Jun 20 17:16 /dev/sdd
brw-r----- 1 oracle dba      8, 49 Jun 22 13:39 /dev/sdd1
7. Unmount all the diskgroups in the node and set the asm_diskstring. Trying to set asm_diskstring while one is mounted would result in an error. See earlier post's step 9
SQL> alter diskgroup flash dismount;
Diskgroup altered.

SQL> alter diskgroup data dismount;
Diskgroup altered.

SQL> alter system set asm_diskstring ='/dev/sdc1','/dev/sdd1' scope=memory;
System altered.

SQL> alter diskgroup data mount;
Diskgroup altered.

SQL> alter diskgroup flash mount;
Diskgroup altered.
From ASM alert log
NOTE: cache registered group DATA number=1 incarn=0x7378464e
NOTE: cache began mount (not first) of group DATA number=1 incarn=0x7378464e
NOTE: Assigning number (1,0) to disk (/dev/sdc1)
kfdp_query(DATA): 16
Fri Jun 22 13:41:12 2012
kfdp_queryBg(): 16
NOTE: cache opening disk 0 of grp 1: DATA path:/dev/sdc1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 1/0x7378464E (DATA)
Fri Jun 22 13:41:12 2012
kjbdomatt send to node 1
...
...
SQL> alter diskgroup flash mount
NOTE: cache registered group FLASH number=2 incarn=0x73784652
NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x73784652
NOTE: Assigning number (2,0) to disk (/dev/sdd1)
kfdp_query(FLASH): 19
kfdp_queryBg(): 19
NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdd1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 2/0x73784652 (FLASH)
kjbdomatt send to node 1
Confirm first node's ASM diskgroups are now using block devices
SQL> select inst_id,name,label,path from gv$asm_disk;

   INST_ID NAME       LABEL      PATH
---------- ---------- ---------- ----------
         2 DATA       DATA       ORCL:DATA
         2 FLASH      FLASH      ORCL:FLASH
         1 DATA                  /dev/sdc1
         1 FLASH                 /dev/sdd1
8. Create udev rules so the block device ownerships are persistent across reboots.
# ASM DATA
KERNEL=="sdc[1]", OWNER="oracle", GROUP="dba", MODE="660"

# ASM FLASH
KERNEL=="sdd[1]", OWNER="oracle", GROUP="dba", MODE="660"
9. Set the new asm_diskstring values to ASM spfile
SQL> alter system set asm_diskstring ='/dev/sdc1','/dev/sdd1' scope=both;
10. Unload the oracleasm module. To unload the oracleasm module the ASM instance must be shutdown.
lsmod  | grep oracleasm
oracleasm              84136  1

# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [FAILED]

srvctl stop asm -n rac1

# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# /etc/init.d/oracleasm disable
Writing Oracle ASM library driver configuration: done
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# chkconfig oracleasm off
11. Start the ASM instance on the node
srvctl start asm -n rac1
Verify the block devices are used by monitoring the ASM alert log
NOTE: cache opening disk 0 of grp 1: DATA path:/dev/sdc1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 1/0x737E1A2E (DATA)
kjbdomatt send to node 1
NOTE: attached to recovery domain 1
NOTE: LGWR attempting to mount thread 1 for diskgroup 1
NOTE: LGWR mounted thread 1 for disk group 1
NOTE: opening chunk 1 at fcn 0.69396 ABA
NOTE: seq=102 blk=4866
NOTE: cache mounting group 1/0x737E1A2E (DATA) succeeded
NOTE: cache ending mount (success) of group DATA number=1 incarn=0x737e1a2e
kfdp_query(FLASH): 4
kfdp_queryBg(): 4
NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdd1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 2/0x737E1A2F (FLASH)
Start the database instance on the node

12. Next step is to mount the asm diskgroup on the remaining node with the block devices. Stop the database instance running on the node and change the ownership of the block devices. These were identified earlier
# chown oracle:dba /dev/sdc1
# chown oracle:dba /dev/sdd1
13. Since asm_diskstring was set as both no need to make any changes simply dismount and mount one diskgroup to test the configuration
SQL> alter diskgroup flash dismount;

Diskgroup altered.

SQL> alter diskgroup flash mount;

Diskgroup altered.
14. Create a udev rules file in this node as well
# ASM DATA
KERNEL=="sdc[1]", OWNER="oracle", GROUP="dba", MODE="660"

# ASM FLASH
KERNEL=="sdd[1]", OWNER="oracle", GROUP="dba", MODE="660"
15. Stop the asm instance and unload the oracleasm module
srvctl stop asm -n rac2

# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# /etc/init.d/oracleasm disable
Writing Oracle ASM library driver configuration: done
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

#chkconfig oracleasm off
15. Start asm instance and verify block devices being used through ASM alert log
srvctl start asm -n rac2

NOTE: cache opening disk 0 of grp 1: DATA path:/dev/sdc1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 1/0x73701F4C (DATA)
kjbdomatt send to node 0
NOTE: attached to recovery domain 1
NOTE: LGWR attempting to mount thread 2 for diskgroup 1
NOTE: LGWR mounted thread 2 for disk group 1
NOTE: opening chunk 2 at fcn 0.69424 ABA
NOTE: seq=86 blk=4843
NOTE: cache mounting group 1/0x73701F4C (DATA) succeeded
NOTE: cache ending mount (success) of group DATA number=1 incarn=0x73701f4c
kfdp_query(FLASH): 4
kfdp_queryBg(): 4
NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdd1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 2/0x73701F4D (FLASH)
Querying the data view
SQL> select inst_id,name,label,path from gv$asm_disk order by 1;

   INST_ID NAME       LABEL      PATH
---------- ---------- ---------- ----------
         1 DATA                  /dev/sdc1
         1 FLASH                 /dev/sdd1
         2 DATA                  /dev/sdc1
         2 FLASH                 /dev/sdd1
16. As the last step uninstall the oracleasm libraries
# rpm -e oracleasmlib-2.0.4-1.el5
# rpm -e oracleasm-2.6.18-308.el5-2.0.5-1.el5
# rpm -e oracleasm-support-2.1.3-1.el5
Related Posts
Migrating block devices using ASM instance to ASMLib
Moving ASMLib disk to block devices (Non-ASMLib) in 11gR2 Standalone

Useful Metalink Notes
How To Migrate ASMLIB Devices To Block Devices (Non-ASMLIB)? [ID 567508.1]

Tuesday, June 19, 2012

Moving ASMLib disk to block devices (Non-ASMLib) in 11gR2 Standalone

This post list steps to move ASMLib devices to block devices for a standalone 11gR2 installation with role separation created earlier.
1. As the grid user login to ASM instance and create a pfile from the ASM spfile
sqlplus  / as sysasm
SQL> create pfile='/home/grid/asmpfile.ora' from spfile;

File created.

$ more asmpfile.ora
+ASM.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from in memory value
+ASM.asm_diskgroups='FLASH','DATA'#Manual Mount
*.asm_power_limit=1
*.diagnostic_dest='/opt/app/oracle'
*.instance_type='asm'
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'
2. Current ASMLib disks are
SQL> select name,label,path from v$asm_disk;

NAME    LABEL   PATH
------- ------- -----------
DATA    DATA    ORCL:DATA
FLASH   FLASH   ORCL:FLASH
3. Before ASMLib disks could be moved the database must be stopped. This is required as ASM diskgroups will be dismounted later on.
srvctl stop database -d stdalone
4. Dismount and mount the flash diskgroup and monitor the ASM alert log which will show the ASM disk(s) used with the diskgroup. This information is useful to verify if the new block device disk assignment is working.
SQL> alter diskgroup flash dismount;

Diskgroup altered.
From the alert log
NOTE: cache deleting context for group FLASH 2/0x9328489f
Thu Jun 14 11:57:49 2012
NOTE: diskgroup resource ora.FLASH.dg is offline
SUCCESS: alter diskgroup flash dismount
NOTE: diskgroup resource ora.FLASH.dg is updated
When mounting
SQL> alter diskgroup flash mount
From alert log
NOTE: cache registered group FLASH number=2 incarn=0xed0848a6
NOTE: cache began mount (first) of group FLASH number=2 incarn=0xed0848a6
NOTE: Assigning number (2,0) to disk (ORCL:FLASH)
Thu Jun 14 11:58:18 2012
NOTE: GMON heartbeating for grp 2
5. Identify the block device used to create the ASMLib disk by querying the ASMLib disk
# /etc/init.d/oracleasm querydisk -d FLASH
Disk "FLASH" is a valid ASM disk on device [8, 33]
Block device could be identified with
ls -l /dev/sd*
brw-r----- 1 root disk 8,  0 Jun 14 11:47 /dev/sda
brw-r----- 1 root disk 8,  1 Jun 14 11:48 /dev/sda1
brw-r----- 1 root disk 8,  2 Jun 14 11:47 /dev/sda2
brw-r----- 1 root disk 8,  3 Jun 14 11:48 /dev/sda3
brw-r----- 1 root disk 8, 16 Jun 14 11:47 /dev/sdb
brw-r----- 1 root disk 8, 17 Jun 14 11:48 /dev/sdb1
brw-r----- 1 root disk 8, 32 Jun 14 11:47 /dev/sdc
brw-r----- 1 root disk 8, 33 Jun 14 11:48 /dev/sdc1
The block device used to create the FLASH ASMLib disk is /dev/sdc1.

6. Set the permission on the block device to grid:asmadmin
# chown grid:asmadmin /dev/sdc1
# ls -l /dev/sdc*
brw-r----- 1 root disk     8, 32 Jun 14 11:47 /dev/sdc
brw-r----- 1 grid asmadmin 8, 33 Jun 14 11:48 /dev/sdc1
7. Dismount the FLASH diskgroup again and set the asm_diskstring to include the block device. Still the DATA diskgroup is using ASMLib therefore ORCL* is also set. This mix setting is alright for testing purposes but it's better not to run production environments in the mix mode (both ASMLib and Block devices).
SQL> alter system set asm_diskstring ='ORCL:DATA','/dev/sdc1' SCOPE=MEMORY;
Mount the FLASH diskgroup and monitor the ASM alert log and verify block device is used
SQL> ALTER DISKGROUP flash mount
On the alert log
NOTE: cache registered group FLASH number=2 incarn=0x3d4848aa
NOTE: cache began mount (first) of group FLASH number=2 incarn=0x3d4848aa
NOTE: Assigning number (2,0) to disk (/dev/sdc1)
Thu Jun 14 12:01:15 2012
NOTE: GMON heartbeating for grp 2
GMON querying group 2 at 18 for pid 19, osid 3268
NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdc1
Further verification from the v$asm_disk view
SQL> select name,label,path from v$asm_disk;

NAME    LABEL   PATH
------- ------- -----------
DATA    DATA    ORCL:DATA
FLASH   FLASH   /dev/sdc1
8. Do the same with other diskgroup
# /etc/init.d/oracleasm querydisk -d DATA
Disk "DATA" is a valid ASM disk on device [8, 17]
Set the block device permissions
# chown grid:asmadmin /dev/sdb1
# ls -l /dev/sdb*
brw-r----- 1 root disk     8, 16 Jun 14 11:47 /dev/sdb
brw-r----- 1 grid asmadmin 8, 17 Jun 14 11:48 /dev/sdb1
9. Dismount all diskgroups. Trying to change the asm_diskstring while diskgroups are mounted will throw the following error
SQL> alter system set asm_diskstring ='/dev/sdb1','/dev/sdc1' scope=memory;
alter system set asm_diskstring ='/dev/sdb1','/dev/sdc1' scope=memory
*
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-15014: path 'ORCL:DATA' is not in the discovery set
Once all diskgroups are dismounted no issue.
SQL> alter diskgroup data dismount;
Diskgroup altered.

SQL> alter diskgroup flash dismount;
Diskgroup altered.

SQL> alter system set asm_diskstring ='/dev/sdb1','/dev/sdc1' scope=both;
Mount each diskgroup and verify the block device is used. Following from the alert log
Thu Jun 14 12:04:51 2012
SQL> alter diskgroup data mount
NOTE: cache registered group DATA number=1 incarn=0xa8c848b4
NOTE: cache began mount (first) of group DATA number=1 incarn=0xa8c848b4
NOTE: Assigning number (1,0) to disk (/dev/sdb1)
Further verification from v$asm_disk
SQL> select name,label,path from v$asm_disk;

NAME    LABEL   PATH
------- ------- -----------
DATA    DATA    /dev/sdb1
FLASH   FLASH   /dev/sdc1
10. Now that it's been verified that block devices are used for ASM create udev rules.
# ASM DATA
KERNEL=="sdb[1]", OWNER="grid", GROUP="asmadmin", MODE="660"

# ASM FLASH
KERNEL=="sdc[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
11. Stop the high availability stack
crsctl stop has
and unload the oracleasm modules
# lsmod  | grep oracleasm
oracleasm              84136  1

# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# lsmod  | grep oracleasm
disable loading of oracleasm when server is rebooted
# chkconfig oracleasm off
Restart the high availability service
crsctl start has
CRS-4123: Oracle High Availability Services has been started.
and check all ASM starting with block devices without an issue. When the restart is complete check the ASM alert log
SQL> ALTER DISKGROUP ALL MOUNT /* asm agent call crs *//* {0:0:2} */
NOTE: Diskgroups listed in ASM_DISKGROUPS are
         DATA
         FLASH
NOTE: cache registered group DATA number=1 incarn=0xa71a3a36
NOTE: cache began mount (first) of group DATA number=1 incarn=0xa71a3a36
NOTE: cache registered group FLASH number=2 incarn=0xa71a3a37
NOTE: cache began mount (first) of group FLASH number=2 incarn=0xa71a3a37
Errors in file /opt/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_rbal_4010.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
NOTE: Assigning number (1,0) to disk (/dev/sdb1)
NOTE: Assigning number (2,0) to disk (/dev/sdc1)
and/or using kfod tool to confirm block devices are being used correctly
kfod asm_diskstring='/dev/sd*' disk=all
--------------------------------------------------------------------------------
 Disk          Size Path                                     User     Group
================================================================================
   1:      10236 Mb /dev/sdb1                                grid     asmadmin
   2:      10236 Mb /dev/sdc1                                grid     asmadmin
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME
================================================================================
      +ASM /opt/app/oracle/product/11.2.0/grid
12. If possible restart the server and check all changes (udev rules, disk permissions) and are working without issue and as the final step uninstall the ASMLib
# rpm -e oracleasmlib-2.0.4-1.el5
# rpm -e oracleasm-2.6.18-194.el5-2.0.5-1.el5
# rpm -e oracleasm-support-2.1.3-1.el5
Related Post
Migrating block devices using ASM instance to ASMLib
Moving ASMLib disk to block devices (Non-ASMLib) in 11gR1 RAC

Useful Metalink Notes
How To Migrate ASMLIB Devices To Block Devices (Non-ASMLIB)? [ID 567508.1]

Friday, June 15, 2012

Implementing Class of Secure Transport (COST) to Restrict Instance Registration in Oracle 11gR2 SE RAC (Solution mentioned in Oracle Security Alert for CVE-2012-1675)

This post is related to implementing the solution for security vulnerability mentioned in Oracle Security Alert for CVE-2012-1675. There is another post but that is related to version 11gR1 and this one is specific to 11gR2.
The metalink note related to applying the solution in a RAC environment is Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle RAC [ID 1340831.1]
However this solution fails when tried on a standard edition RAC environment as standard edition Oracle binaries are not linked with TCPS protocol. The standard edition RAC environment created for RHEL 6 is used in this case.
All the steps in section 1 in (1340831.1) up to step 1.4 work same on SE as on EE edition and problem will be encountered trying to implement step 1.5.
As oracle user (since this is a role separated environment) try to set TCPS protocol for the remote listener using scan ips.
show parameter remote_listener

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
remote_listener                      string      rhel6m-scan:1521
This environment only used one scan ip (not recommended by Oracle)
srvctl config scan
SCAN name: rhel6m-scan, Network: 1/192.168.0.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /rhel6m-scan/192.168.0.91
Trying to set TCPS will give the following error
SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*';
alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*'
*
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-00119: invalid specification for system parameter REMOTE_LISTENER
ORA-00130: invalid listener address
'(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523))'
Error messages here doesn't help much as it doesn't specifically say which parameter is invalid unlike in 11gR1 which mentioned invalid protocol. It could be tested indeed it's the protocol that is causing the problem by testing the same command with TCP instead of TCPS, in this case it would work without an error
SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*';

System altered.
Solution is to relink the binaries with tcps same as before. Stop all processes running out of the ORACLE_HOME
srvctl stop home -o $ORACLE_HOME -s status -n `hostname -s`
Change into $ORACLE_HOME/lib and make a backup of the existing libntcps11.a file and copy the libntcps11_ee.a.dbl
cp libntcps11.a libntcps11.a.orig
cp libntcps11_ee.a.dbl libntcps11.a
Execute relink all and monitor the log file
$ relink all
writing relink log to: /opt/app/oracle/product/11.2.0/dbhome_1/install/relink.log
Relinking ends with following text
-L/opt/app/oracle/product/11.2.0/dbhome_1/lib
test ! -f /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle ||\
           mv -f /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracleO
mv /opt/app/oracle/product/11.2.0/dbhome_1/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle
It is important to relink all the oracle homes in the cluster if not the error will still persist
SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*';
alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*'
*
ERROR at line 1:
ORA-32008: error while processing parameter update at instance std11g21
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-00119: invalid specification for system parameter REMOTE_LISTENER
ORA-00130: invalid listener address
'(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523))'
But to confirm relinking has worked the above command could be executed at instance level
SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='std11g22';

System altered.
Once all oracle homes are relinked execute the original command.
SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*';

System altered.
Rest of the steps in (1340831.1) could be executed without any additional work.
If the libntcps11_ee.a.dbl file is missing it could obtained by extracting the
database/stage/Components/oracle.network.rsf/11.2.0.3.0/1/DataFiles/filegroup5.jar

Related Post
Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle SE RAC Fails (Solution mentioned in Oracle Security Alert for CVE-2012-1675)




Update on 11th November 2015
For database version 11.2.0.4 Oracle has introduced "Valid Node Checking for Registration (VNCR)" as an alternative for implementing COST, if the sole reason for implementing cost is preventing remote registration. MOS note 1340831.1 has been updated with this information. VNCR related information could be found on following docs.
How to Enable VNCR on RAC Database to Register only Local Instances [ID 1914282.1]
Valid Node Checking For Registration (VNCR) [ID 1600630.1]

Thursday, June 14, 2012

Installing 11gR2 (11.2.0.3) RAC with Role Separation on RHEL 6

Installing GI with role separation in RHEL 6 is identical to installing on OEL 6. This blog post is not meant to be an extensive how to install GI on RHEL 6, it will highlight the key aspects relating to RHEL 6 and role separation. There's also a post on installing standalone server with role separation on RHEL 6.
1. According to GI installation guide for RHEL 6 the kernel version must be 2.6.32-71.el6.x86_64 or later or if the RHEL 6 kernel is with unbreakable enterprise linux kernel then it must be 2.6.32-100.28.5.el6.x86_64 or later. In this case the kernel version is
uname -r
2.6.32-220.el6.x86_64
2. Create OS groups and users needed for role separation. Oracle user will own database software and grid user will own the clusterware.
create groups and users
groupadd -g 501 oinstall
groupadd -g 502 dba
groupadd -g 503 oper
groupadd -g 505 asmadmin
groupadd -g 506 asmdba
groupadd -g 507 asmoper

useradd -u 501 -g oinstall -G dba,oper,asmdba oracle
useradd -u 502 -g oinstall -G asmadmin,asmdba,asmoper,dba grid
grid user should be part of dba group as per metalink note 1084186.1

3. Create user equivalence for both grid user and oracle user.
$ /usr/bin/ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_rsa):
Created directory '/home/grid/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_rsa.
Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
...
...
4. Create directory structure for clusterware and Oracle base and set with relevant ownership and permission.
# mkdir -p /opt/app/11.2.0/grid
# chown grid:oinstall /opt/app/11.2.0/grid
# chmod 775 /opt/app/11.2.0/grid

oracle base

# mkdir -p /opt/app/oracle
# chown oracle:oinstall /opt/app/oracle
# chmod 775 /opt/app/oracle

oraInventory

# mkdir -p /opt/app/oraInventory
# chown grid:oinstall /opt/app/oraInventory
# chmod 775 /opt/app/oraInventory
5. Verify pre-req package list is installed. Follow GI installation guide and metalink note Installing Oracle 11gR2 RDBMS on RHEL6 or OL6 64-bit (x86-64) [ID 1441282.1]

6.Set resource limit for both oracle and grid user in /etc/security/limits.conf file
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
oracle hard stack 32768

grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
grid soft stack 10240
grid hard stack 32768
7. Disable x11 forward for grid and oracle user by adding the following to each user's .ssh/config file
Host *
 ForwardX11 no
8. Disable tty on .bashrc for both grid and oracle user
if [ -t 0 ]; then
 stty intr ^C
fi
and also set umask value in .bashrc for grid and oracle user
umask 022
9. As per metalink note (1089399.1) for RHEL6/OEL6 to get oracleasm libraries it would require a UEL account . For this installation block devices will be used for ASM diskgroups. Setup udev rules for block devices with following ownerships and permissions.
#ASM OCR
KERNEL=="sdb[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
KERNEL=="sdc[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
KERNEL=="sdd[1]", OWNER="grid", GROUP="asmadmin", MODE="660"

# ASM DATA
KERNEL=="sde[1]", OWNER="grid", GROUP="asmadmin", MODE="660"

# ASM FLASH
KERNEL=="sdf[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
If the permissions are incorrect this will be flagged during OUI pre-req check.

10. Edit the cvu_config file and set CV_ASSUME_DISTID=OEL6 (more on 1454982.1 and explained here). This will prevent pdksh package check being flagged as failed when running the installer using OUI. Still running the cluster verification will flag this as failed but this could be ignored.
./runcluvfy.sh stage -pre crsinst -n rhel6m1,rhel6m2  -fixup  -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "rhel6m1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rhel6m1                               yes
  rhel6m2                               yes
Result: Node reachability check passed from node "rhel6m1"


Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
Result: User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed

Verification of the hosts config file successful


Interface information for node "rhel6m2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.86    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:8C:E5:85 1500
 eth0   192.168.0.90    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:8C:E5:85 1500
 eth1   192.168.1.88    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:E7:2E:BB 1500


Interface information for node "rhel6m1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.85    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:FF:7A:38 1500
 eth0   192.168.0.89    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:FF:7A:38 1500
 eth1   192.168.1.87    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:F7:AB:A0 1500


Check: Node connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m2[192.168.0.86]           rhel6m2[192.168.0.90]           yes
  rhel6m2[192.168.0.86]           rhel6m1[192.168.0.85]           yes
  rhel6m2[192.168.0.86]           rhel6m1[192.168.0.89]           yes
  rhel6m2[192.168.0.90]           rhel6m1[192.168.0.85]           yes
  rhel6m2[192.168.0.90]           rhel6m1[192.168.0.89]           yes
  rhel6m1[192.168.0.85]           rhel6m1[192.168.0.89]           yes
Result: Node connectivity passed for subnet "192.168.0.0" with node(s) rhel6m2,rhel6m1


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1:192.168.0.85            rhel6m2:192.168.0.86            passed
  rhel6m1:192.168.0.85            rhel6m2:192.168.0.90            passed
  rhel6m1:192.168.0.85            rhel6m1:192.168.0.89            passed
Result: TCP connectivity check passed for subnet "192.168.0.0"


Check: Node connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m2[192.168.1.88]           rhel6m1[192.168.1.87]           yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) rhel6m2,rhel6m1


Check: TCP connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1:192.168.1.87            rhel6m2:192.168.1.88            passed
Result: TCP connectivity check passed for subnet "192.168.1.0"


Interfaces found on subnet "192.168.0.0" that are likely candidates for VIP are:
rhel6m2 eth0:192.168.0.86 eth0:192.168.0.90
rhel6m1 eth0:192.168.0.85 eth0:192.168.0.89

Interfaces found on subnet "192.168.1.0" that are likely candidates for a private interconnect are:
rhel6m2 eth1:192.168.1.88
rhel6m1 eth1:192.168.1.87
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking ASMLib configuration.
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
Result: Check for ASMLib configuration passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       1.9597GB (2054884.0KB)    1.5GB (1572864.0KB)       passed
  rhel6m1       1.9597GB (2054884.0KB)    1.5GB (1572864.0KB)       passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       1.8421GB (1931556.0KB)    50MB (51200.0KB)          passed
  rhel6m1       1.7056GB (1788460.0KB)    50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       4GB (4194296.0KB)         2.9395GB (3082326.0KB)    passed
  rhel6m1       4GB (4194296.0KB)         2.9395GB (3082326.0KB)    passed
Result: Swap space check passed

Check: Free disk space for "rhel6m2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rhel6m2       /             22.5312GB     1GB           passed
Result: Free disk space check passed for "rhel6m2:/tmp"

Check: Free disk space for "rhel6m1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rhel6m1       /             21.4365GB     1GB           passed
Result: Free disk space check passed for "rhel6m1:/tmp"

Check: User existence for "grid"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists(502)
  rhel6m1       passed                    exists(502)

Checking for multiple users with UID value 502
Result: Check for multiple users with UID value 502 passed
Result: User existence check passed for "grid"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Result: Group existence check passed for "dba"

Check: Membership of user "grid" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           yes           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           yes           passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed

Check: Membership of user "grid" in group "dba"
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           passed
Result: Membership check for user "grid" in group "dba" passed

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       3                         3,5                       passed
  rhel6m1       3                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           hard          65536         65536         passed
  rhel6m1           hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          1024          1024          passed
  rhel6m1           soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           hard          16384         16384         passed
  rhel6m1           hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          2047          2047          passed
  rhel6m1           soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       x86_64                    x86_64                    passed
  rhel6m1       x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       2.6.32-220.el6.x86_64     2.6.9                     passed
  rhel6m1       2.6.32-220.el6.x86_64     2.6.9                     passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           3010          3010          250           passed
  rhel6m1           3010          3010          250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           385280        385280        32000         passed
  rhel6m1           385280        385280        32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           3010          3010          100           passed
  rhel6m1           3010          3010          100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           128           128           128           passed
  rhel6m1           128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           68719476736   68719476736   1052100608    passed
  rhel6m1           68719476736   68719476736   1052100608    passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4096          4096          4096          passed
  rhel6m1           4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4294967296    4294967296    2097152       passed
  rhel6m1           4294967296    4294967296    2097152       passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           6815744       6815744       6815744       passed
  rhel6m1           6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
  rhel6m1           between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4194304       4194304       262144        passed
  rhel6m1           4194304       4194304       262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4194304       4194304       4194304       passed
  rhel6m1           4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           1048576       1048576       262144        passed
  rhel6m1           1048576       1048576       262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           2097152       2097152       1048576       passed
  rhel6m1           2097152       2097152       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           3145728       3145728       1048576       passed
  rhel6m1           3145728       3145728       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       make-3.81-19.el6          make-3.80                 passed
  rhel6m1       make-3.81-19.el6          make-3.80                 passed
Result: Package existence check passed for "make"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       binutils-2.20.51.0.2-5.28.el6  binutils-2.15.92.0.2      passed
  rhel6m1       binutils-2.20.51.0.2-5.28.el6  binutils-2.15.92.0.2      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "gcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       gcc(x86_64)-4.4.6-3.el6   gcc(x86_64)-3.4.6         passed
  rhel6m1       gcc(x86_64)-4.4.6-3.el6   gcc(x86_64)-3.4.6         passed
Result: Package existence check passed for "gcc(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.105    passed
  rhel6m1       libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.105    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc(x86_64)-2.12-1.47.el6  glibc(x86_64)-2.3.4-2.41  passed
  rhel6m1       glibc(x86_64)-2.12-1.47.el6  glibc(x86_64)-2.3.4-2.41  passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
  rhel6m1       compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "elfutils-libelf(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       elfutils-libelf(x86_64)-0.152-1.el6  elfutils-libelf(x86_64)-0.97  passed
  rhel6m1       elfutils-libelf(x86_64)-0.152-1.el6  elfutils-libelf(x86_64)-0.97  passed
Result: Package existence check passed for "elfutils-libelf(x86_64)"

Check: Package existence for "elfutils-libelf-devel"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       elfutils-libelf-devel-0.152-1.el6  elfutils-libelf-devel-0.97  passed
  rhel6m1       elfutils-libelf-devel-0.152-1.el6  elfutils-libelf-devel-0.97  passed
Result: Package existence check passed for "elfutils-libelf-devel"

Check: Package existence for "glibc-common"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-common-2.12-1.47.el6  glibc-common-2.3.4        passed
  rhel6m1       glibc-common-2.12-1.47.el6  glibc-common-2.3.4        passed
Result: Package existence check passed for "glibc-common"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-devel(x86_64)-2.12-1.47.el6  glibc-devel(x86_64)-2.3.4  passed
  rhel6m1       glibc-devel(x86_64)-2.12-1.47.el6  glibc-devel(x86_64)-2.3.4  passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "glibc-headers"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-headers-2.12-1.47.el6  glibc-headers-2.3.4       passed
  rhel6m1       glibc-headers-2.12-1.47.el6  glibc-headers-2.3.4       passed
Result: Package existence check passed for "glibc-headers"

Check: Package existence for "gcc-c++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       gcc-c++(x86_64)-4.4.6-3.el6  gcc-c++(x86_64)-3.4.6     passed
  rhel6m1       gcc-c++(x86_64)-4.4.6-3.el6  gcc-c++(x86_64)-3.4.6     passed
Result: Package existence check passed for "gcc-c++(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.105  passed
  rhel6m1       libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.105  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libgcc(x86_64)-4.4.6-3.el6  libgcc(x86_64)-3.4.6      passed
  rhel6m1       libgcc(x86_64)-4.4.6-3.el6  libgcc(x86_64)-3.4.6      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libstdc++(x86_64)-4.4.6-3.el6  libstdc++(x86_64)-3.4.6   passed
  rhel6m1       libstdc++(x86_64)-4.4.6-3.el6  libstdc++(x86_64)-3.4.6   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libstdc++-devel(x86_64)-4.4.6-3.el6  libstdc++-devel(x86_64)-3.4.6  passed
  rhel6m1       libstdc++-devel(x86_64)-4.4.6-3.el6  libstdc++-devel(x86_64)-3.4.6  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       sysstat-9.0.4-18.el6      sysstat-5.0.5             passed
  rhel6m1       sysstat-9.0.4-18.el6      sysstat-5.0.5             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "pdksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       missing                   pdksh-5.2.14              failed
  rhel6m1       missing                   pdksh-5.2.14              failed
Result: Package existence check failed for "pdksh"

Check: Package existence for "expat(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       expat(x86_64)-2.0.1-9.1.el6  expat(x86_64)-1.95.7      passed
  rhel6m1       expat(x86_64)-2.0.1-9.1.el6  expat(x86_64)-1.95.7      passed
Result: Package existence check passed for "expat(x86_64)"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed

Check for consistency of root user's primary group passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "grid" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    does not exist
  rhel6m1       passed                    does not exist
Result: User "grid" is not part of "root" group. Check passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       0022                      0022                      passed
  rhel6m1       0022                      0022                      passed
Result: Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "domain.net" as found on node "rhel6m2"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
The DNS response time for an unreachable node is within acceptable limit on all nodes

File "/etc/resolv.conf" is consistent across nodes

Check: Time zone consistency
Result: Time zone consistency check passed

Pre-check for cluster services setup was unsuccessful on all the nodes.
If only the pdksh has the status as failed this could be ignored and consider the pre-check is successful.

11. Start the clusterware installation.

Since block devices are used for ASM disks set the disk discovery path to block devices.


Set OS group chosen for ASM administration

When CV_ASSUME_DISTID=OEL6 is set the OUI doesn't flag the pdksh but correctly identify the ksh pre-check requirement.

Summary

Begin the installation

12. Run root scripts when prompted

Running root.sh on first node (rhel6m1)
# /opt/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded

ASM created and started successfully.

Disk Group CLUSTER_DG created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 2c90c4f4a0fb4faebfb264244a836b96.
Successful addition of voting disk 1d7890a346284fa4bf0e278b2f3b1d3d.
Successful addition of voting disk 433f41571bf44fcebfe636a97d94777a.
Successfully replaced voting disk group with +CLUSTER_DG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   2c90c4f4a0fb4faebfb264244a836b96 (/dev/sdb1) [CLUSTER_DG]
 2. ONLINE   1d7890a346284fa4bf0e278b2f3b1d3d (/dev/sdc1) [CLUSTER_DG]
 3. ONLINE   433f41571bf44fcebfe636a97d94777a (/dev/sdd1) [CLUSTER_DG]
Located 3 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1'
CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.CLUSTER_DG.dg' on 'rhel6m1'
CRS-2676: Start of 'ora.CLUSTER_DG.dg' on 'rhel6m1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Running root.sh on the second node (rhel6m2)
# /opt/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rhel6m1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
CRS-4402 has been explained on (1212703.1), this is not an error on 11.2.0.3

13. Once the root.sh scripts have run on all nodes the clusterware installation will continue with running configuration assistants.

This will complete the clusterware installation as grid user. Check post installation configuration with
 cluvfy stage -post crsinst -n rhel6m1,rhel6m2
and using asmca created additional diskgroup that will be used when RAC DB is created.

14. Edit the cvu_config file in database software installation. Similar to clusterware check the cluvfy will flag pdksh as failed
cluvfy stage -pre dbinst -n rhel6m1,rhel6m2 -fixup -verbose
...
...
Check: Package existence for "pdksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       missing                   pdksh-5.2.14              failed
  rhel6m1       missing                   pdksh-5.2.14              failed
Result: Package existence check failed for "pdksh"
...
...
But OUI will correctly identify the ksh package requirement.

Apart from this rest of the installation steps are same as in RHEL 5.
Select software only install. Database will be created after the installation using DBCA

Select all nodes part of the cluster

Installation path under Oracle base

OS group for database software management

Summary

Begin installation and execute the root.sh script when prompted.


15. Before beginning database created with dbca set read,write and execute permission for cfgtoollogs for Oracle user.
cd $ORACLE_BASE
#chmod 770 cfgtoollogs
Select all nodes the instance will run and complete DB creation.

Continue wizard to complete the remaining steps.


Related Posts
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
Installing 11.2.0.3 on RHEL 6