Oracle 11gR2 RAC Installation with GNS, without SCAN

Step by step procedure for installing Oracle 11g (11gR2, 11.2.0.2) Real Application Clusters (RAC) on Linux (OEL 5.5) without SCAN, DNS,NTP setup and with GNS and CTSS is describe here. You can pretty much run this installation on a desktop pc at home. I have used Oracle VM linux guests in this exercise. As always, do let me know any questions or comments you might have.

Introduction
Pre-install tasks
Install grid infrastructure
Install database software and create database
Oracle Enterprise Manager (dbconsole)

Introduction

Oracle 11gR2 has brought in a few new features. Couple of interesting new features and notes about 11g grid infrastructure:

Patch set installation: Starting with 11.2.0, direct upgrades to the most recent patch set are supported, meaning you don’t have to install 11.2.0 and apply patch set to get it to 11.2.0.2. You can directly perform the 11.2.0.2 installation. Download the installable from patch 10098816 in My Oracle Support.
Out-of-place upgrades for CRS/Grid home are supported. CRS/Grid goes into a seperate , new oracle home, instead of same CRS home as before.
Oracle Automatic Storage Management Cluster File System (Oracle ACFS) is introduced. We know that ASMLib is integrated with OS kernel. ACFS is one step further, offering ASM disks as OS cluster file system. Regular file systems are needed for Oracle binaries, needing another LVM for these regular file systems. With ACFS, you do not need any other LVM to manage oracle storage (let it be binaries, data files, etc..)
ASM and CRS are delivered into the same home, now called Grid Home

Single client access name (SCAN) is introduced in Oracle 11gR2 Grid Infrastructure. SCAN is meant to facilitate single name for all Oracle clients to connect to the cluster database, irrespective of number of nodes and node location. Until now, we have to keep adding multiple address records in all clients tnsnames.ora, when a new node gets added to or deleted from the cluster.
Cluster Time Synchronization Service (CTSS) is introduced to take care of the time synchronization between cluster nodes. Clusterware (CRS or Grid) used to depend on external services for time sync. This has been one of the key problems with node evictions and cluster instability. CTSS would be configured during the grid installation if NTP is not configured, which means /etc/ntp.conf should not exist and ntpd should not be running. With CTSS, you have one less thing to worry..
Grid Naming Service (GNS) is introduced to manage DNS lookups for the cluster node names, SCAN, and dynamically assign IP addresses to Oracle cluster nodes using Dynamic Host Configuration Protocol (DHCP). In short, GNS uses both DNS and DHCP to better address network requirements of the RAC. GNS would be creating a DNS service for a sub domain (for example: grid.freeoraclehelp.com) and regular DNS server delegates name resolutions to this GNS Server for the sub domain. Of course, GNS is optional, which needs SCAN to be configured in existing DNS servers of your organization. SCAN needs three address(A) records in DNS. This could be pretty simple for DNS administrator to set this up. There is no PTR required for these IP addresses.
However, if you’re installing RAC at your home, where you don’t have standard DNS to set up SCAN, you can spin up DNS service on one of the RAC nodes and set up SCAN quite easily. Check my another blog post (Oracle 11gR2 RAC SCAN - DNS (bind) Configuration on Linux) to set up SCAN.
GNS assigns three IP addresses to SCAN VIPs, node VIPs, and even private interconnect interfaces. We just need to assign static IPs to the regular host names (as you normally would for any other linux server) only.
Multicasting is introduced in 11gR2 for private interconnect traffic. Check if your network supports multicasting before the installation. Refer to 1212703.1 for more information.
I have received a couple of requests for the RAC installation at home without SCAN setup. So, I would like to utilize both CTSS and GNS, to install grid infrastructure without SCAN or DNS server setup in this post.
This article explain a fresh 11gR2 installation from scratch. If you’re not installing it new and planning to upgrade, refer to Oracle 11gR2 RAC Upgrade from 10gR2 RAC on Linux
I have not set up ssh either. I wanted to have Installer (OUI) set this for both grid and oracle users
Two users are used in this install. Grid user for Grid infrastructure and oracle user for Oracle database software and databases. Role separation is another key feature in 11gR2. This comes handy in environments where storage and databases are managed by different teams.
OCR & Voting files should be stored in ASM Disk Groups from 11gR2 (for a new installation). Normal redundancy means 3 disks and high redundancy means 5 disks.

OCR & Voting ASM storage requirements:

Redundancy	Min no# of disks	OCR Only	Voting only	Both OCR & Voting (min space)
External	1	300MB	300MB	600MB
Normal	3	600MB	900MB	2GB
High	5	900MB	1.5GB	2.4GB

This is the minimum storage required.. So, better get some room here. Normal redundancy needs only two disks in a regular ASM disk group for data files.. and external needs min three disks. but, ASM disk group for OCR and Voting is special.

Grid Infrastructure home should NOT be under grid base and Oracle database home should be under oracle base. So, plan the homes accordingly. Also, /etc/resolv.conf should have GNS VIP listed, if GNS is going to be used.

Pre-install tasks

All of the following steps need to be executed on all cluster nodes (unless specified explicitly).

[root@rac4 ~]# groupadd -g 500 oinstall
[root@rac4 ~]# groupadd -g 501 asmadmin
[root@rac4 ~]# groupadd -g 502 asmdba
[root@rac4 ~]# groupadd -g 503 asmoper
[root@rac4 ~]# groupadd -g 504 oper
[root@rac4 ~]# groupadd -g 505 dba
[root@rac4 ~]# useradd -u 200  -g oinstall -G dba,oper,asmdba oracle
[root@rac4 ~]# useradd -u 201  -g oinstall -G asmdba,asmadmin,asmoper grid
[root@rac4 ~]# passwd oracle
[root@rac4 ~]# passwd grid

You may want to download asmlib rpm from http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html

[root@rac3 ~]# yum install oracleasm-support oracleasmlib oracleasm-`uname -r`
[root@rac3 ~]# rpm -qa |grep asm
oracleasmlib-2.0.4-1.el5
oracleasm-2.6.18-194.el5xen-2.0.5-1.el5
oracleasm-support-2.1.3-1.el5
[root@rac3 ~]#

Once ASMLib is installed, configure ASMlib:

[root@rac4 ~]# /usr/sbin/oracleasm configure -i -e -u grid -g asmadmin -o "xvd"

Create ASM Disk labels on one node only, because these disks are shared. All other pre-install tasks to be run on all rac nodes.

[root@rac4 ~]# /etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@rac4 ~]# /usr/sbin/oracleasm-discover 'ORCL:*'
Using ASMLib from /opt/oracle/extapi/32/asm/orcl/1/libasm.so
[ASM Library - Generic Linux, version 2.0.4 (KABI_V2)]
[root@rac4 ~]# /usr/sbin/oracleasm createdisk OCRV  /dev/xvdc1
Writing disk header: done
Instantiating disk: done
[root@rac4 ~]# /usr/sbin/oracleasm createdisk DATA1 /dev/xvdd1
Writing disk header: done
Instantiating disk: done
[root@rac4 ~]# /usr/sbin/oracleasm createdisk DATA2 /dev/xvde1
Writing disk header: done
Instantiating disk: done
[root@rac4 ~]# /usr/sbin/oracleasm createdisk DATA3 /dev/xvdf1
Writing disk header: done
Instantiating disk: done
[root@rac4 ~]# /usr/sbin/oracleasm-discover 'ORCL:*'
Using ASMLib from /opt/oracle/extapi/32/asm/orcl/1/libasm.so
[ASM Library - Generic Linux, version 2.0.4 (KABI_V2)]
Discovered disk: ORCL:DATA1 [4192902 blocks (2146765824 bytes), maxio 64]
Discovered disk: ORCL:DATA2 [4192902 blocks (2146765824 bytes), maxio 64]
Discovered disk: ORCL:DATA3 [4192902 blocks (2146765824 bytes), maxio 64]
Discovered disk: ORCL:OCRV [2088387 blocks (1069254144 bytes), maxio 64]
[root@rac4 ~]#

ASM disks should be visible on all other RAC nodes.

[root@rac3 ~]# /usr/sbin/oracleasm-discover 'ORCL:*'
Using ASMLib from /opt/oracle/extapi/32/asm/orcl/1/libasm.so
[ASM Library - Generic Linux, version 2.0.4 (KABI_V2)]
[root@rac3 ~]#  /etc/init.d/oracleasm disable
Writing Oracle ASM library driver configuration: done
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
[root@rac3 ~]# /etc/init.d/oracleasm enable  
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@rac3 ~]# /usr/sbin/oracleasm-discover 'ORCL:*'
Using ASMLib from /opt/oracle/extapi/32/asm/orcl/1/libasm.so
[ASM Library - Generic Linux, version 2.0.4 (KABI_V2)]
Discovered disk: ORCL:DATA1 [4192902 blocks (2146765824 bytes), maxio 64]
Discovered disk: ORCL:DATA2 [4192902 blocks (2146765824 bytes), maxio 64]
Discovered disk: ORCL:DATA3 [4192902 blocks (2146765824 bytes), maxio 64]
Discovered disk: ORCL:OCRV [2088387 blocks (1069254144 bytes), maxio 64]
[root@rac3 ~]#

[root@rac3 ~]# cat /etc/hosts
127.0.0.1       localhost.localdomain localhost
192.168.1.30   rac3.freeoraclehelp.com     rac3
192.168.1.40   rac4.freeoraclehelp.com     rac4

Install the following pre-requisite Linux packages

[root@rac3 ~]# yum install -y binutils compat-libstdc++-33 compat-libstdc++ elfutils-libelf elfutils-libelf-devel expat gcc gcc-c++ glibc glibc glibc-common glibc-devel glibc-headers libaio libaio libaio-devel libaio-devel libgcc libgcc libstdc++ libstdc++ libstdc++-devel make pdksh sysstat unixODBC unixODBC unixODBC-devel unixODBC-devel

Also, install cvuqdisk-1.0.9-1 rpm available from database installation dump directory.
Verify that multicast is supported in your network. The mcasttest script available in Note: 1212703.1.

[oracle@rac3 mcasttest]$ perl mcasttest.pl -n rac3,rac4 -i eth1
###########  Setup for node rac3  ##########
Checking node access 'rac3'
Checking node login 'rac3'
Checking/Creating Directory /tmp/mcasttest for binary on node 'rac3'
Distributing mcast2 binary to node 'rac3'
###########  Setup for node rac4  ##########
Checking node access 'rac4'
Checking node login 'rac4'
Checking/Creating Directory /tmp/mcasttest for binary on node 'rac4'
Distributing mcast2 binary to node 'rac4'
###########  testing Multicast on all nodes  ##########

Test for Multicast address 230.0.1.0

Sep 17 18:42:39 | Multicast Succeeded for eth1 using address 230.0.1.0:42000

Test for Multicast address 224.0.0.251

Sep 17 18:42:44 | Multicast Succeeded for eth1 using address 224.0.0.251:42001
[oracle@rac3 mcasttest]$

Change kernel parameters as per Oracle documentation and make sure that the same parameter is not duplicated. Here is my sysctl.conf

[root@rac3 ~]# tail /etc/sysctl.conf
kernel.shmall = 2097152
kernel.shmmax = 2147573760
kernel.shmmni = 4096
kernel.sem = 256 32000 100 142
fs.file-max = 6815744
net.ipv4.ip_local_port_range = 10000 65000
kernel.msgmni = 2878 
kernel.msgmax = 8192 
kernel.msgmnb = 65535
xen.independent_wallclock=1
fs.aio-max-nr = 1048576
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
[root@rac3 ~]# sysctl -p

Update ulimit settings for oracle and grid users. Add the following to /etc/security/limits.conf

cat >> /etc/security/limits.conf <<EOF
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
EOF

Run cluvfy script from the grid directory and address all the problems:

[root@rac3 ~]# /dumps/Database/11gR2/grid/runcluvfy.sh stage -pre crsinst -n rac3,rac4 -verbose

NTP Configuration has to be removed.

[root@rac3 ~]# mv /etc/ntp.conf /etc/ntp.conf.orig
[root@rac3 ~]# chkconfig ntpd off
[root@rac3 ~]# service ntpd stop

Install grid infrastructure

“Device Checks for ASM” seems to be a known bug. Please ignore.

/etc/resolv.conf does not have GNS as the name server. After correcting the nameserver to GNS VIP, Cluster verification succeeded.

Edit /etc/resolv.conf and replace

search freeoraclehelp.com
nameserver 192.168.1.1

search freeoraclehelp.com
nameserver 192.168.1.31

[root@rac4 ~]# nslookup scan.grid.freeoraclehelp.com
Server:         192.168.1.31
Address:        192.168.1.31#53

Name:   scan.grid.freeoraclehelp.com
Address: 192.168.1.101
Name:   scan.grid.freeoraclehelp.com
Address: 192.168.1.102
Name:   scan.grid.freeoraclehelp.com
Address: 192.168.1.103

[root@rac4 ~]#

Oracle 11gR2 RAC Installation

Install database software and create database

I have not run ASMCA or created disk groups manually after grid installation. So, there are no disk groups here for database file storage. I have manually created the ASM disk group for database on the first node:

SQL> CREATE DISKGROUP DATA_DG EXTERNAL REDUNDANCY DISK
  2  'ORCL:DATA1',
  3  'ORCL:DATA2',
  4  'ORCL:DATA3' ;

Diskgroup created.

SQL> 

SQL> select group_number,name,state,type,total_mb,free_mb,usable_file_mb from v$asm_diskgroup;
           1 OCRV_GDG                       MOUNTED     EXTERN       1019        623            623
           2 DATA_DG                        MOUNTED     EXTERN       6141       6087           6087

SQL>

Then, this mount this data group on the second RAC node.

SQL>  select group_number,name,state,type,total_mb,free_mb,usable_file_mb from v$asm_diskgroup;
           1 OCRV_GDG                       MOUNTED     EXTERN       1019        623            623
           0 DATA_DG                        DISMOUNTED                  0          0              0

SQL>  alter diskgroup DATA_DG mount ;

Diskgroup altered.

SQL>  select group_number,name,state,type,total_mb,free_mb,usable_file_mb from v$asm_diskgroup;
           1 OCRV_GDG                       MOUNTED     EXTERN       1019        623            623
           2 DATA_DG                        MOUNTED     EXTERN       6141       6044           6044

SQL> 

SQL> show parameter asm_diskgroups
asm_diskgroups                       string      DATA_DG
SQL>

Once disk group is created and mounted on all RAC nodes, try refresh button here. You should see the newly created disk group.