RAC Build on Solaris : First Phase

Objective : The two Solaris 10 prod nodes have Oracle 9i databases and I  want to export database to Oracle 10g X 4 node Solaris 10 RAC.

Exporting Database (from Oracle 9i Database):-

vi export.par
userid = “sys/change_on_install as sysdba” buffer 1MB FILESIZE=750MB file=dmpfil<1-20> FULL=Y DIRECT=Y LOG=exp_full.log
At later stage create identical tables paces with enough size in the target db and then import it.

Setting up 4 Solaris 10 nodes (tempracsrv3,tempracsr4) and (prodracsrv1,prodracsrv2) with NFS setup between them

For this Type of configuration you should have two Ethernet cards in each sever ,two switches for private interconnect(for networking the second eth. i.e. eth1).

Install Solaris 10 from ISO image or CD, Hit Enter to install in graphical mode.

Skip the media test and start the installation(IF ANY)
. Language Selection: <select your language preference>.
. Keyboard Configuration: <select your keyboard preference>.
. Installation Type: Custom.
. Disk Partitioning Setup: Automatically partition. .
Hostname –enter tempracsrv3

Same way install 4 nodes namely tempracsrv4, prodrac1,prodrac2

After installation give the ip address for eth0 and eth1 as follows

For tempracsrv3
192.168.2.131 tempracsrv3.mycorpdomain.com tempracsrv3 — for eth0
192.168.2.31 tempracsrv3-vip.mycorpdomain.com tempracsrv3-vip
10.10.10.31 tempracsrv3-priv.mycorpdomain.com tempracsrv3-priv —for eth1 ( leave gateway blank)

For tempracsrv4
192.168.2.132 tempracsrv4.mycorpdomain.com tempracsrv4
192.168.2.32 tempracsrv4-vip.mycorpdomain.com tempracsrv4-vip
10.10.10.32 tempracsrv4-priv.mycorpdomain.com tempracsrv4-priv – for eth1( leave gateway blank)

Connect the network wire from the second eth on each machine to the One Gigabit Switch)—Note this is separate switch

After that verify that both ip’s are pinging from both machines.

From tempracsrv3
Ping tempracsrv4
Ping tempracsrv4-priv
Ping tempracsrv3
Ping tempracsrv3-priv

From tempracsrv4
Ping tempracsrv3
Ping tempracsrv3-priv
ping tempracsrv4
Ping tempracsrv4-priv

Ensure that everything is pinging successfully. No need to worry about the vip.

After the Installation of Solaris 10 in tempracsrv3 . Make directory u01 under / Using the following command “mkdir /u01”

Create the oracle user.

As the root user, execute
Create group, —dba ,oinstall
# /usr/sbin/groupadd oinstall
# /usr/sbin/groupadd dba
# /usr/sbin/groupadd -g 200 oinstall
# /usr/sbin/groupadd -g 201 dba
# /usr/sbin/groupadd -g 202 oper

Test
# id -a oracle
Create user—oracle(oracle software owner)…

# /usr/sbin/useradd -u 200 -g oinstall -G dba[,oper] oracle
Verify that the user nobody exists in the system.

# /usr/sbin/useradd nobody
# passwd oracle

Preventing Oracle Clusterware Installation Errors Caused by stty Commands

During an Oracle Clusterware installation, Oracle Universal Installer uses SSH (if available) to run commands and copy files to the other nodes. During the installation, hidden files on the system (for example, .bashrc or .cshrc) will cause installation errors if they contain stty commands.

To avoid this problem, you must modify these files to suppress all output on STDERR, as in the following examples:

Configuring the oracle User’s Environment

Bourne, Bash, or Korn shell:
if [ -t 0 ]; then
stty intr ^C
fi

C shell:
test -t 0
if ($status == 0) then
stty intr ^C
endif

Configuring Kernel Parameters on Solaris 10

On Solaris 10 operating systems, verify that the kernel parameters shown in the following table are set to values greater than or equal to the recommended value shown. The table also contains the resource controls that replace the /etc/system file for a specific kernel parameter. The procedure following the table describes how to verify and set the values.

Configuring Kernel Parameters

Pre-Installation Tasks

On Solaris 10, use the following procedure to view the current value specified for resource controls, and to change them if necessary:

1. To view the current values of the resource control, enter the following commands:

# id -p // to verify the project id
uid=0(root) gid=0(root) projid=1 (user.root)
# prctl -n project.max-shm-memory -i project user.root
# prctl -n project.max-sem-ids -i project user.root

2. If you must change any of the current values, then:

To modify the value of max-shm-memory to 6 GB:
# prctl -n project.max-shm-memory -v 6442450944 -r -i project user.root
To modify the value of max-sem-ids to 256:
# prctl -n project.max-sem-ids -v 256 -r -i project user.root

Use the following procedure to modify the resource control project settings, so that they persist after a system restart:

Note: In Solaris 10, you are not required to make changes to the /etc/system file to implement the System V IPC. Solaris 10 uses the resource control facility for its implementation. However, Oracle recommends that you set both resource control and /etc/system/ parameters.

Operating system parameters not replaced by resource controls continue to affect performance and security on Solaris 10 systems. For further information, contact your Sun vendor. In case you have any problem just edit the /etc/system file.

Parameter Replaced by Resource Control : Recommended value

noexec_user_stack NA 1
semsys:seminfo_semmns project.max-sem-ids 100
semsys:seminfo_semmns NA 1024
semsys:seminfo_semmsl process.max-sem-nsems 256
semsys:seminfo_semvmx NA 32767
shmsys:shminfo_shmmax project.max-shm-memory 4294967295
shmsys:shminfo_shmmin NA 1
shmsys:shminfo_shmmni project.max-shm-ids 100
shmsys:shminfo_shmseg NA 10

Note: When you use the command prctl (Resource Control) to change system parameters, you do not need to restart the system for these parameter changes to take effect. However, the changed parameters do not persist after a system restart.

Checking UDP Parameter Settings

1. By default, Oracle instances are run as the oracle user of the dba group . A project with the name group.dba is created to serve as the default project for the oracle user. Run the command id to verify the default project for the oracle user:

# su – oracle
$ id -p
uid=100(oracle) gid=100(dba) projid=100(group.dba)
$ exit

2. To set the maximum shared memory size to 2 GB, run the projmod command:

# projmod -sK “project.max-shm-memory=(privileged,2G,deny)” group.dba
Alternatively, add the resource control value  project.max-shm-memory=(privileged,2147483648,deny) to the last field of the project entries for the Oracle project.

3. After these steps are complete, check the values for the /etc/project file using the following command:

# cat /etc/project
The output should be similar to the following:
system:0::::
user.root:1::::
noproject:2::::
default:3::::
group.staff:10::::
group.dba:100:Oracle default
project:::project.max-shmmemory=(privileged,2147483648,deny)

4. To verify that the resource control is active, check process ownership, and run the commands id and prctl, as in the following example:

# su – oracle
$ id -p
uid=100(oracle) gid=100(dba) projid=100(group.dba)

$ prctl -n project.max-shm-memory -i process $$
process: 5754: -bash
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-shm-memory
privileged 2.00GB – deny

Checking UDP Parameter Settings

The User Data Protocol (UDP) parameter settings define the amount of send and receive buffer space for sending and receiving datagrams over an IP network. these settings affect cluster interconnect transmissions. If the buffers set by these parameters are too small, then incoming UDP datagrams can be dropped due to insufficient space, which requires send-side retransmission. This can result in poor cluster performance.

On Solaris, the UDP parameters are udp_recv_hiwat and udp_xmit_hiwat. OnSolaris 10 the default values for these parameters are 57344 bytes. Oracle recommends that you set these parameters to at least 65536 bytes.

Note: For additional information, refer to the Solaris Tunable Parameters Reference Manual.

Checking the Operating System Requirements Setup with CVU. To check current settings for udp_recv_hiwat and udp_xmit_hiwat, enter the following commands:

# ndd /dev/udp udp_xmit_hiwat
# ndd /dev/udp udp_recv_hiwat

To set the values of these parameters to 65536 bytes in current memory, enter the following commands:

# ndd -set /dev/udp udp_xmit_hiwat 65536
# ndd -set /dev/udp udp_recv_hiwat 65536

To set the values of these parameters to 65536 bytes on system restarts, open the /etc/system file, and enter the following lines:

set udp:xmit_hiwat=65536
set udp:udp_recv_hiwat=65536

Checking the Hardware and Operating System Setup with CVU

/dev/dvdrom/crs/Disk1/cluvfy/runcluvfy.sh stage –post hwos –n node1,node2

Checking NFS Buffer Size Parameters

then you must set the values for the NFS buffer size parameters rsize and wsize to at least 16384. Oracle recommends that you use the value 32768. For example, if you decide to use rsize and wsize buffer settings with the value 32768, then update the /etc/vfstab file on each node with an entry similar to the following:

nfs_server:/vol/DATA/oradata /home/oracle/netapp nfs -yes
rw,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3

If you use NFS mounts, then Oracle recommends that you use the option forcedirectio to force direct I/O for better performance. However, if you add forcedirectio to the mount option, then the same mount point cannot be used for Oracle software binaries, executables, shared libraries, and objects. You can only use the forcedirectio option for Oracle data files, the OCR, and voting disks.

For these mount points, enter the following line:

nfs_server:/vol/DATA/oradata /home/oracle/netapp nfs -yes
rw,hard,nointr,rsize=32768,wsize=32768,tcp,noac,forcedirectio,vers=3

Create the oracle user environment file.

/export/home/oracle/.profile
export PS1=”`/bin/hostname -s`-> ”
export EDITOR=vi
export ORACLE_SID=devdb1
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=$ORACLE_BASE/product/10.2.0/crs_1
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:/bin:
/usr/bin:/usr/sbin:/usr/local/bin:/usr/X11R6/bin
umask 022

Create the filesystem directory structure. As the oracle user, execute

tempracsrv3-> mkdir -p $ORACLE_BASE/admin
tempracsrv3-> mkdir -p $ORACLE_HOME
tempracsrv3-> mkdir -p $ORA_CRS_HOME
tempracsrv3-> mkdir -p /u01/oradata/devdb

Increase the shell limits for the Oracle user. So that it has unlimited resources in the oracle user

Modify the /etc/hosts file.

# more /etc/hosts
127.0.0.1 localhost
192.168.2.131 tempracsrv3.mycorpdomain.com tempracsrv3
192.168.2.31 tempracsrv3-vip.mycorpdomain.com tempracsrv3-vip
10.10.10.31 tempracsrv3-priv.mycorpdomain.com tempracsrv3-priv
192.168.2.132 tempracsrv4.mycorpdomain.com tempracsrv4
192.168.2.32 tempracsrv4-vip.mycorpdomain.com tempracsrv4-vip
10.10.10.32 tempracsrv4-priv.mycorpdomain.com tempracsrv4-priv

Configuring SSH, Add in all the cluster.

$ ps -ef | grep sshd

Create RSA and DSA keys on each node: Complete the following steps on each node:

1. Log in as the oracle user.

2. If necessary, create the .ssh directory in the oracle user’s home directory and set the correct permissions on it:

$ mkdir ~/.ssh
$ chmod 700 ~/.ssh
$ chmod 700

3. Enter the following commands to generate an RSA key for version 2 of the SSH protocol:

$ /usr/bin/ssh-keygen -t rsa
At the prompts: Accept the default location for the key file.

Enter and confirm a pass phrase that is different from the oracle user’s password. This command writes the public key to the ~/.ssh/id_rsa.pub file and the  private key to the ~/.ssh/id_rsa file. Never distribute the private key to anyone.

4. Enter the following commands to generate a DSA key for version 2 of the SSH protocol:

$ /usr/bin/ssh-keygen -t dsa

At the prompts:

• Accept the default location for the key file
• Enter and confirm a pass phrase that is different from the oracle user’s password

This command writes the public key to the ~/.ssh/id_dsa.pub file and the private key to the ~/.ssh/id_dsa file. Never distribute the private key to anyone.

Add keys to an authorized key file: Complete the following steps:

1. On the local node, determine if you have an authorized key file (~/.ssh/authorized_keys). If the authorized key file already exists, then proceed to step 2. Otherwise, enter the following commands:

$ touch ~/.ssh/authorized_keys
$ cd ~/.ssh
$ ls

You should see the id_dsa.pub and id_rsa.pub keys that you have created.

2. Using SSH, copy the contents of the ~/.ssh/id_rsa.pub and ~/.ssh/id_dsa.pub files to the file ~/.ssh/authorized_keys, and provide the Oracle user password as prompted. This process is illustrated in the following syntax example with a two-node cluster, with nodes node1 and node2, where the Oracle user path is /home/oracle:

[oracle@node1 .ssh]$ ssh node1 cat /home/oracle/.ssh/id_rsa.pub >>
authorized_keys
oracle@node1’s password:
[oracle@node1 .ssh]$ ssh node1 cat /home/oracle/.ssh/id_dsa.pub >>
authorized_keys
[oracle@node1 .ssh$ ssh node2 cat /home/oracle/.ssh/id_rsa.pub >>
authorized_keys
oracle@node2’s password:
[oracle@node1 .ssh$ ssh node2 cat /home/oracle/.ssh/id_dsa.pub
>>authorized_keys
oracle@node2’s password:

3. Use SCP (Secure Copy) or SFTP (Secure FTP) to copy the authorized_keys file  to the Oracle user .ssh directory on a remote node. The following example is with SCP, on a node called node2, where the Oracle user path is /home/oracle:

[oracle@node1 .ssh]scp authorized_keys node2:/home/oracle/.ssh/

Note: Repeat this process for each node in the cluster.

Creating Required Operating System Groups and User

Pre-Installation Tasks

4. Repeat step 2 and 3 for each cluster node member. When you have added keys from each cluster node member to the authorized_keys file on the last node you want to have as a cluster node member, then use SCP to copy the complete authorized_keys file back to each cluster node member

5. Change the permissions on the Oracle user’s /.ssh/authorized_keys file on all cluster nodes:

$ chmod 600 ~/.ssh/authorized_keys

At this point, if you use ssh to log in to or run a command on another node, you are prompted for the pass phrase that you specified when you created the DSA key.

Enabling SSH User Equivalency on Cluster Member Nodes

To enable Oracle Universal Installer to use the ssh and scp commands without being prompted for a pass phrase, follow these steps:

1. On the system where you want to run Oracle Universal Installer, log in as the oracle user.

2. Enter the following commands:

$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add

3. At the prompts, enter the pass phrase for each key that you generated. If you have configured SSH correctly, then you can now use the ssh or scp commands without being prompted for a password or a pass phrase.

4. If you are on a remote terminal, and the local node has only one visual (which is typical), then use the following syntax to set the DISPLAY environment Variable:

Bourne, Korn, and Bash shells
$ export DISPLAY=hostname:0
C shell:
$ setenv DISPLAY 0
For example, if you are using the Bash shell, and if your hostname is node1, then enter the following command:
$ export DISPLAY=node1:0

5. To test the SSH configuration, enter the following commands from the same terminal session, testing the configuration of each cluster node, where tempracsrv3, tempracsrv4, and so on, are the names of nodes in the cluster:

$ ssh tempracsrv3 date
$ ssh tempracsrv4 date

Note: The Oracle user’s /.ssh/authorized_keys file on every node must contain the contents from all of the /.ssh/id_rsa.pub and /.ssh/id_dsa.pub files that you generated on all cluster nodes.

These commands should display the date set on each node. If any node prompts for a password or pass phrase, then verify that the ~/.ssh/authorized_keys file on that node contains the correct public keys. If you are using a remote client to connect to the local node, and you see a message similar to “Warning: No xauth data; using fake authentication data for X11 forwarding,” then this means that your authorized keys file is configured correctly, but your ssh configuration has X11 forwarding enabled. To correct this, proceed to step 6.

6. To ensure that X11 forwarding will not cause the installation to fail, create a user-level SSH client configuration file for the Oracle software owner user, as follows:

a. Using any text editor, edit or create the ~oracle/.ssh/config file.

b. Make sure that the ForwardX11 attribute is set to no. For example:

Host *
ForwardX11 no

Note: The first time you use SSH to connect to a node from a particular system, you may see a message similar to the following: The authenticity of host ‘node1 (140.87.152.153)’ can’t be established.

RSA key fingerprint is
7z:ez:e7:f6:f4:f2:4f:8f:9z:79:85:62:20:90:92:z9.
Are you sure you want to continue connecting (yes/no)?

Enter yes at the prompt to continue. You should not see this message again when you connect from this system to that node. If you see any other messages or text, apart from the date, then the installation can fail. Make any changes required to ensure that only the date is displayed when you enter these commands. You should ensure that any parts of login scripts that generate any output, or ask any questions, are modified so that they act only when the shell is an interactive shell.

At Ttempracsrv3

Create NFS mount point /u01. Create directory using following command Mkdir /u01

1. enable NFS sever by running the following:
svcadm -v enable -r network/nfs/server

3. Run the following command to share via NFS
share -F nfs -o rw /u01
Note: The above share command will not persist over reboots. To persist over reboots, add an entry to /etc/dfs/dfstab

At Tempracsrv4

4. Run the following command to mount from tempracsrv4: mount -F nfs tempracsrv3:/u01 /u01
Note: The above mount command will not persist over reboots. To persist over reboots, add the following line in /etc/vfstab:

tempracsrv3:/u01 – /u01 nfs – yes rw,soft

Now copy the first machine as we did for tempracsrv4 and configure it as temprac1,temprac2.(for failover).also configure /u02 as nfs in temprac1, and mount it in temprac2.

creating blank files which will be used later for placing voting disk and ocr

touch /u01/crs_config
touch /u01/voting_disk

Now configure prodrac1,prodrac2 as we did for tempracsrv3,tempracsrv4.(for failover).

Change the sid in .bash_profile from devdb to prod. Use the following ip’s for prodrac1,prodrac2

127.0.0.1 localhost
192.168.2.133 prodrac1.mycorpdomain.com prodrac1— eth0 on prodrac1
192.168.2.34 prodrac1-vip.mycorpdomain.com prodrac1-vip
10.10.10.33 prodrac1-priv.mycorpdomain.com prodrac1-priv –eth1 on prodrac1
192.168.2.133 prodrac2.mycorpdomain.com prodrac2—eth0 on prodrac2
192.168.2.34 prodrac2-vip.mycorpdomain.com prodrac2-vip
10.10.10.34 prodrac2-priv.mycorpdomain.com prodrac2-priv—eth1 on prodrac2