Automatically start ASM/Database/EM/LSNR services

Posted by Sagar Patil
In theory there is no need to start services as they are brought online by Oracle Clusterware
Shell File dbora.txt

Copy dbora at /etc/init.d directory

# cd /etc/init.d # chmod 755 dbora # chown root:root dbora # ln -s /etc/init.d/dbora /etc/rc5.d/S99dbora # ln -s /etc/init.d/dbora /etc/rc0.d/K10dbora # ln -s /etc/init.d/dbora /etc/rc6.d/K10dbora

–  Change oratab entries from “N” to “Y” 

+ASM:/u01/app/oracle/product/10.1.0/db_1:Y      
TESTDB:/u01/app/oracle/product/10.1.0/db_1:Y   

–  Manually edit /etc/inittab so that entry to respawn init.cssd comes before running the runlevel 3

Orignal /etc/inittab file: (…)
# System initialization.
l5:5:wait:/etc/rc.d/rc 5
l6:6:wait:/etc/rc.d/rc 6
(…)
h1:35:respawn:/etc/init.d/init.cssd run >/dev/null 2>&1 </dev/null

Modified /etc/inittab file: (…)
# System initialization.
si::sysinit:/etc/rc.d/rc.sysinit
l0:0:wait:/etc/rc.d/rc 0
l1:1:wait:/etc/rc.d/rc 1
l2:2:wait:/etc/rc.d/rc 2
h1:35:respawn:/etc/init.d/init.cssd run >/dev/null 2>&1 </dev/null
l3:3:wait:/etc/rc.d/rc 3
(…)

TAF Failover Configuration and Testing

Posted by Sagar Patil

Configure the service on RAC servers for a failover

TNS Client side config

PROD =
(DESCRIPTION =
(enable=broken)
(LOAD_BALANCE = yes)
(ADDRESS = (PROTOCOL = TCP)(HOST = oravip01.oracledbasupport.com)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = oravip02.oracledbasupport.com)(PORT = 1521))
(CONNECT_DATA =
(SERVICE_NAME = prod)
(failover_mode=(type=select)(method=basic))
)
)

Let’s test a Failover – Connect to an Oracle Instance 1 or 2

oracle@ora02 ~]$ showcrs
HA Resource Target State
———– —— —–
ora.ora01.ASM1.asm ONLINE ONLINE on ora01
ora.ora01.LISTENER_ora01.lsnr ONLINE ONLINE on ora01
ora.ora01.gsd ONLINE UNKNOWN on ora01
ora.ora01.ons ONLINE UNKNOWN on ora01
ora.ora01.vip ONLINE ONLINE on ora01
ora.ora02.ASM2.asm ONLINE ONLINE on ora02
ora.ora02.LISTENER_ora02.lsnr ONLINE ONLINE on ora02
ora.ora02.gsd ONLINE UNKNOWN on ora02
ora.ora02.ons ONLINE UNKNOWN on ora02
ora.ora02.vip ONLINE ONLINE on ora02
ora.prod.db ONLINE ONLINE on ora01
ora.prod.prod.cs ONLINE ONLINE on ora02
ora.prod.prod.prod1.srv ONLINE ONLINE on ora01
ora.prod.prod.prod2.srv ONLINE ONLINE on ora02
ora.prod.prod1.inst ONLINE ONLINE on ora01
ora.prod.prod2.inst ONLINE ONLINE on ora02

SQL> select instance_name from v$instance;
INSTANCE_NAME
—————-
prod2

[oracle@ora02 ~]$ crs_stop ora.prod.prod2.inst
Attempting to stop `ora.prod.prod2.inst` on member `ora02`
Stop of `ora.prod.prod2.inst` on member `ora02` succeeded.
At this stage the connections are diverted to prod1 instance.

SQL> select instance_name from v$instance;
INSTANCE_NAME
—————-
prod1

[oracle@ora02 ~]$ showcrs
HA Resource Target State
———– —— —–
ora.ora01.ASM1.asm ONLINE ONLINE on ora01
ora.ora01.LISTENER_ora01.lsnr ONLINE ONLINE on ora01
ora.ora01.gsd ONLINE UNKNOWN on ora01
ora.ora01.ons ONLINE UNKNOWN on ora01
ora.ora01.vip ONLINE ONLINE on ora01
ora.ora02.ASM2.asm ONLINE ONLINE on ora02
ora.ora02.LISTENER_ora02.lsnr ONLINE ONLINE on ora02
ora.ora02.gsd ONLINE UNKNOWN on ora02
ora.ora02.ons ONLINE UNKNOWN on ora02
ora.ora02.vip ONLINE ONLINE on ora01
ora.prod.db ONLINE ONLINE on ora01
ora.prod.prod.cs ONLINE ONLINE on ora02
ora.prod.prod.prod1.srv ONLINE ONLINE on ora01
ora.prod.prod.prod2.srv ONLINE OFFLINE
ora.prod.prod1.inst ONLINE ONLINE on ora01
ora.prod.prod2.inst OFFLINE OFFLINE

[oracle@ora02 ~]$ crs_start ora.prod.prod2.inst
Attempting to start `ora.prod.prod2.inst` on member `ora02`
Start of `ora.prod.prod2.inst` on member `ora02` succeeded.

What happens if Server is restarted?

I am connected to prod2 instance and a reboot migrates my connection to prod1 automatically.

SQL> select instance_name from v$instance;
INSTANCE_NAME
—————-
prod2

SQL> select count(*) from
(select * from dba_source union select * from dba_source union select * from dba_source union select * from dba_source union select * from dba_source)
COUNT(*)
———-
292465

SQL> select instance_name from v$instance;
INSTANCE_NAME
—————-
prod1

Let’s see how a RAC Load balancing works? Write a small sql test Script (verify.sql) like below

REM the following query is for TAF connection verification
col sid format 999
col serial# format 9999999
col failover_type format a13
col failover_method format a15
col failed_over format a11
SELECT   sid,
 serial#,
 failover_type,
 failover_method,
 failed_over
 FROM   v$session
 WHERE   username = 'SU';

REM the following query is for load balancing verification
SELECT   instance_name FROM v$instance;
exit

REM We can also combine two queries:
col inst_id format 999
col sid format 999
col serial# format 9999999
col failover_type format a13
col failover_method format a15
col failed_over format a11
SELECT   inst_id,
 sid,
 serial#,
 failover_type,
 failover_method,
 failed_over
 FROM   gv$session
 WHERE   username = 'SU';

REM a simple select to see the distribution of users when testing connection : load balancing
 SELECT   inst_id, COUNT ( * )
 FROM   gv$session
GROUP BY   inst_id;

Write loop.sh file to make number SQL connections. Please copy and paste at least 100 entries of line below. Oracle Listener will load balance connections by diverting new connections to least loaded oracle RAC instance.

nohup sqlplus system/0ra01@failover @verify.sql &
sleep 1
nohup sqlplus system/0ra01@failover @verify.sql &
sleep 1
nohup sqlplus system/0ra01@failover @verify.sql &
sleep 1
nohup sqlplus system/0ra01@failover @verify.sql &
sleep 1

Run loop.sh and note down connections shared between RAC 1 & RAC 2 nodes

[oracle@ora01 scripts]$ grep prod2 nohup.out | wc -l
35
[oracle@ora01 scripts]$ grep prod1 nohup.out | wc -l
41

 

RAC | How to use SRVCTL Command

Posted by Sagar Patil

Check out current configuration information

srvctl config database Displays the configuration information of the cluster database.
srvctl config service Displays the configuration information for the services.
srvctl config nodeapps Displays the configuration information for the node applications.
srvctl config asm Displays the configuration for the ASM instances on the node.

Summary of srvctl commands.

Command Targets Description
srvctl add
srvctl modify
srvctl remove
database
instance
service
nodeapps
srvctl add / remove adds/removes target‘s configuration information to/from the OCR.srvctl modify allows you to change some of target‘s configuration information in the OCR without wiping out the rest.
srvctl relocate service Allows you to reallocate a service from one named instance to another named instance.
srvctl config database
service
nodeapps
asm
Lists configuration information for target from the OCR.
srvctl disable
srvctl enable
database
instance
service
asm
srvctl disable disables target, meaning CRS will not consider it for automatic startup, failover, or restart. This option is useful to ensure an object that is down for maintenance is not accidentally automatically restarted.srvctl enable reenables the specified object.
srvctl getenv
srvctl setenv
srvctl unsetenv
database
instance
service
nodeapps
srvctl getenv displays the environment variables stored in the OCR for target.srvctl setenv allows these variables to be set, and unsetenv unsets them.
srvctl start
srvctl status
srvctl stop
database
instance
service
nodeapps
asm
Start, stop, or display status (started or stopped) of target.

Adding a Database Service
srvctl add service -d <database_name> -s <5ervice_name> -r “<preferred list>”

londonl$ srvctl add service -6 RAC -s SERVICE2 -i “RACl,RAC2” -a “RAC3,RAC4”

Starting a Database Service
srvctl start service -d <database_name> [-s “<service_name_li$t>” [-1 <in$t_name>]] [-0 <start_options>] [-c <connect_str> | -q]

londonl$ srvctl start service -d RAC -s “SERVICEl,SERVICE2”

Stopping a Database Service
srvctl stop service -d <database_name> [-s “<service_naine_list>” [-1 <inst_name>]] [-C <connect_str> | -q] [-f]

londonl$ srvctl stop service -d RAC -s “SERVICE2,SERVICE3” -f

Checking the Current Database Service Configuration
srvctl config service -d <database_name> [-s <service_name>] [-a] [-S <level>]

londonl$ srvctl config service -d RAC -a
The -a option includes information about the configuration of TAF for the database service

Checking Current Database Service Status
srvctl status service -d <name> -s “<service_name_list>” [-f] [-v] [-S <level>]

londonl$ srvctl status service -d RAC -s “SERVICEl,SERVICE4”

Enabling and Disabling a Database Service
srvctl disable service -d <database_name> -s “<service_name_list>” [-i <in$t_name>]

londonl$ srvctl disable service -d RAC -s SERVICE2 -i RAC4

srvctl enable service -d <database_name> -s “<service_name_list>” [-i <inst_name>]

londonl$ srvctl enable service -d RAC -s SERVICE2 -i RAC4

Removing a Database Service
srvctl remove service -d <database_name> -s <service_name> [ – i <inst_narne>] [-f]

londonl$ srvctl remove service -d RAC -s SERVICE4

Relocating a Database Service
srvctl relocate service -d <database_name> -s <service_name> -i <old_inst_name> -r <new_inst_name> [-f]

londonl$ srvctl relocate service -d RAC -s SERVICES -i RAC2 -t RAC4

Administering Instances

Starting an instance : srvctl start instance -d prod -i “prod1,prod2”

Stopping an instance: srvctl stop instance -d prod -i “prod1,prod2”

Checking the status of an instance : srvctl status instance -d prod – i “prod1,prod2”

Adding a new instance configuration : srvctl add instance -d prod – i prod3 -n prod3_node

Removing an existing instance configuration: srvctl remove instance -d prod3-i prod3_node

Disabling an instance: srvctl disable instance -d prod -i “prod1,prod2”

Enabling an instance : srvctl enable instance -d prod -i “prod1,prod2

Oracle Clusterware Log/Clusterware log files

Posted by Sagar Patil

In Oracle 10.2, Oracle Clusterware log files are created in the $ORA_CRS_HOME/log directory.

Read more…

Oracle Installer can’t discover ASM disks Ora-15186 Asmlib Error =[ASM_OPEN], ERROR = [1]

Posted by Sagar Patil

I could get ASM commandline to discover ASM disks at both RAC nodes. It can map VOL1, VOL2 at both nodes but Oracle RAC database installer won’t see ASM disks.

Operating System Linux Red Hat Advanced Server OS Version 4.0 Update Version Nahant Update 3
On WYGORA01
[root@wygora01 software]# /etc/init.d/oracleasm createdisk VOL1 /dev/mapper/mpath5
Marking disk “/dev/mapper/mpath5” as an ASM disk: [ OK ]
[root@wygora01 software]# /etc/init.d/oracleasm createdisk VOL1 /dev/mapper/mpath6
root@wygora01 software]# /etc/init.d/oracleasm createdisk VOL2 /dev/mapper/mpath6
Marking disk “/dev/mapper/mpath6” as an ASM disk: [ OK ]
On WYGORA02
[root@wygora02 ~]# /etc/init.d/oracleasm scandisks
Scanning system for ASM disks: [ OK ]

[root@wygora02 ~]# /etc/init.d/oracleasm listdisks
VOL1
VOL2
[root@wygora01 ~]# /etc/init.d/oracleasm listdisks
VOL1
VOL2
Initially I was trying to get oracle installer to create ASM instance. Somehow installer couldn’t list VOL1 & VOL2 disks though ASM listdisk could see it. After carrying ASM dropdisk/createdisk fixed it but restart was causing same issue. I decided to go ahead with installation and see if Oracle builds an environment . Installer managed to start ASM1 instance but failed to build ASM2 instance on another RAC node.

I tried starting the ASM2 instance manually and came across error

SQL> startup pfile=/u01/app/oracle/product/10.2.0/asm/admin/+ASM/pfile/init.ora
ASM instance started
Total System Global Area 130023424 bytes
Fixed Size 2019032 bytes
Variable Size 102838568 bytes
ASM Cache 25165824 bytes
ORA-15032: not all alterations performed
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “DATA”
SQL> select PATH, MOUNT_STATUS, HEADER_STATUS, MODE_STATUS, STATE from V$ASM_DISK;
no rows selected
SQL> alter diskgroup all mount;
alter diskgroup all mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “DATA”

Reason behind Error

Multipath storage devices were accessed by different paths on both nodes.

Solution : Add asm_diskstring = ‘/dev/oracleasm/disks/*’ at ASM init.ora file & bounce the box

ORA-15186 ASMLIB ERROR FUNCTION = [ASM_OPEN], ERROR = [1], ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"

Posted by Sagar Patil

Oracle Clusterware installed OK. ASM1 instance works OK while ASM2 would return ORA-15186/ORA-15063

Read more…

OCFS2 Support Guide Linux/Solaris

Posted by Sagar Patil

This support guide is a supplement to the OCFS2 User’s Guide and the OCFS2 FAQ. The information provided is directed towards support. End users should consult theUsers’ Guide and/or FAQ for information on setting up and using OCFS2.

Read more…

Creating Data Guard Primary/Standby on 2 Windows Servers

Posted by Sagar Patil

Oracle Data Guard is the management, monitoring, and automation software infrastructure that creates, maintains, and monitors one or more standby databases to protect enterprise data from failures, disasters, errors, and corruptions.

The process below is applicable if you have separate Windows/Unix  servers for Datguard primary & Standby Install . For single machine Dataguard Install use my other post.

Oracle Data Guard Overview

Windows 2000 with Oracle 9.2.0.8
Primary Server is called “Primary”
Standby Server is called “Standby”
A working DNS is available for both servers.
Oracle Instance is called DATAGUARD on both Servers
TNS Listener is configured as Primary.world and Standby.world

IMP:
– Configure listener.ora /Tnsnames.ora and add both Primary as well as Standby Database Services
– Always Copy the up -to- date password file from Primary to Standby
– Use Spfile for all configuration settings

Init file for Primary Database InitPrimary.ora
Init file for Standby Database Inittstandby.ora

Sample Listener file listener.ora
Sample Tnsnames file tnsnames.ora

At Primary Database

1. Check Archivelog mode and force database into Archivelog

SQLPLUS>archive log list;
If this returns “Noarchivelog Mode” then
Add following parameters at initPrimary.ora ….
log_archive_dest_1 = ‘location=f:\Oracle\oradata\Primary MANDATORY REOPEN=5’
log_archive_format = “Primary_%S.ARC”
log_archive_start = true
log_buffer = 1000000
log_checkpoint_interval = 10000
log_checkpoint_timeout = 1800

SQL>Startup nomount;
SQL>Alter database archivelog;
Do “alter system switch logfile” and see archive files are dunped at right location

SQL> archive log list;
Standbybase log mode Archive Mode
Automatic archival Enabled
Archive destination f:\Oracle\primary\DATAGUARD
Oldest online log sequence 14
Next log sequence to archive 17
Current log sequence 17

SQL> create spfile from pfile;
File created.

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

At this stage
– COLD backup Primary Database files and ship it to Standby Server.
– At primary create standby control file
SQLPLUS> Alter Database Create Standby Controlfile as ‘D:\Oracle\Oradata\Primary\control.ctl’;
– Copy standby.ctl on to Standby Machine

SQL> startup;
SQL> show parameter spfile;
NAME TYPE VALUE
———————————— ———– ——————————
spfile string %ORACLE_HOME%\StandbyBASE\SPFILE%
ORACLE_SID%.ORA

SQL> connect / as sysdba
Connected.

SQL> alter system set dg_broker_start=TRUE scope=both;
System altered.

C:\Documents and Settings\oracle>dgmgrl
DGMGRL for 32-bit Windows: Version 9.2.0.1.0 – Production.
(c) Copyright 2002 Oracle Corporation. All rights reserved.
Welcome to DGMGRL, type “help” for information.

DGMGRL> connect sys/oracle
Connected.

DGMGRL> show configuration verbose;
Error: ORA-16532: Standby Guard configuration does not exist

SQL> connect / as sysdba
Connected.

SQL> show parameter dg_broker_start
NAME TYPE VALUE
———————————— ———– ——————————
dg_broker_start boolean TRUE

DGMGRL> show configuration verbose;
Error: ORA-16532: Standby Guard configuration does not exist

Above Error indicates we need configuration at Data Guard

Configure Data Guard

Create a new DRC and define a primary Site

DGMGRL> create configuration ‘Primary Site’
as primary site is ‘Primary_site’
resource is ‘Primary_db’
hostname is ‘Primary’
instance name is ‘dataguard’
service name is ‘primary’
site is maintained as physical;
Configuration “Primary Site” added with primary site “Primary_site”
Standbybase resource “Primary_db” added.
Create Secondary Site

DGMGRL> create site ‘Standby_site’
resource is ‘Standby_db’
hostname is ‘Standby’
instance name is ‘dataguard’
service name is ‘standby’
site is maintained as physical;
Site “Standby_site” added to configuration.
Standbybase resource “Standby_db” added.
View DRC config

DGMGRL> show configuration verbose;
Configuration
Name: ‘Primary Site ‘
Enabled: ‘no’
Default state: ‘ONLINE’
Intended state: ‘OFFLINE’
Protection Mode: ‘MaxPerformance’
Number of sites: 2
Sites:
Primary Site: Primary_site
Standby Site: Standby_site
Current status for “Primary Site”:
DISABLED
Enable DRC
Please review alrtlog as well as log for DMON process($BDUMP/alrtDMON.log).

DGMGRL> enable configuration
Enabled.

DGMGRL> show configuration;
Configuration ‘Primary Site’ is
Primary Site is ‘Primary_site’
Standby Site is ‘Standby_site’
Current status for “Genesys Newport”:
SUCCESS

DGMGRL> show resource Primary_db
Resource ‘Primary_db’ on site ‘Primary_site’
depends on ‘Primary_site’
Current status for “Primary_db”:
SUCCESS

DGMGRL> show resource Standby_db
Resource ‘Standby_db’ on site ‘Standby_site’
depends on ‘Standby_site’
Current status for “Standby_db”:
SUCCESS

DGMGRL> exit

At Physical Standby Database

– Configure listener.ora /Tnsnames.ora and add both Primary as well as Standby Database Services
– Always Copy up -to- date password file from Primary to Standby
– Use Spfile for all Configuration Settings
=========================================================================

Add following 2 parameters at standby init.ora file
standby_file_management=auto
standby_archive_dest=’f:\oracle\oradata\standby’

C:\Documents and Settings\oracle>oradim -new -sid DATAGUARD -startmode m

C:\Documents and Settings\oracle>set oracle_sid=DATAGUARD

C:\Documents and Settings\oracle>sqlplus /nolog
SQL*Plus: Release 9.2.0.8.0 – Production on Wed Feb 14 16:12:33 2007
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

SQL> connect / as sysdba
Connected to an idle instance.

SQL> startup nomount;
ORACLE instance started.
Total System Global Area 655843824 bytes
Fixed Size 455152 bytes
Variable Size 335544320 bytes
Database Buffers 318767104 bytes
Redo Buffers 1077248 bytes

SQL> alter database mount standby database;
Database altered.

SQL> recover managed standby database disconnect from session;
Media recovery complete.

SQL> archive log list;
Database log mode Archive Mode
Automatic archival Enabled
Archive destination f:\Oracle\oradata\primary
Oldest online log sequence 14
Next log sequence to archive 17
Current log sequence 17

SQL> create spfile from pfile;
File created.

SQL> shutdown immediate;
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.

SQL> startup mount;
ORACLE instance started.
Total System Global Area 655843824 bytes
Fixed Size 455152 bytes
Variable Size 335544320 bytes
Database Buffers 318767104 bytes
Redo Buffers 1077248 bytes
ORA-01666: controlfile is for a standby database
ORACLE instance shut down.

SQL> startup nomount;
ORACLE instance started.
Total System Global Area 655843824 bytes
Fixed Size 455152 bytes
Variable Size 335544320 bytes
Database Buffers 318767104 bytes
Redo Buffers 1077248 bytes

SQL> alter database mount standby database;
Database altered.

SQL> recover managed standby database disconnect from session;
Media recovery complete.

SQL> alter system set dg_broker_start=TRUE scope=both;
System altered.

SQL> show parameter dg_broker_start;
NAME TYPE VALUE
———————————— ———– ——————————
dg_broker_start boolean TRU

Init file for Primary Database InitPrimary.ora
Init file for Standby Database Inittstandby.ora

Sample Listener file listener.ora
Sample Tnsnames file tnsnames.ora

 

Top of Page

Top menu