Following Script will display attributes for each configuration item/element displayed at each “level”

def show( hier, here = 0, scope = None ) :
if here < len( hier ) :
pad, Type = here * ‘  ‘, hier[ here ];
for e in AdminConfig.list( Type, scope ).splitlines() :
name  = AdminConfig.showAttribute( e, ‘name’ );
print ‘%s%s : %s ‘ % ( pad, Type, name );
if here < len( hier ) :
show( hier, here + 1, e );
show( [ ‘Cell’, ‘Node’, ‘Server’ ] );

wsadmin>show( [ ‘Cell’, ‘Node’, ‘Server’ ] );
Cell : Server1_Cell
Node : Server1_Manager
Server : dmgr
Node : Server1_Node01
Server : ihs
Server : nodeagent
Server : IHS2
Server : server_member1
Server : server_member2

After successful websphere install one will see “first steps” page as below.

Click on “Samples Gallery” which should pop up an html page with list of application samples

Naviagte to $WAS_HOME/samples and run install.bat (Windows) else install.sh (linux).

This script will install number of default applications listed below

“Startup” applications by selecting application & clicking on “start” option at Deployment Manager. Once done visit following URL and explore apps you wish

The first failure data capture (FFDC) feature preserves the information that is generated from a processing failure and returns control to the affected engines. The captured data is saved in a log file for analyzing the problem. FFDC is intended primarily for use by IBM Service.

Use following steps to configure number of days between FFDC log file purges. The value is in days.

The FFDC file management function removes the FFDC log files that have reached the maximum age and generates a message in the SystemOut.log file.

Procedure :

1. Open the ffdcRun.properties file. The file is located in the $WAS_ROOT/properties  directory
2. Change the value for the ExceptionFileMaximumAge property to the number of days between the FFDC log file purges. The value of the ExceptionFileMaximumAge property must be a positive number. The default is seven days. For example, ExceptionFileMaximumAge = 3 sets the default time to three days. The FFDC log file is purged after three days
3. Save the ffdcRun.properties file and exit
4. Repeat the previous steps to modify the ffdcStart.properties and ffdcStop.properties files

When stopping and restarting application servers, it is important to consider the correlation with the Web server plug-in retry interval. This is especially important in an environment with only a few application servers. You need to understand the rate between starting and stopping application servers in a cluster and the retry interval so that you do not get HTTP request failures when you stop application servers in a cluster.

The retry interval tells the Web server plug-in how long to wait before retrying an HTTP request against a specific application server. If you stop and start application servers in a cluster faster than the retry rate, then a situation can occur where the plug-in assumes that two or more application server are down, when in fact they are not.

Here is an example of this situation. Let us assume following:

• The retry interval is set to 60 seconds (which is the default)
• There are two application servers in the cluster
• We are running a servlet application
• We alternate taking the two servers down. We take them down and restart them in a 45 second interval.

The Web server plug-in sends a request to a specific application server and if that server is unavailable, it marks it as down.

The request is then sent to the next application server in the cluster. The plug-in does not retry the marked down server until the retry interval has passed.

In our example, because the retry interval is longer than the recycle time, there are time slots where one server’s retry count has not yet expired. So, the plug-in does not retry that server (even though it might be back up), and the other server is down. Thus, there might be a time slot where the Web server plug-in thinks that both servers are down.

Figure illustrates this example

In an environment with two application servers per cluster, the plug-in cannot forward the request to any application server and, thus, returns an HTTP request failure. If there are more than two application servers in the cluster, the plug-in routes the request to the remaining active servers in the cluster. However, depending on the difference between the retry interval and the application server recycle time, there might be more than two application servers that are perceived to be unavailable at any given time.
To avoid this problem, increase the recycle rate to be at least as long in duration as the retry interval. If your retry interval is 60 seconds, then delay at least 60 seconds between starting one application server and stopping another application server.

You can verify the retry interval setting either by using the Administrative Console

(Servers → Web servers → WebServer_Name → Plug-in properties → Request Routing)    or

by looking at the plugin_cfg.xml file

If it is not possible to add an appropriate delay between recycling servers, then ensure that your remaining application server environment has the capacity to handle requests for the application assuming that two (or more) of the application servers are unavailable at the same time. So, there are two maxims to remember:

1. Do not stop an application server in a shorter duration than the retry interval.
2. If you stop an application server in a shorter duration than the retry interval, ensure that the set of remaining active servers can handle the capacity of having two application servers unavailable for a period of time equal to the retry interval.

I had a working datagard configuration between 1 primary and 2 physical standby systems. Today I noticed my dataguard configuration files were missing from all servers (Primary/Standby).

DGMGRL> show configuration;
ORA-16532: Data Guard broker configuration does not exist
Configuration details cannot be determined by DGMGRL

Although my dataguard primary-standby were working with existing init values any switchover /failover thru Dataguard single command won’t be possible until broker configuration is in place.

At Both Primary and Secondary :  Set values for parameters fal_server, fal_client,  log_archive_config,
dg_broker_start, log_archive_dest_n

sys@ PRIMARY> show parameter fal
NAME                                 TYPE        VALUE
------------------------------------ ----------- ---------------------------
fal_client                           string      primary
fal_server                           string      secondary

sys@ SECONDARY> show parameter fal;
NAME                                 TYPE        VALUE
———————————— ———– ——————————
fal_client                           string      secondary
fal_server                           string      primary

sys@ PRIMARY/Secondary> alter system set fal_client=”;
System altered.

sys@ PRIMARY/Secondary> alter system set fal_server=”;
System altered.

sys@ PRIMARY/Secondary> alter system set log_archive_dest_2=”;
System altered.

sys@ PRIMARY/SECONDARY> show parameter log_archive_config;
NAME                                 TYPE        VALUE
———————————— ———– ——————————
log_archive_config                   string      dg_config=(secondary,primary)

sys@ PRIMARY/SECONDARY> alter system set log_archive_config=” scope=both;

System altered.
sys@ PRIMARY/SECONDARY> show parameter log_archive_config;
NAME                                 TYPE        VALUE
———————————— ———– ——————————
log_archive_config                   string

sys@ PRIMARY/SECONDARY> show parameter dg_broker_start
NAME                                 TYPE        VALUE
———————————— ———– ——————————
dg_broker_start                      boolean     FALSE

At this point I see there were no archives shipped/ applied at Standby Database

PRIMARY> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            C:\app\patilsa\product\11.2.0\dbhome_1\oradata\primary
Oldest online log sequence     338
Next log sequence to archive   340
Current log sequence           340

Waited for 10 minutes .. No Archive files shipped from Primary to Standby

sys@ SECONDARY> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     323
Next log sequence to archive   0
Current log sequence           328

Create a Dataguard Config

Enable Dataguard broker at both systems

sys@ SECONDARY> select instance_name,status from v$instance;
INSTANCE_NAME    STATUS
—————- ————
secondary        MOUNTED

sys@ PRIMARY> select instance_name,status from v$instance;
INSTANCE_NAME    STATUS
—————- ————
primary          OPEN

sys@ PRIMARY> ALTER SYSTEM SET dg_broker_start=TRUE SCOPE=spfile ;
System altered.

sys@ PRIMARY>  show parameter dg_broker
NAME                                 TYPE        VALUE
———————————— ———– ——————————
dg_broker_config_file1               string      C:\APP\PATILSA\PRODUCT\11.2.0\DBHOME_1\DATABASE\DR1PRIMARY.DAT
dg_broker_config_file2               string      C:\APP\PATILSA\PRODUCT\11.2.0\DBHOME_1\DATABASE\DR2PRIMARY.DAT
dg_broker_start                      boolean     TRUE

sys@ SECONDARY> show parameter dg_broker
NAME                                 TYPE        VALUE
———————————— ———– ——————————
dg_broker_config_file1               string      C:\APP\PATILSA\PRODUCT\11.2.0\DBHOME_1\DATABASE\DR1SECONDARY.DAT
dg_broker_config_file2               string      C:\APP\PATILSA\PRODUCT\11.2.0\DBHOME_1\DATABASE\DR2SECONDARY.DAT
dg_broker_start                      boolean     TRUE

DGMGRL> show configuration;
ORA-16532: Data Guard broker configuration does not exist

Configuration details cannot be determined by DGMGRL

DGMGRL> CREATE CONFIGURATION ‘sample’ AS PRIMARY DATABASE IS ‘primary’ CONNECT I
DENTIFIER IS primary;
Configuration “sample” created with primary database “primary”

DGMGRL> show configuration;

Configuration – sample

Protection Mode: MaxPerformance
Databases:
primary – Primary database

Fast-Start Failover: DISABLED

Configuration Status:
DISABLED

DGMGRL> ADD DATABASE ‘secondary’ AS CONNECT IDENTIFIER IS secondary MAINTAINED AS PHYSICAL;
Database “secondary” added
DGMGRL> show configuration;
Configuration – sample
Protection Mode: MaxPerformance
Databases:
primary   – Primary database
secondary – Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
DISABLED

DGMGRL> enable configuration;
Enabled

DGMGRL> show configuration;
Configuration – sample
Protection Mode: MaxPerformance
Databases:
primary   – Primary database
secondary – Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS

DGMGRL> switchover to secondary;
Performing switchover NOW, please wait…
New primary database “secondary” is opening…
Operation requires shutdown of instance “primary” on database “primary”
Shutting down instance “primary”…
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance “primary” on database “primary”
Starting instance “primary”…
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is “secondary”

sys@ SECONDARY> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     367
Next log sequence to archive   369
Current log sequence           369

sys@ PRIMARY> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            C:\app\patilsa\product\11.2.0\dbhome_1\oradata\primary
Oldest online log sequence     368
Next log sequence to archive   0
Current log sequence           369

Difference between init files :    INITprimary.ORA

Difference between init files :  INITsecondary.ORA

This is most likely because the special service db_unique_name_DGMGRL has not been registered properly with the listener.

This is one of the requirements when configuring Data Guard broker.

From Oracle Docs :

To enable DGMGRL to restart instances during the course of broker operations, a service with a specific name must be statically registered with the local listener of each instance. The value for the GLOBAL_DBNAME attribute must be set to a concatenation of db_unique_name_DGMGRL.db_domain.

sys@ PRIMARY> show parameter db_uniqu
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_unique_name                       string      primary

sys@ SECONDARY> show parameter db_unique
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_unique_name                       string      secondary

Updated Listener.ora  : I had to add GLOBAL_DBNAME for both databases

SID_LIST_LISTENER =
 (SID_LIST =
 (SID_DESC =
 (SID_NAME = primary)
 (ORACLE_HOME = C:\app\patilsa\product\11.2.0\dbhome_1)
 (GLOBAL_DBNAME = primary_DGMGRL)
 )
 (SID_DESC =
 (SID_NAME = secondary)
 (ORACLE_HOME = C:\app\patilsa\product\11.2.0\dbhome_1)
 (GLOBAL_DBNAME = secondary_DGMGRL)
 )
 )

Create  .run_awr with following details  :

"TNS-connect-string : recipient-list : hrs of AWR snapshot"
[oracle@ ~]$ cat .run_awr
prod:root@oracledbasupport.co.uk:11
I added this script in my crontab for a daily emails:

########## Daily Export of AWR reports
02 18 * * * /home/oracle/.awr_daily.sh  >> /home/oracle/awr.log 2>&1

[oracle@awr_reports]$ ls -lrt
-rw-r--r-- 1 oracle oracle 315104 Oct 26 10:02 AWR_26102010_1002_prod.HTML
-rw-r--r-- 1 oracle oracle 343839 Oct 26 18:02 AWR_26102010_1802_prod.HTML
-rw-r--r-- 1 oracle oracle 342611 Oct 27 18:02 AWR_27102010_1802_prod.HTML
-rw-r--r-- 1 oracle oracle 282057 Oct 28 18:02 AWR_28102010_1802_prod.HTML

1. Create AWR report between sysdate  and sysdate – hours (download)

#       The file “.run_awr” in the “$HOME” directory contains one or more
#       lines with the following format, three fields delimited by “semicolon”:
#
#               TNS-connect-string : recipient-list : hrs

    2.Create AWR report between sysdate-days  and sysdate – hours  (download)

#       The file “.run_awr” in the “$HOME” directory contains one or more
#       lines with the following format, three fields delimited by “semicolon”:
#
#               TNS-connect-string : recipient-list : daysInPast : hrs

#!/usr/bin/ksh

 #==============================================================================
# File:         run_awr.sh
# Type:         korn shell script
#
# Description:
#       UNIX Korn-shell script to run under the UNIX "cron" utility to
#       automatically generate and email Oracle "AWR" reports in HTML against
#       the database accessed via the specified TNS connect-string, to a
#       specified list of email addresses.
#
# Parameters:
#       Zero, one, or more parameters may be passed.  These parameters
#       are TNS connect-strings, each of which refer to entries in the
#       script's configuration file (named ".run_awr", described below).
#
#       If no parameters are specified, then the script processes all of
#       the lines in the configuration file.
#
#       For each of the parameters specified, the script will process
#       each of the corresponding lines in the configuration file.
#
#       Each TNS connect-string should be separated by whitespace.
#
# Configuration file:
#       The file ".run_awr" in the "$HOME" directory contains one or more
#       lines with the following format, three fields delimited by "commas":
#
#               TNS-connect-string : recipient-list : hrs
#
#       where:
#
#               TNS-connect-string      Oracle TNS connect-string for the db
#               recipient-list          comma-separated list of email addresses
#               hrs                     "sysdate - <hrs>" is the beginning
#                                       time of the AWR report and "sysdate"
#                                       is the ending time of the AWR report
#
# Modification history:
#==============================================================================
#
#------------------------------------------------------------------------------
# Set up Oracle environment variables...
#------------------------------------------------------------------------------
export ORACLE_SID=prod
export ORAENV_ASK=NO
. /usr/local/bin/oraenv > /dev/null 2>&1
unset ORAENV_ASK
#
#------------------------------------------------------------------------------
# Verify that the Oracle environment variables and directories are set up...
#------------------------------------------------------------------------------
if [[ "${ORACLE_HOME}" = "" ]]
then
 echo "ORACLE_HOME not set; aborting..."
 exit 1
fi
if [ ! -d ${ORACLE_HOME} ]
then
 echo "Directory \"${ORACLE_HOME}\" not found; aborting..."
 exit 1
fi
if [ ! -d ${ORACLE_HOME}/bin ]
then
 echo "Directory \"${ORACLE_HOME}/bin\" not found; aborting..."
 exit 1
fi
if [ ! -x ${ORACLE_HOME}/bin/sqlplus ]
then
 echo "Executable \"${ORACLE_HOME}/bin/sqlplus\" not found; aborting..."
 exit 1
fi
if [ ! -x ${ORACLE_HOME}/bin/tnsping ]
then
 echo "Executable \"${ORACLE_HOME}/bin/tnsping\" not found; aborting..."
 exit 1
fi
#
#------------------------------------------------------------------------------
# Set shell variables used by the shell script...
#------------------------------------------------------------------------------
_Pgm=AWR_`date '+%d%m%Y_%H%M'`
_RunAwrListFile=${HOME}/.run_awr
if [ ! -r ${_RunAwrListFile} ]
then
 echo "Script configuration file \"${_RunAwrListFile}\" not found;
aborting..."
 exit 1
fi
#
#------------------------------------------------------------------------------
# ...loop through the list of database instances specified in the ".run_awr"
# list file...
#
# Entries in this file have the format:
#
#       dbname:rcpt-list:hrs
#
# where:
#       dbname          - is the TNS connect-string of the database instance
#       rcpt-list       - is a comma-separated list of email addresses
#       hrs             - is the number of hours (from the present time)
#                         marking the starting point of the AWR report
#------------------------------------------------------------------------------
grep -v "^#" ${_RunAwrListFile} | awk -F: '{print $1" "$2" "$3}' | \
while read _ListDb _ListRcpts _ListHrs
do
 #----------------------------------------------------------------------
 # If command-line parameters were specified for this script, then they
 # must be a list of databases...
 #----------------------------------------------------------------------
 if (( $# > 0 ))
 then
 #
 #---------------------------------------------------------------
 # If a list of databases was specified on the command-line of
 # this script, then find that database's entry in the ".run_awr"
 # configuration file and retrieve the list of email recipients
 # as well as the #-hrs for the AWR report...
 #---------------------------------------------------------------
 _Db=""
 _Rcpts=""
 _Hrs=""
 for _SpecifiedDb in $*
 do
 #
 if [[ "${_ListDb}" = "${_SpecifiedDb}" ]]
 then
 _Db=${_ListDb}
 _Rcpts=${_ListRcpts}
 _Hrs=${_ListHrs}
 fi
 #
 done
 #
 #---------------------------------------------------------------
 # if the listed DB is not specified on the command-line, then
 # go onto the next listed DB...
 #---------------------------------------------------------------
 if [[ "${_Db}" = "" ]]
 then
 continue
 fi
 #---------------------------------------------------------------
 else    # ...else, if no command-line parameters were specified, then
 # just use the information in the ".run_awr" configuration file...
 #---------------------------------------------------------------
 _Db=${_ListDb}
 _Rcpts=${_ListRcpts}
 _Hrs=${_ListHrs}
 #
 fi
 #
 #----------------------------------------------------------------------
 # Verify that the name of the database is a valid TNS connect-string...
 #----------------------------------------------------------------------
 ${ORACLE_HOME}/bin/tnsping ${_Db} > /dev/null 2>&1
 if (( $? != 0 ))
 then
 echo "\"tnsping ${_Db}\" failed; aborting..."
 exit 1
 fi
 #
 #----------------------------------------------------------------------
 # Create script variables for the output files...
 #----------------------------------------------------------------------
 _TmpSpoolFile="/home/oracle/awr_reports/${_Pgm}_${_Db}.HTML"
 _AwrReportFile="${_Pgm}_${_Db}.html"
 #
 #----------------------------------------------------------------------
 # Call SQL*Plus, retrieve some database instance information, and then
 # call the AWR report as specified...
 #----------------------------------------------------------------------
 ${ORACLE_HOME}/bin/sqlplus -s /nolog << __EOF__ > /dev/null 2>&1
set echo off feedback off timing off pagesize 0 linesize 300 trimspool on
verify off heading off
connect / as sysdba

col dbid new_value V_DBID noprint
select  dbid from v\$database;

col instance_number new_value V_INST noprint
select  instance_number from v\$instance;

col snap_id new_value V_BID
select  min(snap_id) snap_id
from    dba_hist_snapshot
where   end_interval_time >= (sysdate-(${_Hrs}/24))
and     startup_time <= begin_interval_time
and     dbid = &&V_DBID
and     instance_number = &&V_INST;

col snap_id new_value V_EID
select  max(snap_id) snap_id
from    dba_hist_snapshot
where   dbid = &&V_DBID
and     instance_number = &&V_INST;

spool ${_TmpSpoolFile}
select  'BEGIN='||trim(to_char(begin_interval_time, 'HH24:MI')) snap_time
from    dba_hist_snapshot
where   dbid = &&V_DBID
and     instance_number = &&V_INST
and     snap_id = &&V_BID ;
select  'END='||trim(to_char(end_interval_time, 'HH24:MI')) snap_time
from    dba_hist_snapshot
where   dbid = &&V_DBID
and     instance_number = &&V_INST
and     snap_id = &&V_EID ;
spool off

select output from table(dbms_workload_repository.awr_report_html(&&V_DBID,
&&V_INST, &&V_BID, &&V_EID, 0))

spool /tmp/${_AwrReportFile}
/
exit success
__EOF__
 #
 #----------------------------------------------------------------------
 # Determine if the "start time" and "end time" of the AWR report was
 # spooled out...
 #----------------------------------------------------------------------
 if [ -f ${_TmpSpoolFile} ]
 then
 _BTstamp=`grep '^BEGIN=' ${_TmpSpoolFile} | awk -F= '{print
$2}'`
 _ETstamp=`grep '^END=' ${_TmpSpoolFile} | awk -F= '{print $2}'`
 fi
 #
 #----------------------------------------------------------------------
 # Determine if an AWR report was spooled out...
 #----------------------------------------------------------------------
#                if [ -f /tmp/${_AwrReportFile} ]
#                then
#
#                        uuencode /tmp/${_AwrReportFile} ${_AwrReportFile} | \
#                                mailx -s "AWR Report for ${_Db}
#        (${_BTstamp}-${_ETstamp} GMT)" ${_Rcpts}
#
#                fi
#
mv /tmp/${_AwrReportFile} ${_TmpSpoolFile}
done
#
#------------------------------------------------------------------------------
# Finish up...
#------------------------------------------------------------------------------
exit 0

In this post, I will show you how easy it is to fail-over on standby dataguard database when primary is not available.

sys@ PRIMARY> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            C:\app\product\11.2.0\dbhome_1\oradata
Oldest online log sequence     207
Next log sequence to archive   209
Current log sequence           209

sys@ SECONDARY> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     207
Next log sequence to archive   0
Current log sequence           209

DGMGRL> connect sys/manager@secondary
Connected.
DGMGRL> startup mount;
ORACLE instance started.
Database mounted.

DGMGRL> connect sys/manager@primary
Connected.
DGMGRL> shutdown abort;
ORACLE instance shut down.

DGMGRL> switchover to ‘secondary’;
Performing switchover NOW, please wait…
Error: ORA-01034: ORACLE not available
Error: ORA-16625: cannot reach database “primary”
Failed.
Unable to switchover, primary database is still “primary”

DGMGRL> connect sys/manager@secondary
Connected.
DGMGRL> failover to secondary;
Performing failover NOW, please wait…
Failover succeeded, new primary is “secondary”
DGMGRL> show configuration;
Configuration – sample
Protection Mode: MaxPerformance
Databases:
secondary – Primary database
primary   – Physical standby database (disabled)
ORA-16661: the standby database needs to be reinstated
fast-Start Failover: DISABLED
Configuration Status: SUCCESS

Please note Secondary which is now a PRIMARY Database , it’s ARCHIVE log sequence is reset to 1

sys@ SECONDARY> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     1
Next log sequence to archive   1
Current log sequence           1

If you try and start Primary database (failed over) it won’t be opened by dgmgrl , alrtlog should have following error message

Data Guard: version check completed
Data Guard determines a failover has occurred – this is no longer a primary database
ORA-16649 signalled during: alter database open…

This morning I moved my Standby database from Maxperformance to Maxprotection while it returned ORA-16826

DGMGRL> show configuration;
Configuration – sample
Protection Mode: MaxProtection
Databases:
primary   – Primary database
secondary – Physical standby database
Warning: ORA-16826: apply service state is inconsistent with the DelayMins
property
Fast-Start Failover: DISABLED
Configuration Status:
WARNING

When I tried to remove secondary database to correct ORA-16826, I have following error ORA-16627

DGMGRL> remove database ‘secondary’;
Error: ORA-16627: operation disallowed since no standby databases would remain to support protection mode

To correct this issue, I need to move it to MAXPerformance mode

DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS  MAXPerformance;
Succeeded.

DGMGRL> remove database secondary;
Removed database “secondary” from the configuration

DGMGRL> ADD DATABASE ‘secondary’ AS CONNECT IDENTIFIER IS secondary MAINTAINED A
S PHYSICAL;
Database “secondary” added

DGMGRL> show database secondary;
Database – secondary
Role:            PHYSICAL STANDBY
Intended State:  OFFLINE
Transport Lag:   (unknown)
Apply Lag:       (unknown)
Real Time Query: OFF
Instance(s): secondary
Database Status:
DISABLED

DGMGRL> enable database secondary;
Enabled.

DGMGRL> show configuration;
Configuration – sample
Protection Mode: MaxPerformance
Databases:
primary   – Primary database
secondary – Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS

DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS  MAXPROTECTION;
Error: ORA-16627: operation disallowed since no standby databases would remain to support protection mode
Failed.

DGMGRL> EDIT DATABASE ‘secondary’ SET PROPERTY ‘LogXptMode’=’SYNC’;
Property “LogXptMode” updated

DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS  MAXPROTECTION;
Operation requires shutdown of instance “primary” on database “primary”
Shutting down instance “primary”…
Database closed.
Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance “primary” on database “primary”
Starting instance “primary”…

DGMGRL> connect sys/manager@primary
Connected.
DGMGRL> show configuration;
Configuration – sample
Protection Mode: MaxProtection
Databases:
primary   – Primary database
secondary – Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS

Following post will give you an idea on Dataguard data protection mode in which you want configuration to run when configuration is enabled. The possible protection modes are:

MAXPROTECTION
MAXAVAILABILITY
MAXPERFORMANCE

Usage Notes :
Before you use EDIT CONFIGURATION command to set the protection mode to either MAXPROTECTION or MAXAVAILABILITY mode, ensure that standby redo log files are configured on a standby database.

* The following table shows the configuration protection modes and the minimum corresponding settings for log transport services:

The following table shows the configuration protection modes and the minimum corresponding
settings for log transport services:

The default protection mode for the configuration is MAXPERFORMANCE.

DGMGRL> show configuration
Configuration – sample
Protection Mode: MaxPerformance
Databases:
primary   – Primary database
secondary – Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS

To move the database from MAXPERFORMANCE to MAXPROTECTION or MAXAVAILABILITY we need to configure Log transport mode as SYNC

DGMGRL> EDIT DATABASE ‘primary’ SET PROPERTY ‘LogXptMode’=’SYNC’;
Property “LogXptMode” updated

DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS  MAXPROTECTION;
Error: ORA-16627: operation disallowed since no standby databases would remain to support protection mode
Failed.

DGMGRL> EDIT DATABASE ‘secondary’ SET PROPERTY ‘LogXptMode’=’SYNC’;
Error: ORA-16789: standby redo logs not configured
Failed.

Let’s add Standby Redo Logs at Secondary database

sys@ SECONDARY> select * from v$standby_log;
no rows selected

sys@ SECONDARY> ALTER DATABASE ADD STANDBY LOGFILE GROUP 4 SIZE 50M;
ALTER DATABASE ADD STANDBY LOGFILE GROUP 4 SIZE 50M
*
ERROR at line 1:
ORA-01156: recovery or flashback in progress may need access to files

sys@ SECONDARY> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE cancel;
Database altered.

sys@ SECONDARY> ALTER DATABASE ADD STANDBY LOGFILE GROUP 4 SIZE 50M;
Database altered.

sys@ SECONDARY> ALTER DATABASE ADD STANDBY LOGFILE GROUP 5 SIZE 50M;
Database altered.

sys@ SECONDARY> ALTER DATABASE ADD STANDBY LOGFILE GROUP 6 SIZE 50M;
Database altered.

sys@ SECONDARY> select * from v$standby_log;
GROUP# DBID                                        THREAD#  SEQUENCE#      BYTES  BLOCKSIZE       USED ARC STATUS     FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# N
EXT_TIME LAST_CHANGE# LAST_TIME
———- —————————————- ———- ———- ——
4 UNASSIGNED                                        0          0   5242
8800        512        512 YES UNASSIGNED             0                      0
0
5 UNASSIGNED                                        0          0   5242
8800        512        512 YES UNASSIGNED             0                      0
0
6 UNASSIGNED                                        0          0   5242
8800        512        512 YES UNASSIGNED             0                      0
0
3 rows selected.

sys@ SECONDARY> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT from session;
Database altered.

sys@ SECONDARY> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     145
Next log sequence to archive   0
Current log sequence           147

DGMGRL> EDIT DATABASE ‘secondary’ SET PROPERTY ‘LogXptMode’=’SYNC’;
Property “LogXptMode” updated

DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS  MAXPROTECTION;

Operation requires shutdown of instance “primary” on database “primary”
Shutting down instance “primary”…
Database closed.
Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance “primary” on database “primary”
Starting instance “primary”…

DGMGRL> show configuration;
Configuration – sample
Protection Mode: MaxProtection
Databases:
primary   – Primary database
secondary – Physical standby database
Warning: ORA-16826: apply service state is inconsistent with the DelayMins property How to resolve ORA-16826 error?
Fast-Start Failover: DISABLED
Configuration Status:
WARNING

Download them from  http://www.ibm.com/developerworks/websphere/library/samples/SampleScripts.html

ex1.py  – Create and modify a server, load an application onto the server, and start the server.
ex2.py  – Reverse of ex1.p1 Stop a server on a given node, uninstalls the given application, and removes the server from the configuration.
ex3.py  – Creates a cluster and starts the server
ex4.py  – Demonstrate the invocation of some useful problem
determination actions involving traces and thread dumps.
ex5.py  – Demonstrate the invocation of various application install commands.
ex6.py  – Produce a short summary of some configuration and runtime information about the WebSphere installation.
ex7.py  – Create a JDBCProvider object using a template.
ex8.py  – Demonstrate J2C Security configuration, including the installation of a J2CResourceAdapter and creation of a J2CConnectionFactory.
ex9.py  – Demonstrate the creation of a JDBCProvider, DataSource and CMPConnectorFactory objects
ex10.py – Demonstrate the creation and use of a variable that is used in the creation of a URLProvider object.
ex11.py – Demonstrate the setting of various port numbers
ex12.py – Demonstrate methods for updating configuration attributes that are lists

My DGMGRL configuration is returning warnings as below.

DGMGRL> show database primary;
Database – primary
Role:            PRIMARY
Intended State:  TRANSPORT-ON
Instance(s):
primary
Database Warning(s):
ORA-16789: standby redo logs not configured

Database Status:
WARNING

sys@ PRIMARY> SELECT GROUP#,THREAD#,SEQUENCE#,ARCHIVED,STATUS FROM V$STANDBY_LOG;
no rows selected

Let’s go ahead & add Standby Redo Log files at primary

sys@ PRIMARY> select bytes from v$log;
BYTES
———-
52428800
52428800
52428800
3 rows selected.

sys@ PRIMARY> select group#, member from v$logfile;
GROUP#                    MEMBER
——————————————————————————–
1     C:\APP\PATILSA\ORADATA\PRIMARY\REDO01.LOG
2    C:\APP\PATILSA\ORADATA\PRIMARY\REDO02.LOG
3    C:\APP\PATILSA\ORADATA\PRIMARY\REDO03.LOG
3 rows selected.

sys@ PRIMARY> ALTER DATABASE ADD STANDBY LOGFILE GROUP 1 SIZE 50M;
ALTER DATABASE ADD STANDBY LOGFILE GROUP 1 SIZE 50M
*
ERROR at line 1:
ORA-01184: logfile group 1 already exists

My primary database had 3 log file groups originally and I created 3 standby redo log groups using the following commands:

sys@ PRIMARY> ALTER DATABASE ADD STANDBY LOGFILE GROUP 4 SIZE 50M;
Database altered.
sys@ PRIMARY> ALTER DATABASE ADD STANDBY LOGFILE GROUP 5 SIZE 50M;
Database altered.
sys@ PRIMARY> ALTER DATABASE ADD STANDBY LOGFILE GROUP 6 SIZE 50M;
Database altered.

sys@ PRIMARY> select * from v$standby_log;
GROUP# DBID                                        THREAD#  SEQUENCE#      B
YTES  BLOCKSIZE       USED ARC STATUS     FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME LAST_CHANGE# LAST_TIME
———- —————————————- ———- ———- ——
4 UNASSIGNED                                        0          0   52428800        512        512 YES UNASSIGNED             0                      0
0
5 UNASSIGNED                                        0          0   52428800        512        512 YES UNASSIGNED             0                      0
0
6 UNASSIGNED                                        0          0   52428800        512        512 YES UNASSIGNED             0                      0
0
3 rows selected.

DGMGRL> show database primary;
Database – primary
Role:            PRIMARY
Intended State:  TRANSPORT-ON
Instance(s):
primary

Database Status:
SUCCESS

If you try and switchover at this point after adding the Standby Redo logs, you will receive a ORA-16826 from the Data Guard Broker.

DGMGRL> switchover to secondary;
Performing switchover NOW, please wait…
Error: ORA-16775: target standby database in broker operation has potential dataloss
Failed.
Unable to switchover, primary database is still “primary”

DGMGRL> show configuration verbose;
Configuration – sample
Protection Mode: MaxPerformance
Databases:
primary – Primary database
secondary – Physical standby database
Warning: ORA-16826: apply service state is inconsistent with the DelayMins
property
Fast-Start Failover: DISABLED
Configuration Status:
WARNING

DGMGRL> remove database secondary;
Removed database “secondary” from the configuration

DGMGRL> ADD DATABASE ‘secondary’ AS CONNECT IDENTIFIER IS secondary MAINTAINED AS PHYSICAL;
Database “secondary” added

DGMGRL> show database secondary;
Database – secondary
Role:            PHYSICAL STANDBY
Intended State:  OFFLINE
Transport Lag:   (unknown)
Apply Lag:       (unknown)
Real Time Query: OFF
Instance(s):
secondary
Database Status:
DISABLED

DGMGRL> enable database secondary;
Enabled.

DGMGRL> show configuration;
Configuration – sample
Protection Mode: MaxPerformance
Databases:
primary   – Primary database
secondary – Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS

DGMGRL> switchover to secondary;
Performing switchover NOW, please wait…
New primary database “secondary” is opening…
Operation requires shutdown of instance “primary” on database “primary”
Shutting down instance “primary”…
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance “primary” on database “primary”
Starting instance “primary”…
Unable to connect to database
ORA-12514: TNS:listener does not currently know of service requested in connectdescriptor Failed.
Warning: You are no longer connected to ORACLE.
Please complete the following steps to finish switchover:
start up and mount instance “primary” of database “primary”

Don’t worry on TNS error ORA-12514: I had to start my PRIMARY database Manually  to fix it.

sys@ SECONDARY> select instance_name, status from v$instance;
INSTANCE_NAME    STATUS
—————- ————
secondary        OPEN
sys@ SECONDARY> select count(*) from tab;
COUNT(*)
———-
4731

sys@ SECONDARY> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     129
Next log sequence to archive   131
Current log sequence           131

sys@ PRIMARY> select instance_name, status from v$instance;

INSTANCE_NAME    STATUS
—————- ————
primary          MOUNTED

sys@ PRIMARY> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            C:\app\patilsa\product\11.2.0\dbhome_1\oradata\pr
imary
Oldest online log sequence     131
Next log sequence to archive   0
Current log sequence           133

DGMGRL> connect sys/manager@secondary
Connected.
DGMGRL> show configuration verbose;

Configuration – sample

Protection Mode: MaxPerformance
Databases:
secondary – Primary database
primary – Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS

DGMGRL> show database ‘primary’ StatusReport;
STATUS REPORT
INSTANCE_NAME   SEVERITY ERROR_TEXT

DGMGRL> show database ‘secondary’ StatusReport;
STATUS REPORT
INSTANCE_NAME   SEVERITY ERROR_TEXT

There are no errors so everything is fine now.

Generally you don’t have to set up Dataguard on a single machine. But there are occasions when you have to build test environment for destruction and this post is tuned for such opportunity.

There are couple of things you need to worry about when primary & standby are on the same box. This post will address those issues. I have created a new Database called Primary using DBCA. The standby will be addressed as “Secondar” instance here. Log and init parameter files attached.

Files Used : INITprimary.ORA , INITsecondary.ORA

1.    Check the primary database name:

SQL> select name from v$database;
NAME
———
PRIMARY

2.    For physical Dataguard we need to have a Forced logging ON . Let’s check that and enable it.

SQL> select force_logging from v$database;
FOR
—
NO
SQL> alter database force logging;
Database altered.
SQL> select force_logging from v$database;
FOR
—
YES

3.    Create a password file

D:\oracle\product\10.2.0\db_1\database>orapwd file=PWDbsl01lab.ora password=manager entries=5
–To verify the new password file
D:\oracle\product\10.2.0\db_1\database>sqlplus /nolog

SQL*Plus: Release 10.2.0.1.0 – Production on Tue Jul 25 22:24:41 2006
Copyright (c) 1982, 2005, Oracle.  All rights reserved.

SQL> connect sys/manager as sysdba
Connected to an idle instance.

4.    Configure a Standby Redo Log

Ensure log file sizes are identical on the primary and standby database
SQL> select group#,thread#,bytes/1024/1024,members from v$log;
GROUP#    THREAD# BYTES/1024/1024    MEMBERS
———- ———- ————— ———-
1          1              50          1
2          1              50          1
3          1              50          1

Determine the appropriate number of standby redo log file groups
The minimal number should be 3+1=4

Verify related database parameters and settings
Maxlogfiles
Maxlogmembers

Create standby redo log file groups
SQL> alter database add standby logfile (‘C:\app\patilsa\oradata\primary\redo01d.log’) size 50M;
Database altered.
SQL> alter database add standby logfile (‘C:\app\patilsa\oradata\primary\redo02d.log’) size 50M;
Database altered.
SQL> alter database add standby logfile (‘C:\app\patilsa\oradata\primary\redo03d.log’) size 50M;
Database altered.

Step 5 Verify the standby redo log file groups were created
SQL> select group#,thread#,sequence#,archived,status from v$standby_log;

GROUP#    THREAD#  SEQUENCE# ARC STATUS
———- ———- ———- — ———-
4          0          0 YES UNASSIGNED
5          0          0 YES UNASSIGNED
6          0          0 YES UNASSIGNED

5.     Set Primary Database Initialization Parameters

SQL> create pfile from spfile;
File created.

–Content of initprimary.ora

primary.__db_cache_size=100663296
primary.__java_pool_size=4194304
primary.__large_pool_size=4194304
primary.__oracle_base='C:\app\patilsa'#ORACLE_BASE set from environment
primary.__pga_aggregate_target=146800640
primary.__sga_target=281018368
primary.__shared_io_pool_size=0
primary.__shared_pool_size=163577856
primary.__streams_pool_size=0
*.audit_file_dest='C:\app\patilsa\admin\primary\adump'
*.audit_trail='db'
*.compatible='11.2.0.0.0'
*.control_files='C:\app\patilsa\oradata\primary\control01.ctl','C:\app\patilsa\product\11.2.0\dbhome_1\oradata\primary\control02.ctl'
*.db_block_size=8192
*.db_domain=''
*.db_name='primary'
*.db_recovery_file_dest='C:\app\patilsa\product\11.2.0\dbhome_1\oradata'
*.db_recovery_file_dest_size=5218762752
*.dg_broker_start=TRUE
*.diagnostic_dest='C:\app\patilsa'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=primaryXDB)'
*.log_archive_format='ARC%S_%R.%T'
*.memory_target=427819008
*.open_cursors=300
*.processes=150
*.remote_login_passwordfile='EXCLUSIVE'
*.undo_tablespace='UNDOTBS1'

SQL> create spfile from pfile;
File created.

6.    Check DB Archiving

SQL> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     58
Next log sequence to archive   60
Current log sequence           60

If primary is not in Archive mode use following commands to Enable primary DB Archiving

SQL> startup mount;
ORACLE instance started.
Total System Global Area  167772160 bytes
Fixed Size                  1247900 bytes
Variable Size              83887460 bytes
Database Buffers           79691776 bytes
Redo Buffers                2945024 bytes
Database mounted.

SQL> alter database archivelog;
Database altered.
SQL> alter database open;
Database altered.

7.    Create a Backup Copy of the Standby Database

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

Now Copy all data files/Redos from “”C:\app\patilsa\oradata\primary”  to “C:\app\patilsa\oradata\secondary”

8.    Create a Control File for the Standby Database at Primary

Primary> startup mount
ORACLE instance started.

Total System Global Area  167772160 bytes
Fixed Size                  1247900 bytes
Variable Size              83887460 bytes
Database Buffers           79691776 bytes
Redo Buffers                2945024 bytes
Database mounted.

Primary> Alter Database Create Standby Controlfile as ‘C:\app\patilsa\oradata\secondary\control01.ctl’;
Database altered.

SQL> alter database open;
Database altered.

9.    Create the parameter file for Standby Database : InitStandby.ora

secondary.__db_cache_size=247463936
secondary.__java_pool_size=4194304
secondary.__large_pool_size=4194304
secondary.__oracle_base='C:\app\patilsa'#ORACLE_BASE set from environment
secondary.__pga_aggregate_target=339738624
secondary.__sga_target=507510784
secondary.__shared_io_pool_size=0
secondary.__shared_pool_size=243269632
secondary.__streams_pool_size=0
*.audit_file_dest='C:\app\patilsa\admin\secondary\adump'
*.audit_trail='db'
*.compatible='11.2.0.0.0'
*.control_files='C:\app\patilsa\oradata\secondary\control01.ctl','C:\app\patilsa\flash_recovery_area\secondary\control02.ctl'
*.db_block_size=8192
*.db_domain=''
*.db_name='primary'
*.db_unique_name='secondary'
*.db_recovery_file_dest='C:\app\patilsa\flash_recovery_area'
*.db_recovery_file_dest_size=5218762752
*.db_unique_name='secondary'
*.diagnostic_dest='C:\app\patilsa'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=secondaryXDB)'
*.log_archive_format='ARC%S_%R.%T'
*.memory_target=847249408
*.open_cursors=300
*.processes=150
*.remote_login_passwordfile='EXCLUSIVE'
*.standby_file_management='AUTO'
*.undo_tablespace='UNDOTBS1'
db_file_name_convert=('C:\app\patilsa\oradata\primary' 'C:\app\patilsa\oradata\secondary')
log_file_name_convert=('C:\app\patilsa\oradata\primary' 'C:\app\patilsa\oradata\secondary')
FAL_SERVER=primary
FAL_CLIENT=secondary
STANDBY_FILE_MANAGEMENT=auto 

*.db_name=’primary’ at secondary & *.db_unique_name=’secondary’ # If you don’t set this parameter the standby will fail to start with ORA-01103:

10.    Setup the Environment to Support the Standby Database

Create a windows-based service
D:\oracle\product\10.2.0\db_1>oradim -new -sid secondary -intpwd secondary -startmode manual
Instance created.

Create a password file
D:\oracle\product\10.2.0\db_1>set ORACLE_SID=secondar

C:\data guard\database>orapwd file=PWDsecondary.ora password=secondary entries=5

Configure listeners for the primary and standby database

SID_LIST_LISTENER =
 (SID_LIST =
 (SID_DESC =
 (SID_NAME = CLRExtProc)
 (ORACLE_HOME = C:\app\patilsa\product\11.2.0\dbhome_1)
 (PROGRAM = extproc)
 (ENVS = "EXTPROC_DLLS=ONLY:C:\app\patilsa\product\11.2.0\dbhome_1\bin\oraclr11.dll")
 )
 )
LISTENER =
 (DESCRIPTION_LIST =
 (DESCRIPTION =
 (ADDRESS = (PROTOCOL = TCP)(HOST = sagar-pc)(PORT = 1521))
 (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
 )
 )

Create a server parameter file for a Standby database
D:\oracle\product\10.2.0\db_1\database>set oracle_sid=secondary

D:\oracle\product\10.2.0\db_1\database>sqlplus /nolog

SQL*Plus: Release 10.2.0.1.0 – Production on Wed Jul 26 03:11:04 2006

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

SQL> connect sys/secondary as sysdba
Connected to an idle instance.

SQL> create spfile from pfile;
File created.

SQL> show parameter spfile;

NAME                                 TYPE        VALUE
———————————— ———– ——————————
spfile                               string      C:\APP\PATILSA\PRODUCT\11.2.0\DBHOME_1\DATABASE\SPFILESECONDARY.ORA

11.    Mount the standby DB

SQL> startup nomount;
ORACLE instance started.

Total System Global Area  535662592 bytes
Fixed Size                  1375792 bytes
Variable Size             281018832 bytes
Database Buffers          247463936 bytes
Redo Buffers                5804032 bytes
SQL> alter database mount standby database;
Database altered.

12.    Start Redo Apply

SQL> alter database recover managed standby database disconnect from session;
Database altered.

SQL> show parameter dump

NAME                                 TYPE        VALUE
———————————— ———– ——————————
background_core_dump                 string      partial
background_dump_dest string      c:\app\patilsa\diag\rdbms\secondary\secondary\trace
core_dump_dest                       string      c:\app\patilsa\diag\rdbms\secondary\secondary\cdump
max_dump_file_size                   string      unlimited
shadow_core_dump                     string      none
user_dump_dest                       string      c:\app\patilsa\diag\rdbms\secondary\secondary\trace

13.    Verify the status of primary and standby database

D:\oracle\product\10.2.0\db_1\database>set oracle_sid=primary
SQL> connect / as sysdba
Connected.

SQL> select instance_name, status from v$instance;
INSTANCE_NAME    STATUS
—————- ————
primary      OPEN

D:\oracle\product\10.2.0\db_1\database>set oracle_sid=secondar

SQL> select instance_name,status from v$instance;
INSTANCE_NAME    STATUS
—————- ————
secondar    MOUNTED
Both instance status are OK

Let’s configure a Dataguard BROKER utility for this configuration

• In order to get Dataguard Broker working make sure you configure a primary & Sandby with spfile
• Also make sure password file is same for both systems

sys@ PRIMARY>CREATE SPFILE FROM PFILE;
sys@ PRIMARY>SHUTDOWN IMMEDIATE
sys@ PRIMARY>STARTUP
sys@ Secondary>STARTUP MOUNT

To get a clean start, we have to remove all “evidence” left over from possible failed attempts earlier

1. Make sure the listener is configured with a special entry for DGMGRL
2. CleanUp archive log destination on the STANDBY database:  [ALTER SYSTEM SET log_archive_dest_1=”;]
3. Stop DG broker process (DMON) on both, primary and standby database: [ALTER SYSTEM SET dg_broker_start=FALSE SCOPE=spfile SID=’*’;]
4. Delete any existing DG broker configuration files from the following locations:

$ORACLE_BASE/admin/<db_unique_name> or
$ORACLE_HOME/dbs, namely dr1<db_unique_name>.dat and dr2<db_unique_name>.dat

5. Start the DG broker process (DMON) again on both, the primary and standby databases:
ALTER SYSTEM SET dg_broker_start=TRUE  SCOPE=spfile;

Make sure it is running – check output of SHOW PARAMETER DG

sys@ PRIMARY> ALTER SYSTEM SET dg_broker_start=TRUE SCOPE=spfile ;
System altered.
sys@ PRIMARY> shutdown abort;
ORACLE instance shut down.
sys@ PRIMARY> startup;
ORACLE instance started.
Total System Global Area  426852352 bytes
Fixed Size                  1375060 bytes
Variable Size             318768300 bytes
Database Buffers          100663296 bytes
Redo Buffers                6045696 bytes
Database mounted.
Database opened.
sys@ PRIMARY> SHOW PARAMETER DG
NAME                                 TYPE        VALUE
———————————— ———– ——————————
dg_broker_start                      boolean     TRUE
sys@ PRIMARY> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 – Production With the Partitioning, OLAP, Data Mining and Real Application Testing options

C:\Documents and Settings\patilsa>dgmgrl
DGMGRL> connect sys/manager@primary
Connected.
DGMGRL> CREATE CONFIGURATION ‘sample’ AS PRIMARY DATABASE IS ‘primary’ CONNECT IDENTIFIER IS primary;
Configuration “sample” created with primary database “primary”
DGMGRL> show  configuration
Configuration – sample
Protection Mode: MaxPerformance
Databases:
primary – Primary database
Fast-Start Failover: DISABLED
Configuration Status:
DISABLED

DGMGRL> ADD DATABASE ‘secondary’ AS CONNECT IDENTIFIER IS secondary MAINTAINED A
S PHYSICAL;
Error: ORA-16796: one or more properties could not be imported from the database
Failed.

ORA-16796 as Standby database was not started in Recovery mode. I had to run following commands at standby.

sys@ SECONDARY> startup Mount;
sys@ SECONDARY> RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;

DGMGRL> ADD DATABASE ‘secondary’ AS CONNECT IDENTIFIER IS secondary MAINTAINED AS PHYSICAL;
Database “secondary” added
DGMGRL> show configuration;
Configuration – sample
Protection Mode: MaxPerformance
Databases:
primary   – Primary database
secondary – Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
DISABLED

DGMGRL > ENABLE CONFIGURATION ;
Enabled.

DGMGRL> SHOW CONFIGURATION
Configuration – sample
Protection Mode: MaxPerformance
Databases:
primary   – Primary database
secondary – Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS

dataguard broker primary after Install.log ,  dataguard broker secondary after Install.log

DGMGRL> show database verbose 'primary';
Database - primary
  Role:            PRIMARY
  Intended State:  TRANSPORT-ON
  Instance(s):   primary
  Properties:
    DGConnectIdentifier             = 'primary'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'MANUAL'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '30'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = ''
    LogFileNameConvert              = ''
    FastStartFailoverTarget         = ''
    StatusReport                    = '(monitor)'
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'sagar-pc'
    SidName                         = 'primary'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=Sagar-pc)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=primary_DGMGRL)(INSTANCE_NAME=primary)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'C:\app\patilsa\product\11.2.0\dbhome_1\oradata\primary'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = 'ARC%S_%R.%T'
    TopWaitEvents                   = '(monitor)'
Database Status:
SUCCESS

DGMGRL> show database verbose 'secondary';
Database - secondary
  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       19 hours 47 minutes 43 seconds
  Real Time Query: ON
  Instance(s):   secondary
  Properties:
    DGConnectIdentifier             = 'secondary'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ARCH'
    DelayMins                       = '0'
    Binding                         = 'OPTIONAL'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '0'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '30'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = 'C:\app\patilsa\oradata\primary, C:\app\patilsa\oradata\secondary'
    LogFileNameConvert              = 'C:\app\patilsa\oradata\primary, C:\app\patilsa\oradata\secondary'
    FastStartFailoverTarget         = ''
    StatusReport                    = '(monitor)'
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'sagar-pc'
    SidName                         = 'secondary'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=sagar-pc)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=secondary_DGMGRL)(INSTANCE_NAME=secondary)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = 'ARC%S_%R.%T'
    TopWaitEvents                   = '(monitor)'
Database Status:
SUCCESS

DGMGRL> show database 'primary' StatusReport;
STATUS REPORT
       INSTANCE_NAME   SEVERITY ERROR_TEXT

DGMGRL> show database 'secondary' StatusReport;
STATUS REPORT
       INSTANCE_NAME   SEVERITY ERROR_TEXT

No Errors at   StatusReport means everything is fine. Let’s have a look at Archive listings on Primary and Standby

sys@ PRIMARY> alter system switch logfile;
System altered.
sys@ PRIMARY> alter system switch logfile;
System altered.
sys@ PRIMARY> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            C:\app\patilsa\product\11.2.0\dbhome_1\oradata\primary
Oldest online log sequence     102

sys@ PRIMARY> SELECT PROCESS, STATUS, THREAD#, SEQUENCE#, BLOCK#, BLOCKS FROM V$
MANAGED_STANDBY;
PROCESS   STATUS          THREAD#  SEQUENCE#     BLOCK#     BLOCKS
--------- ------------ ---------- ---------- ---------- ----------
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CLOSING               1         93          1       1361
ARCH      CLOSING               1         97          1          1
ARCH      CLOSING               1         94          1          1
ARCH      CLOSING               1         95          1         11
ARCH      CLOSING               1         99          1         26
ARCH      CLOSING               1         94          1          1
ARCH      CLOSING               1         96          1          2
ARCH      CLOSING               1         98       2049        510
ARCH      CLOSING               1         98          1       2558
ARCH      CLOSING               1         96          1          2
ARCH      CLOSING               1         99          1         26
ARCH      CONNECTED             0          0          0          0
ARCH      CLOSING               1        100          1        136
ARCH      CLOSING               1        101          1         17
ARCH      CLOSING               1        102          1          8
ARCH      CLOSING               1        101          1         17
ARCH      CLOSING               1        102          1          8
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CLOSING               1        103          1          8
ARCH      CLOSING               1        103          1          8
ARCH      CONNECTED             0          0          0          0
ARCH      CLOSING               1         97          1          1
ARCH      CLOSING               1         95          1         11
ARCH      CONNECTED             0          0          0          0
ARCH      CLOSING               1         93          1       1361
30 rows selected.

sys@ SECONDARY> SELECT PROCESS, STATUS, THREAD#, SEQUENCE#, BLOCK#, BLOCKS FROM
V$MANAGED_STANDBY;
PROCESS   STATUS          THREAD#  SEQUENCE#     BLOCK#     BLOCKS
--------- ------------ ---------- ---------- ---------- ----------
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
ARCH      CONNECTED             0          0          0          0
MRP0 WAIT_FOR_LOG 1 104 0 0
RFS       IDLE                  0          0          0          0
RFS       IDLE                  0          0          0          0
RFS       IDLE                  0          0          0          0
RFS       IDLE                  0          0          0          0
RFS       IDLE                  0          0          0          0
RFS       IDLE                  0          0          0          0
RFS       IDLE                  0          0          0          0
RFS       IDLE                  0          0          0          0
RFS       IDLE                  0          0          0          0
RFS       IDLE                  0          0          0          0
RFS       IDLE                  0          0          0          0
RFS       IDLE                  0          0          0          0
43 rows selected.

See tail of Broker log files on what broker does when switchover happens    : drcprimary.log & drcsecondary.log

DGMGRL> show configuration;
Configuration – sample
Protection Mode: MaxPerformance
Databases:
secondary – Primary database
primary   – Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS

DGMGRL> show database ‘secondary’ StatusReport;
STATUS REPORT
INSTANCE_NAME   SEVERITY ERROR_TEXT

DGMGRL> show database ‘primary’ StatusReport;
STATUS REPORT
INSTANCE_NAME   SEVERITY ERROR_TEXT

Let’s carry another switchover to make sure it’s all OK.

Check if previous version installed by root

If WAS was previously installed as root, it would have created the temp dir /tmp/niflogs, owned by root. When you install as was61, it needs to write to this dir, so remember to chown / chmod this dir, or the install will fail in a very obscure way.

Uninstall WAS

First uninstall WAS if already installed (should be done as the same user account as originally installed it – probably root. Future installs should be installed as an application user, eg was61)

Uninstall app server

cd /opt/IBM/WebSphere/AppServer/uninstall
./uninstall -silent
cd ..
rm -Rf /opt/IBM/WebSphere/AppServer/*

Uninstall http server

cd /opt/IBM/HTTPServer/uninstall
./uninstall -silent
cd ..
rm -Rf /opt/IBM/HTTPServer/*

Uninstall updater

cd /opt/IBM/WebSphere/UpdateInstaller/uninstall
./uninstall -silent

cd ..
rm -Rf /opt/IBM/WebSphere/UpdateInstaller/*

Start installation as user was61

Change perms for http logs , first as root need to change ownerships on log dir: chown was61:web HTTPServer

Install IHS

As user was61: install ihs

cd /opt/IBM/source/webserver/IHS

vi responsefile.txt

-OPT silentInstallLicenseAcceptance=”true”
-OPT allowNonRootSilentInstall=”true”
-OPT disableOSPrereqChecking=”true”
-OPT installLocation=”/opt/IBM/HTTPServer”
-OPT installGSKit=”true”
-OPT httpPort=”80″
-OPT adminPort=”8008″
-OPT createAdminAuth=”false”
-OPT installPlugin=”true”
-OPT webserverDefinition=”ihs”
-OPT washostname=”server1″

Edit to change hostnames, also add non-root silent install  by adding -OPT allowNonRootSilentInstall=”true”

run  $./install -options responsefile.txt -silent

Check log files

cd /opt/IBM/HTTPServer/logs/install

While installation is going on you will see a filename “installconfig.log” and later log.txt

[was61@install]$ tail -f installconfig.log
<method>messageLogged</method>
<thread>11</thread>
<message>Execute:Java13CommandLauncher: Executing ‘/opt/IBM/source/webserver/plugin/install’ with arguments:
‘-silent’
‘-options’
‘/opt/IBM/HTTPServer/properties/version/nif/config/install/null1014430787’
The ‘ characters around the executable and arguments are
not part of the command.</message>
</record>

When log.txt is available search for word “INSTCONFSUCCESS” for successful installation.

(Nov 15, 2008 10:30:19 AM), Process, com.installshield.product.service.product.PureJavaProductServiceImpl, dbg.install, JVM memory after installing LA Files (lafiles): free=25386864 total=50331648
(Nov 15, 2008 10:30:19 AM), Process, com.installshield.product.service.registry.PureJavaRegistryServiceImpl, dbg.registry, writing VPD to /home/was61/vpd.properties
(Nov 15, 2008 10:30:23 AM), Process, com.installshield.wizardx.conditions.PlatformWizardBeanCondition, dbg.platform, target platform: name=”Linux” version=”2.6.18-194.8.1.el5″ arch=”amd64″
(Nov 15, 2008 10:30:23 AM), Process, com.installshield.wizardx.conditions.PlatformWizardBeanCondition, dbg.platform, condition platform: name=”Windows .*” version=”.” arch=”.”
(Nov 15, 2008 10:30:23 AM), Process, com.ibm.ws.install.ni.ismp.actions.SettleNIFRegistryAction, msg1, Current install/uninstall process is successful. Process type is: install
(Nov 15, 2008 10:30:23 AM), Process, com.ibm.ws.install.ni.ismp.actions.SetExitCodeAction, msg1, CWUPI0000I: EXITCODE=0
(Nov 15, 2008 10:30:23 AM), Process, com.ibm.ws.install.ni.ismp.actions.ISMPLogSuccessMessageAction, msg1, INSTCONFSUCCESS

Start httpd as root:

[@Server1 ~]$ ps -ef | grep httpd
18709 10653  0 10:37 pts/1    00:00:00 grep httpd
[@Server1 ~]$ sudo /opt/IBM/HTTPServer/bin/apachectl -k start
[@Server1 ~]$ ps -ef | grep httpd
root     18713     1  3 10:37 ?        00:00:00 /opt/IBM/HTTPServer/bin/httpd -d /opt/IBM/HTTPServer -k start
nobody   18714 18713  0 10:37 ?        00:00:00 /opt/IBM/HTTPServer/bin/httpd -d /opt/IBM/HTTPServer -k start
nobody   18715 18713  0 10:37 ?        00:00:00 /opt/IBM/HTTPServer/bin/httpd -d /opt/IBM/HTTPServer -k start
nobody   18717 18713  0 10:37 ?        00:00:00 /opt/IBM/HTTPServer/bin/httpd -d /opt/IBM/HTTPServer -k start

Let’s see if http server is working for us . http://%Server_name%:80 should return

Install WAS as was61 :

cd /opt/IBM/source/websphere/WAS

vi responsefile.nd.txt

-OPT silentInstallLicenseAcceptance=”true”
-OPT allowNonRootSilentInstall=”true”
-OPT disableOSPrereqChecking=”true”
-OPT installType=”installNew”
-OPT feature=”noFeature”
-OPT installLocation=”/opt/IBM/WebSphere/AppServer”
-OPT profileType=”cell”
-OPT PROF_enableAdminSecurity=”true”
-OPT PROF_adminUserName=”Admin”
-OPT PROF_adminPassword=”admin”
-OPT PROF_dmgrProfileName=”dmgr”
-OPT PROF_appServerProfileName=”Node”
-OPT PROF_appServerNodeName=”Server1_Node01″
-OPT PROF_isDefault=”true”
-OPT PROF_hostName=”Server1″
-OPT PROF_nodeName=”Server1_Manager”
-OPT PROF_cellName=”Server1_Cell”
-OPT PROF_defaultPorts=”true”
-OPT PROF_validatePorts=”true”
-OPT PROF_webServerCheck=”true”
-OPT PROF_webServerType=”IHS”
-OPT PROF_webServerOS=”linux”
-OPT PROF_webServerName=”ihs-prpc”
-OPT PROF_webServerHostname=”Server1″
-OPT PROF_webServerPort=”80″
-OPT PROF_webServerInstallPath=”/opt/IBM/HTTPServer”
-OPT PROF_webServerPluginPath=”/opt/IBM/WebSphere/Plugins”
-OPT PROF_profilePath=”/opt/IBM/WebSphere/AppServer/profiles/Profile01″
-OPT PROF_serverName=”dmgr”
-OPT traceFormat=ALL
-OPT traceLevel=INFO

Edit to change hostnames, allow non-root silent installs

./install -options responsefile.nd.txt -silent

Be patient as it will take some time(15 mins in my case)  to start writing install logs.  I often use “df -k” to see how space is being used on /opt while install under progress.

cd /opt/IBM/WebSphere/AppServer/logs/manageprofiles

$ ls -lrt
-rw-rw-r– 1 was61 was61 1614293 Nov 11 11:49 dmgr_create.log
drwxrwxr-x 2 was61 was61    4096 Nov 11 11:49 dmgr
-rw-rw-r– 1 was61 was61 1302614 Nov 11 11:50 Node_create.log
drwxrwxr-x 2 was61 was61    4096 Nov 11 11:50 Node
-rw-rw-r– 1 was61 was61  421608 Nov 11 11:50 create.log

$vi Dmgr_create.log

<class>com.ibm.ws.profile.cli.WSProfileCLICreateProfileInvoker</class>
<method>executeWSProfileAccordingToMode</method>
<thread>10</thread>
<message>INSTCONFSUCCESS: Success: Profile dmgr now exists. Please consult /opt/IBM/WebSphere/AppServer/profiles/Profile01/dmgr/logs/AboutThisProfile.txt for more information about this profile.</message>
</record>

$vi Node_create.log

<logger>com.ibm.ws.install.configmanager.ConfigManager</logger>
<level>INFO</level>
<class>com.ibm.ws.install.configmanager.ConfigManager</class>
<method>launch</method>
<thread>10</thread>
<message>Returning with return code: INSTCONFSUCCESS</message>
</record>

Also have a look at create.log for detailed information.

Install UpdateInstaller

Install UpdateInstaller as user was61:

cd /opt/IBM/source/UpdateInstaller/UpdateInstaller

vi responsefile.updiinstaller.txt

OPT silentInstallLicenseAcceptance=”true”
-OPT allowNonRootSilentInstall=”true”
-OPT disableOSPrereqChecking=”true”
-OPT installLocation=”/opt/IBM/WebSphere/UpdateInstaller”
-OPT traceFormat=text
-OPT traceLevel=INFO

Edit to change hostnames, allow non-root silent installs more responsefile.updiinstaller.txt

./install -options responsefile.updiinstaller.txt -silent

check logs:

cd /opt/IBM/WebSphere/UpdateInstaller/logs/install

$vi log.txt and look for “INSTCONFSUCCESS”

(Nov 11, 2010 12:14:11 PM), Install, com.ibm.ws.install.ni.ismp.actions.SettleNIFRegistryAction, msg1, Current install/uninstall process is successful. Process type is: install
(Nov 11, 2010 12:14:11 PM), Install, com.ibm.ws.install.ni.ismp.actions.SetExitCodeAction, msg1, CWUPI0000I: EXITCODE=0
(Nov 11, 2010 12:14:11 PM), Install, com.ibm.ws.install.ni.ismp.actions.ISMPLogSuccessMessageAction, msg1, INSTCONFSUCCESS

Let’s apply a fix Patch to Websphere Install : Patch as user was61

vi /opt/IBM/source/UpdateInstaller/UpdateInstaller/install.txt

-OPT checkFilePermissions=”true”
-W maintenance.package=”/opt/IBM/source/6.1.0-WS-WAS-LinuxX64-FP0000021.pak”
-OPT disableNonBlockingPrereqChecking=”true”
-W product.location=”/opt/IBM/WebSphere/AppServer”
-W update.type=”install”

Make sure the update pak files are at defined directory

[install]$ ls -lrt /opt/IBM/source/6.1.0-WS-WAS-LinuxX64-FP0000021.pak
-r-xr-xr-x 1 was61 was61 429023884 Jan 16  2009 /opt/IBM/source/6.1.0-WS-WAS-LinuxX64-FP0000021.pak

Let’s run the update installer now

cd /opt/IBM/WebSphere/UpdateInstaller
./update.sh -options “/opt/IBM/source/UpdateInstaller/UpdateInstaller/install.txt” -silent

Check Install log:

cd /opt/IBM/WebSphere/AppServer/logs/update/6.1.0-WS-WAS-LinuxX64-FP0000021.install
vi updatelog.txt

Look for line: com.ibm.ws.install.ni.ismp.actions.ISMPLogSuccessMessageAction, msg1, INSTCONFSUCCESS

Running configuration command: updateLinuxCollector, percent complete: 100%
(Nov 11, 2010 12:34:54 PM), Install, com.ibm.ws.install.ni.ismp.actions.SettleNIFRegistryAction, msg1, Current install/uninstall process is successful. Process type is: maintenance
(Nov 11, 2010 12:34:54 PM), Install, com.ibm.ws.install.ni.ismp.actions.SetExitCodeAction, msg1, CWUPI0000I: EXITCODE=0
(Nov 11, 2010 12:34:54 PM), Install, com.ibm.ws.install.ni.ismp.actions.ISMPLogSuccessMessageAction, msg1, INSTCONFSUCCESS

Stop httpd, patch IHS

stop httpd as root:

sudo /opt/IBM/HTTPServer/bin/apachectl stop

vi /opt/IBM/source/UpdateInstaller/UpdateInstaller/install-fix-ihs.txt

-OPT checkFilePermissions=”true”
-W maintenance.package=”/opt/IBM/source/6.1.0-WS-IHS-LinuxX64-FP0000021.pak”
-OPT disableNonBlockingPrereqChecking=”true”
-W product.location=”/opt/IBM/HTTPServer”
-W update.type=”install”

[UpdateInstaller]$ ls -lrt /opt/IBM/source/6.1.0-WS-IHS-LinuxX64-FP0000021.pak
-rwxr-xr-x 1 was61 was61 12141372 Mar 19  2009 /opt/IBM/source/6.1.0-WS-IHS-LinuxX64-FP0000021.pak

cd /opt/IBM/WebSphere/UpdateInstaller
./update.sh -options “/opt/IBM/source/UpdateInstaller/UpdateInstaller/install-fix-ihs.txt” -silent

Check logs:

cd /opt/IBM/HTTPServer/logs/update/6.1.0-WS-IHS-LinuxX64-FP0000021.install

vi updatelog.txt & look for INSTCONFSUCCESS

, Current install/uninstall process is successful. Process type is: maintenance
(Nov 11, 2010 12:41:11 PM), Install, com.ibm.ws.install.ni.ismp.actions.SetExitCodeAction, msg1, CWUPI0000I: EXITCODE=0
(Nov 11, 2010 12:41:11 PM), Install, com.ibm.ws.install.ni.ismp.actions.ISMPLogSuccessMessageAction, msg1, INSTCONFSUCCESS

Start httpd, finishing patch IHS

As root user: sudo /opt/IBM/HTTPServer/bin/apachectl start

Update SDK As was61 user:

cd /opt/IBM/WebSphere/UpdateInstaller

vi /opt/IBM/source/UpdateInstaller/UpdateInstaller/install-fix-sdk.txt

-OPT checkFilePermissions=”true”
-W maintenance.package=”/opt/IBM/source/6.1.0-WS-WASSDK-LinuxX64-FP0000021.pak”
-OPT disableNonBlockingPrereqChecking=”true”
-W product.location=”/opt/IBM/WebSphere/AppServer”
-W update.type=”install”

[was61@]$ ls -lrt /opt/IBM/source/6.1.0-WS-WASSDK-LinuxX64-FP0000021.pak
-r-xr-xr-x 1 was61 was61 59745892 Jan 16  2009 /opt/IBM/source/6.1.0-WS-WASSDK-LinuxX64-FP0000021.pak

Run ./update.sh -options “/opt/IBM/source/UpdateInstaller/UpdateInstaller/install-fix-sdk.txt” -silent

Check Logs:

vi /opt/IBM/WebSphere/AppServer/logs/update/6.1.0-WS-WASSDK-LinuxX64-FP0000021.install/updatelog.txt & look for line: com.ibm.ws.install.ni.ismp.actions.ISMPLogSuccessMessageAction, msg1, INSTCONFSUCCESS

cd /opt/IBM/WebSphere/AppServer/logs/update/6.1.0-WS-WASSDK-LinuxX64-FP0000021.install

(Nov 11, 2010 12:46:25 PM), Install, com.ibm.ws.install.ni.ismp.actions.SetExitCodeAction, msg1, CWUPI0000I: EXITCODE=0
(Nov 11, 2010 12:46:25 PM), Install, com.ibm.ws.install.ni.ismp.actions.ISMPLogSuccessMessageAction, msg1, INSTCONFSUCCESS

Update plugins as user was61:

cd /opt/IBM/WebSphere/UpdateInstaller

vi “/opt/IBM/source/UpdateInstaller/UpdateInstaller/install-fix-plugin.txt”

-OPT checkFilePermissions=”true”
-W maintenance.package=”/opt/IBM/source/6.1.0-WS-PLG-LinuxX64-FP0000021.pak”
-OPT disableNonBlockingPrereqChecking=”true”
-W product.location=”/opt/IBM/HTTPServer/Plugins/”
-W update.type=”install”

./update.sh -options “/opt/IBM/source/UpdateInstaller/UpdateInstaller/install-fix-plugin.txt” -silent

Install GSKit

Check for GSKit installation:

rpm -qa|grep -i gsk
gsk7bas-7.0-4.14
gsk7bas64-7.0-4.14

if GSKit is not already installed, you will need to install GSKit as root

/opt/IBM/HTTPServer/GSKitImage/gskit.sh

Check versions

Check plugins versions as user was61:

cd /opt/IBM/HTTPServer/Plugins/bin
./versionInfo.sh

WVER0010I: Copyright (c) IBM Corporation 2002, 2005; All rights reserved. WVER0012I: VersionInfo reporter version 1.15.4.2, dated 6/5/08
IBM WebSphere Application Server Product Installation Status Report

Report at date and time 25 June 2009 16:10:22 BST
Installation
Product Directory /opt/IBM/HTTPServer/Plugins Version Directory /opt/IBM/HTTPServer/Plugins/properties/version DTD Directory /opt/IBM/HTTPServer/Plugins/properties/version/dtd Log Directory /opt/IBM/HTTPServer/Plugins/logs Backup Directory /opt/IBM/HTTPServer/Plugins/properties/version/nif/backup TMP Directory /tmp

Product List
PLG installed

Installed Product
Name Web server plug-ins for IBM WebSphere Application Server Version 6.1.0.21 ID PLG Build Level cf210844.13 Build Date 11/6/08

End Installation Status Report

Check App Server version as user was61:

cd /opt/IBM/WebSphere/AppServer/bin
./versionInfo.sh

WVER0010I: Copyright (c) IBM Corporation 2002, 2005; All rights reserved. WVER0012I: VersionInfo reporter version 1.15.4.2, dated 6/5/08
IBM WebSphere Application Server Product Installation Status Report
Report at date and time 25 June 2009 16:12:29 BST

Installation
Product Directory /opt/IBM/WebSphere/AppServer Version Directory /opt/IBM/WebSphere/AppServer/properties/version DTD Directory /opt/IBM/WebSphere/AppServer/properties/version/dtd Log Directory /opt/IBM/WebSphere/AppServer/logs Backup Directory /opt/IBM/WebSphere/AppServer/properties/version/nif/backup TMP Directory /tmp

Product List
ND installed

Installed Product
Name IBM WebSphere Application Server – ND Version 6.1.0.21 ID ND Build Level cf210844.13 Build Date 11/6/08

End Installation Status Report

Check java version as user was61:

cd /opt/IBM/WebSphere/AppServer/java/bin
./java -version

[was61@bin]$ cd /opt/IBM/WebSphere/AppServer/java/bin
[was61@bin]$ ./java -version
java version “1.5.0”
Java(TM) 2 Runtime Environment, Standard Edition (build pxa64devifx-20080908 (SR8a + IZ29767 + IZ30684 + IZ31214 + IZ31213))
IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64 j9vmxa6423ifx-20080811 (JIT enabled)
J9VM – 20080809_21892_LHdSMr
JIT – 20080620_1845_r8
GC – 200806_19)
JCL – 20080908a

At end of this install you should able to start the DMGR process

IBM’s JVM is written to meet the specs as documented by Sun.  IBM runs the test suit against their JVM to make sure that what should occur does occur.  They (IBM) feel that their code performs the requested functions more efficiently and thus performs better while still providing the same results. IBM may throw one error exception while Sun’s may throw another.

A good discussion on IBM & SUN JVM is at http://forums.anandtech.com/archive/index.php/t-1958785.html

Can I pick up an EAR/WAR file from websphere and install at compatible J2EE application server?
“If” the EAR is compatible, then you can deploy it either on a WebSphere Application Server, or some other J2EE application server.  However, that is a big “if.” Does the application make use of any WebSphere specific settings, or resources?  That might be hard to tell if the application isn’t thoroughly documented.

Having said that JVMs are going to be different. On source it is IBM JVM while at destination (Tomcat/Glassfish) it is SUN JVM.
The WebSphere Application Server environment adds a number of class files to allow applications to access specific resources and capabilities of the WebSphere environment.  This is true even for the Solaris Operating System where the JVM used by WebSphere is a Sun JVM. In addition to these “extra” jar files, are there differences between the JVM’s.

A specific J2EE application server may have its own unique classes that do not exist in other J2EE environments.  So if you developed an application under “J2EE ServerA” and used specific functions/classes that are unique to “J2EE ServerA”, then you can’t just deploy to “J2EE ServerB”.

Are there any utilitiees to test portability of a J2EE application?
The idea behind having a J2EE compliant implementation is so that applications may be easily moved from one environment to another.  So, as long as both the source and target application server environments are J2EE compliant, this “should” be possible.  However, again we are forced to revisit the questions:

But there must be something from these vendors looking for portability of application to grab customers on their platform.

https://migrate2glassfish.dev.java.net/
http://blogs.sun.com/theaquarium/entry/websphere_to_glassfish_migration_example
http://blogs.sun.com/alexismp/entry/glassfish_equivalent_to_websphere_s

IBM do provide a utility to help people migrate their utilities to WebSphere:
http://www.IBM.com/developerworks/websphere/downloads/migration_toolkit.html

Demo is available here

I have One Primary database and 2 Physical Standby Databases under RHEL 5. Here is a process to patch them with JULY CPU update “Critical Patch Update Release 10.2.0.4 for UNIX Released July 13, 2010”

At Primary Site
SQL> show parameter dg_broker_start

NAME                                 TYPE        VALUE
———————————— ———– ——————————
dg_broker_start                      boolean     TRUE

SQL> alter system switch logfile;
System altered.

Make sure the Logs are shipped and both Standby Database and applied. Following SQLs could be used to locate the log sequences on all databases.

Locate Primary Log Sequence :   Select Archived_Seq# From V$Archive_Dest_Status Where Dest_Id=1;

Locate Standby Log Sequence :  Select Archived_Seq#, Applied_Seq# From V$Archive_Dest_Status Where Dest_Id=2;

Locate DR Log Sequence :   Select Archived_Seq#, Applied_Seq# From V$Archive_Dest_Status Where Dest_Id=3;

Find out existing Patch details using following queries :

SELECT comments, action_time, id “PATCH_NUMBER”, version FROM sys.registry$history WHERE action = ‘CPU’;

COMMENTS	ACTION_TIME	PATCH_NUMBER	VERSION
view recompilation	42:39.2	6452863
view recompilation	59:20.3	6452863
view recompilation	23:58.7	6452863
view recompilation	56:19.9	6452863

SELECT comments, action_time, id PATCH_NUMBER, version FROM sys.registry$history

COMMENTS	ACTION_TIME	PATCH_NUMBER	VERSION
Upgraded from 10.2.0.1.0	40:28.8		10.2.0.4.0
CPUApr2009	46:06.0	4	10.2.0.4
view recompilation	42:39.2	6452863
CPUOct2009	56:35.7	6	10.2.0.4
view recompilation	59:20.3	6452863
CPUJan2010	01:47.4	6	10.2.0.4
view recompilation	23:58.7	6452863

Backup Primary database  & then shutdown Primary Database

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

Take TAR backup of ORACLE_HOME

$tar -czvf ORACLE_BKUP.tgz $ORACLE_HOME

At Standby & DR Site
[oracle@Standby bdump]$ sqlplus /nolog

SQL*Plus: Release 10.2.0.4.0 – Production on Mon Mar 29 18:26:46 2010

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

SQL> conn / as sysdba
Connected.
SQL> select NAME,open_mode,GUARD_STATUS,DATABASE_ROLE from v$database;

NAME      OPEN_MODE  GUARD_S DATABASE_ROLE
——— ———- ——- —————-
ROD  MOUNTED    NONE    PHYSICAL STANDBY

SQL> shutdown immediate;
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down. (Took 5 mins + to shutdown)

Take TAR backup of Standby ORACLE_HOME

$tar -czvf ORACLE_BKUP.tgz $ORACLE_HOME

[oracle@ ~]$ lsnrctl stop LISTENER
[oracle@ ~]$ ps -ef | grep oracle
oracle    3861  3802  0 18:45 pts/0    00:00:00 -bash
oracle    7309  7287  0 20:30 pts/1    00:00:01 -bash
oracle    8907  7309  0 21:19 pts/1    00:00:00 ps -ef
oracle    8908  7309  0 21:19 pts/1    00:00:00 grep oracle

cd $ORACLE_HOME/patches
[oracle@patches]$ ls -lrt
total 12
drwxr-xr-x  5 oracle oracle 4096 Feb 15  2010 9119284
drwxr-xr-x 37 oracle oracle 4096 Feb 15  2010 9119226
drwxr-xr-x 42 oracle oracle 4096 Jul  6 16:53 9655017

[oracle@patches]$ cd 9655017

[oracle@patches]$ $ORACLE_HOME/OPatch/opatch napply -skip_subset -skip_duplicate

Stop here go back to Primary now

oracle@ ~]$ lsnrctl stop LISTENER
[oracle@ ~]$ ps -ef | grep oracle
oracle    3861  3802  0 18:45 pts/0    00:00:00 -bash
oracle    7309  7287  0 20:30 pts/1    00:00:01 -bash
oracle    8907  7309  0 21:19 pts/1    00:00:00 ps -ef
oracle    8908  7309  0 21:19 pts/1    00:00:00 grep oracle

cd $ORACLE_HOME/patches
[oracle@patches]$ ls -lrt
total 12
drwxr-xr-x  5 oracle oracle 4096 Feb 15  2010 9119284
drwxr-xr-x 37 oracle oracle 4096 Feb 15  2010 9119226
drwxr-xr-x 42 oracle oracle 4096 Jul  6 16:53 9655017

[oracle@patches]$ cd 9655017

[oracle@patches]$ $ORACLE_HOME/OPatch/opatch napply -skip_subset -skip_duplicate

For each database instance running on the Oracle home being patched, connect to the database using SQL*Plus.

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle.sql cpu apply
SQL> — Execute the next statement only if this is the first 10.2.0.4 CPU applied in the Oracle home.
SQL> @utlrp.sql
SQL> QUIT

Run the view recompilation script. Note that this script is run with the database in upgrade mode, which restricts connections as SYSDBA.

cd $ORACLE_HOME/cpu/view_recompile
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> SHUTDOWN IMMEDIATE
SQL> STARTUP UPGRADE
SQL> @view_recompile_jan2008cpu.sql
SQL> SHUTDOWN;
SQL> STARTUP;
SQL> QUIT

Startup Primary Listener

At Standby & DR Site

SQL> startup nomount;
ORACLE instance started.
Total System Global Area 2147483648 bytes
Fixed Size 2030296 bytes
Variable Size 503317800 bytes
Database Buffers 1627389952 bytes
Redo Buffers 14745600 bytes

SQL> show parameter dg_broker_start
NAME TYPE VALUE
———————————— ———– ——————————
dg_broker_start boolean TRUE

SQL> alter database mount standby database;
Database altered.

SQL> recover managed standby database disconnect from session;
Media recovery complete.

Startup Listeners and monitor alrtlogs at Stanby servers for Log Recovery

Redo Shipping Client Connected as PUBLIC
— Connected User is Valid
RFS[3]: Assigned to RFS process 7804
RFS[3]: Identified database type as ‘physical standby’
RFS[3]: Archived Log: ‘/u06/oradata/prod/arch/1_65479_684936861.arc’
Wed Sep 29 22:08:08 2010
Media Recovery Log /u06/oradata/prod/arch/1_65469_684936861.arc
Wed Sep 29 22:08:08 2010
RFS[1]: Archived Log: ‘/u06/oradata/prod/arch/1_65477_684936861.arc’
Wed Sep 29 22:08:09 2010
Media Recovery Log /u06/oradata/prod/arch/1_65470_684936861.arc
Media Recovery Log /u06/oradata/prod/arch/1_65471_684936861.arc
Media Recovery Log /u06/oradata/prod/arch/1_65472_684936861.arc
Media Recovery Log /u06/oradata/prod/arch/1_65473_684936861.arc
Media Recovery Log /u06/oradata/prod/arch/1_65474_684936861.arc
Media Recovery Log /u06/oradata/prod/arch/1_65475_684936861.arc
Media Recovery Log /u06/oradata/prod/arch/1_65476_684936861.arc

Finally run following commands to make sure logs are appied at standbydatabases:

Locate Primary Log Sequence :   Select Archived_Seq# From V$Archive_Dest_Status Where Dest_Id=1;

Locate Standby Log Sequence :  Select Archived_Seq#, Applied_Seq# From V$Archive_Dest_Status Where Dest_Id=2;

Locate DR Log Sequence :   Select Archived_Seq#, Applied_Seq# From V$Archive_Dest_Status Where Dest_Id=3;

Sample log files attached  opatch_history-1.txt & opatch.log

[Dmgr]$ /opt/IBM/WebSphere/AppServer/profiles/Profile/Dmgr/bin/startManager.sh

ADMU0116I: Tool information is being logged in file
/opt/IBM/WebSphere/AppServer/profiles/Profile/Dmgr/logs/dmgr/startServer.log
ADMU0128I: Starting tool with the Dmgr profile
ADMU3100I: Reading configuration for server: dmgr
ADMU0111E: Program exiting with error: java.io.FileNotFoundException:
/opt/IBM/WebSphere/AppServer/profiles/Profile/Dmgr/config/cells/Cell/nodes/Manager/servers/dmgr/server.xml
(No such file or directory)
ADMU1211I: To obtain a full trace of the failure, use the -trace option.
ADMU0211I: Error details may be seen in the file:
/opt/IBM/WebSphere/AppServer/profiles/Profile/Dmgr/logs/dmgr/startServer.log

Log /opt/IBM/WebSphere/AppServer/profiles/Profile/Dmgr/logs/dmgr/startServer.log has tonnes of errors

[10/12/10 7:14:19:045 BST] 0000000a AdminTool     A   ADMU0111E: Program exiting with error: java.io.FileNotFoundException:
/opt/IBM/WebSphere/AppServer/profiles/Profile/Dmgr/config/cells/Cell/nodes/Manager/servers/dmgr/server.xml (No such file or directory)
at java.io.FileInputStream.<init>(FileInputStream.java:135)
at org.eclipse.emf.ecore.resource.impl.URIConverterImpl.createFileInputStream(URIConverterImpl.java:469)
at org.eclipse.emf.ecore.resource.impl.URIConverterImpl.createInputStream(URIConverterImpl.java:431)
at org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(ResourceImpl.java:897)
at org.eclipse.wst.common.internal.emf.resource.CompatibilityXMIResourceImpl.load(CompatibilityXMIResourceImpl.java:259)
at com.ibm.websphere.resource.WASResourceImpl.load(WASResourceImpl.java:61)
at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoad(ResourceSetImpl.java:249)
at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoadHelper(ResourceSetImpl.java:264)
at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResource(ResourceSetImpl.java:390)
at com.ibm.websphere.resource.WASResourceSetImpl.getResource(WASResourceSetImpl.java:158)
at com.ibm.websphere.resource.WASResourceSetImpl.getResource(WASResourceSetImpl.java:143)
at com.ibm.ws.runtime.service.ConfigRootImpl.getResource(ConfigRootImpl.java:326)
at com.ibm.ws.runtime.service.ConfigRootImpl.getResource(ConfigRootImpl.java:299)
at com.ibm.ws.runtime.service.ConfigRootImpl.getResource(ConfigRootImpl.java:275)
at com.ibm.ws.management.tools.WsServerLauncher.initializeRepositoryAndLauncher(WsServerLauncher.java:397)
at com.ibm.ws.management.tools.WsServerLauncher.runTool(WsServerLauncher.java:265)
at com.ibm.ws.management.tools.AdminTool.executeUtility(AdminTool.java:264)
at com.ibm.ws.management.tools.WsServerLauncher.main(WsServerLauncher.java:120)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at com.ibm.ws.bootstrap.WSLauncher.main(WSLauncher.java:263)

System doesn’t have  “/opt/IBM/WebSphere/AppServer/profiles/Profile/Dmgr/config/cells/Cell/nodes/Manager/servers/dmgr/server.xml” but have
“/opt/IBM/WebSphere/AppServer/profiles/Profile/Dmgr/config/cells/Cell/nodes/Manager/servers/Dmgr/server.xml”

StartServer Dmgr command works fine. Note it’s “Dmgr” instead of “dmgr” here

$ /opt/IBM/WebSphere/AppServer/profiles/Profile/Dmgr/bin/startServer.sh Dmgr
ADMU0116I: Tool information is being logged in file
/opt/IBM/WebSphere/AppServer/profiles/Profile/Dmgr/logs/Dmgr/startServer.log
ADMU0128I: Starting tool with the Dmgr profile
ADMU3100I: Reading configuration for server: Dmgr
ADMU3200I: Server launched. Waiting for initialization status.
ADMU3000I: Server Dmgr open for e-business; process id is 2264

$pwd
/opt/IBM/WebSphere/AppServer/profiles/Profile

[Profile]$ ls -lrt
drwxr-xr-x 17 was61 web  12288 Sep 27 08:28 Node
drwxr-xr-x 18 was61 web   4096 Jul  4  2011 Dmgr

Solution :

Edit “/opt/IBM/WebSphere/AppServer/bin/startManager.sh” and alter D_DMGR from dmgr to Dmgr

#Common args…
D_DMGR=”dmgr” changed to “Dmgr”
D_ARGS=”-Dws.ext.dirs=”$WAS_EXT_DIRS” $DELIM -Dwas.install.root=”$WAS_HOME” $DELIM -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager $DELIM -Djava.util.logging.configureByServer=true”

Edit “/opt/IBM/WebSphere/AppServer/bin/stopManager.sh” and alter line 126

$SHELL “$CONFIG_ROOT” “$WAS_CELL” “$WAS_NODE” dmgr -dmgr “$@”
to
$SHELL “$CONFIG_ROOT” “$WAS_CELL” “$WAS_NODE” Dmgr -dmgr “$@”

The isAppReady command returns a value of true if the system is ready to start the application, or a value of false if the system is not ready to start the application, as the following examples display:

[was61@waslogs]$ /opt/IBM/WebSphere/AppServer/profiles/Profile01/dmgr/bin/wsadmin.sh -conntype SOAP -lang jython
WASX7209I: Connected to process “dmgr” on node Prod_Node_Manager using SOAP connector;  The type of process is: DeploymentManagerWASX7031I: For help, enter: “print Help.help()”
wsadmin>AdminApp.isAppReady(‘application1’);

ADMA5071I: Distribution status check started for application prpc_j2ee14.
WebSphere:cell=Prod_Node_Cell,node=Prod_Node_Node01,distribution=true,expansion=notprocessing
ADMA5011I: The cleanup of the temp directory for application prpc_j2ee14 is complete.
ADMA5072I: Distribution status check completed for application prpc_j2ee14.
‘true’

getDeployStatus command to display additional information about the binary file expansion status, as the following examples display:

[was61@Prod_Node waslogs]$ /opt/IBM/WebSphere/AppServer/profiles/Profile01/dmgr/bin/wsadmin.sh -conntype SOAP -lang jython
WASX7209I: Connected to process “dmgr” on node Prod_Node_Manager using SOAP connector;  The type of process is: DeploymentManager
wsadmin>AdminApp.getDeployStatus(‘application1’);
ADMA5071I: Distribution status check started for application application.
WebSphere:cell=Prod_Node_Cell,node=Prod_Node_Node01,distribution=true,expansion=notprocessing
ADMA5011I: The cleanup of the temp directory for application application is complete.
ADMA5072I: Distribution status check completed for application application.
‘WebSphere:cell=Prod_Node_Cell,node=Prod_Node_Node01,distribution=true,expansion=notprocessing’

Let’s deploy an Application called “application1″

1. wsadmin>AdminApp.update(‘application1’, ‘app’, ‘[-operation update -contents /tmp/Application1.ear -usedefaultbindings -nodeployejb ]’);

ADMA0073W: Custom permissions are found in the [(com.ibm.websphere.security.WebSphereRuntimePermission AdminPermission), (com.ibm.oti.shared.SharedClassPermission

// One cannot grant permissions to code outside of the deployed application via the WAS policy
ADMA5017I: Uninstallation of Application1 started.
ADMA5104I: The server index entry for WebSphere:cell=server_Cell,node=server_Node01 is updated successfully.
ADMA5106I: Application Application1 uninstalled successfully.
ADMA5016I: Installation of Application1 started.
ADMA5001I: The application binaries are saved in /opt/IBM/WebSphere/AppServer/profiles/Profile01/dmgr/wstemp/Script12b3ee78983/workspace/cells/server_Cell/applications/Application1.ear/Application1.ear
ADMA5005I: The application Application1 is configured in the WebSphere Application Server repository.
SECJ0400I: Successfuly updated the application Application1 with the appContextIDForSecurity information.
ADMA5011I: The cleanup of the temp directory for application Application1 is complete.
ADMA5013I: Application Application1 installed successfully.

2. wsadmin>AdminConfig.save();

Currently I have number of websphere 6.1 instances under RHEL 5.3 but not a tool like hyperic, wily introscope to monitor them actively. I started using wsadmin scripts to pull out stats thru wsadmin.

I am basically trying to get same details thru wsadmin which I could thru NAGIOS plugin. Here is a sample output from NAGIOS.

Monitor JVM heapSize :
JVM heapsize is provided for the entire server. It is measured as: percent used/maximum configured
To Monitor, check_was -s heapsize -w 80 -c 90 -p <server name=””>
[was61@Server1 check_was-0.3]$ ./check_was -s heapsize -w 80 -c 90 -p server_member1
OK – heapsize: 1048576/2097152 (50.0%)|heapsize=50.0%;80;90;

MonitorLiveSessions :
Live session usage can be monitored for the entire server (all hosts) or with a named host. It is measured as: Number of live sessions

[was61@Server1 check_was-0.3]$ ./check_was -s sessions -w 200 -c 400 -p server_member1
OK – live sessions: total 0, default_hostCTI 0, default_hostprsysmgmt 0, default_hostprweb 0, default_hostprdbutil 0|total=0.0;200;400; default_hostcti=0.0;200;400; default_hostprsysmgmt=0.0;200;400; default_hostprweb=0.0;200;400; default_hostprdbutil=0.0;200;400;

MonitorJdbcConnectionPools:
JDBC connection pool usage can be monitored for the entire server (all connection pools) or with a named connection pool. It is measured as: percent used/maximum configured

[was61@Server1 check_was-0.3]$ ./check_was -s connectionpool -w 80 -c 90 -p server_member
OK – connection pool size: Oracle JDBC Driver 5/100 (5.0%)|oraclejdbcdriver=5.0%;80;90;

MonitorThreadPools :
Thread pool usage can be monitored for the entire server (all thread pools) or with a named thread pool. It is measured as: percent used/maximum configured

[was61@Server1 check_was-0.3]$ ./check_was -s threadpool -w 80 -c 90 -p server_member1
CRITICAL – thread pool size: WebContainer 4/100 (4.0%), SoapConnectorThreadPool 3/5 (60.0%), SIBFAPInboundThreadPool 0/50 (0.0%), HAManager.thread.pool 2/2 (100.0%), MessageListenerThreadPool 0/50 (0.0%), ORB.thread.pool 0/50 (0.0%), SIBFAPThreadPool 2/50 (4.0%), ProcessDiscovery 1/2 (50.0%), TCPChannel.DCS 3/20 (15.0%)|webcontainer=4.0%;80;90; soapconnectorthreadpool=60.0%;80;90; sibfapinboundthreadpool=0.0%;80;90; hamanager_thread_pool=100.0%;80;90; messagelistenerthreadpool=0.0%;80;90; orb_thread_pool=0.0%;80;90; sibfapthreadpool=4.0%;80;90; processdiscovery=50.0%;80;90; tcpchannel_dcs=15.0%;80;90;</server>

Purpose: wsadmin script used to display user specified information about
         WebSphere Application Server resources.\n
Version: %(version)s
Updated: %(updated)s\n
  Usage: %(cmdName)s [options]\n
Required switches/options:
  -i | --info       <value> = Type of information being requested
  -s | --serverName <value> = Target serverName\n
Optional switches/options:
  -n | --nodeName   <value> = Target nodeName\n
Information types/values:
  connectionpool - Display Connection Pool details
  heapsize       - Display Heap Size details
  threadpool     - Display Thread Pool details
  sessions       - Display Session Details\n
Notes:
- Long form option values may be separated/delimited from their associated
  identifier using either a space, or an equal sign ('=').\n
- Short form option values may be sepearated from their associated letter
  using an optional space.\n
- Text containing blanks must be enclosed in double quotes.\n
Examples:
  wsadmin -f %(cmdName)s.py -i heapsize -s server1 -n node01\n'''

WASX7209I: Connected to process “dmgr” on node Server1_Manager using SOAP connector;  The type of process is: DeploymentManager
WASX7303I: The following options are passed to the scripting environment and are available as arguments that are stored in the argv variable: “[-i, heapsize, -s, Server1server_member2]”
heapsize: node=Server1_Node01  server=Server1server_member2  used=1027.2 MB (65.69%)  free=536.5 MB

WASX7209I: Connected to process “dmgr” on node Server1_Manager using SOAP connector;  The type of process is: DeploymentManager
WASX7303I: The following options are passed to the scripting environment and are available as arguments that are stored in the argv variable: “[-i, threadpool, -s, server_member2]”
|             Low    High
|           Water  Water  Lower  Upper  Current
Name | integral  Mark   Mark   Bound  Bound  Usage    Ratio
————————+—————————————————–
Message.Listener.Pool |      0.0   10     10     10     50      0      0.00%
ORB.thread.pool |      0.0   10     10     10     50      0      0.00%
SIBFAPInboundThreadPool | 9090637.0    1      4      4     50      3     75.00%
SIBFAPThreadPool | 112975.0    1      4      4     50      2     50.00%
TCPChannel.DCS | 255726.0    1      5      5     20      3     60.00%
WebContainer | 12011409.0    1     10     10     50      7     70.00%

So here  “WebContainer | 12011409.0   1  10  10  50   7  70.00%” means WebContainer currently using 7 processes out of 10 allocated and they can expand upto 50 MAX.

p=$(lsof -t -i :1159); echo $p $(tr ‘\0’ ‘\n’ < /proc/$p/cmdline | tail -1)
4066 /u01/app/oracle/product/11.2.0/dbhome_1/oc4j/j2ee/OC4J_DBConsole_Server1.oracledbasupport.co.uk_TEST/config/server.xml

p=$(lsof -t -i :1160); echo $p $(tr ‘\0’ ‘\n’ < /proc/$p/cmdline | tail -1)
8121 /u01/app/oracle/product/11.2.0/dbhome_1/oc4j/j2ee/OC4J_DBConsole_Server1.oracledbasupport.co.uk_Dev/config/server.xml

Please use following process when you have PerfServelet (PervServletApp application ) generated offline XML files.

Objective : I have PervServletApp application to retrieve websphere performance matrix.  The data returned is in XML format (Sample.xml) and I needed something to analyse that RAW data.

While surfing I stumbled at wasstats script https://code.google.com/p/imakerobots/wiki/wasstats

For windows WebSphere Statistic (wsstat) version please visit & scroll down to “Figure 4. The WebSphere Statistic tool” . Download windows batch script

“Currently, this script only processes JVM Runtime, Session Manager, JDBC Connection Pool and Thread Pool statistics ”

I created a sample.xml file using

“http://%Server_Name%:9080/wasPerfTool/servlet/perfservlet?node=Server1_Node01&server=server_member1&module=connectionPoolModule+jvmRuntimeModule+servletSessionsModule+systemModule+threadPoolModule”

Sample.xml XML file used for Analysis

When ran againt RHEL- WAS 6.1 ND with wastats.py I came across some errors :

[was61@Server1 permon]$ /usr/bin/python wasstats.py sample.xml
Processing: sample.xml

INFO|server_member1:Server1_Node01:SuperSnoop#SuperSnoopWeb.war:RangeStatistic:0:0.0:1283788583899:0:0.0:LiveCount:1283528346720:N/A:0
INFO|server_member1:Server1_Node01:perfServletApp#perfServletApp.war:RangeStatistic:0:0.0:1283788583899:0:0.0:LiveCount:1283528346768:N/A:0
INFO|server_member1:Server1_Node01:prsysmgmt_war#prsysmgmt.war:RangeStatistic:0:0.0:1283788583899:0:0.0:LiveCount:1283528346376:N/A:0
CHECK|OK:HeapSize:Server1_Node01:server_member1:JVM Runtime:1048576/2097152:(50%)|1048576:0.0:1283788583896:1048576:1048576:0.0:1283528313727:2097152:1048576:KILOBYTE
INFO|server_member1:Server1_Node01:JVM Runtime:BoundedRangeStatistic:1048576:0.0:1283788583896:1048576:1048576:0.0:HeapSize:1283528313727:KILOBYTE:2097152:1048576
INFO|server_member1:Server1_Node01:JVM Runtime:CountStatistic:688987:1283788583896:UsedMemory:1283528313726:KILOBYTE
INFO|server_member1:Server1_Node01:JVM Runtime:CountStatistic:260270:1283788583896:UpTime:1283528313726:SECOND
INFO|server_member1:Server1_Node01:JVM Runtime:CountStatistic:0:1283788583899:ProcessCpuUsage:1283528313723:N/A
Traceback (most recent call last):
File “wasstats.py”, line 600, in ?
analyze( xmldoc );
File “wasstats.py”, line 202, in analyze
name = pool.getAttribute( ‘name’ );
UnboundLocalError: local variable ‘pool’ referenced before assignment

Here is a revised wasstats.py script tested on WAS 6.1  ND deployment on RHEL 5.5

[was61@Server1 permon]$ /usr/bin/python wasstats.py sample.xml
Processing: sample.xml
INFO|server_member1:Server1_Node01:SuperSnoop#SuperSnoopWeb.war:RangeStatistic:0:0.0:1283788583899:0:0.0:LiveCount:1283528346720:N/A:0
INFO|server_member1:Server1_Node01:perfServletApp#perfServletApp.war:RangeStatistic:0:0.0:1283788583899:0:0.0:LiveCount:1283528346768:N/A:0
CHECK|OK:HeapSize:Server1_Node01:server_member1:JVM Runtime:1048576/2097152:(50%)|1048576:0.0:1283788583896:1048576:1048576:0.0:1283528313727:2097152:1048576:KILOBYTE
INFO|server_member1:Server1_Node01:JVM Runtime:BoundedRangeStatistic:1048576:0.0:1283788583896:1048576:1048576:0.0:HeapSize:1283528313727:KILOBYTE:2097152:1048576
INFO|server_member1:Server1_Node01:JVM Runtime:CountStatistic:688987:1283788583896:UsedMemory:1283528313726:KILOBYTE
INFO|server_member1:Server1_Node01:JVM Runtime:CountStatistic:260270:1283788583896:UpTime:1283528313726:SECOND
INFO|server_member1:Server1_Node01:JVM Runtime:CountStatistic:0:1283788583899:ProcessCpuUsage:1283528313723:N/A
CHECK|OK:PoolSize:Server1_Node01:server_member1:Default:5/100:(5%)|10:1.303999271E9:1283788583899:1:5:5.010175044932727:1283528313697:100:5:N/A
INFO|server_member1:Server1_Node01:Default:BoundedRangeStatistic:10:1.303999271E9:1283788583899:1:5:5.010175044932727:PoolSize:1283528313697:N/A:100:5
CHECK|CRITICAL:PoolSize:Server1_Node01:server_member1:HAManager.thread.pool:2/2:(100%)|2:5.2047867E8:1283788583899:1:2:1.9997628080374719:1283528313697:2:2:N/A
INFO|server_member1:Server1_Node01:HAManager.thread.pool:BoundedRangeStatistic:2:5.2047867E8:1283788583899:1:2:1.9997628080374719:PoolSize:1283528313697:N/A:2:2
CHECK|OK:PoolSize:Server1_Node01:server_member1:Message Listener:0/50:(0%)|10:0.0:1283788583899:10:10:0.0:1283528313697:50:0:N/A
INFO|server_member1:Server1_Node01:Message Listener:BoundedRangeStatistic:10:0.0:1283788583899:10:10:0.0:PoolSize:1283528313697:N/A:50:0
CHECK|OK:PoolSize:Server1_Node01:server_member1:Object Request Broker:0/50:(0%)|10:0.0:1283788583899:10:10:0.0:1283528313697:50:0:N/A
INFO|server_member1:Server1_Node01:Object Request Broker:BoundedRangeStatistic:10:0.0:1283788583899:10:10:0.0:PoolSize:1283528313697:N/A:50:0
CHECK|OK:PoolSize:Server1_Node01:server_member1:ProcessDiscovery:1/2:(50%)|1:2.5928954E8:1283788583899:1:1:0.9966172837551133:1283528414279:2:1:N/A
INFO|server_member1:Server1_Node01:ProcessDiscovery:BoundedRangeStatistic:1:2.5928954E8:1283788583899:1:1:0.9966172837551133:PoolSize:1283528414279:N/A:2:1
CHECK|OK:PoolSize:Server1_Node01:server_member1:SIBFAPInboundThreadPool:2/50:(4%)|4:5.20254968E8:1283788583899:1:4:1.9989522093993581:1283528320064:50:2:N/A
INFO|server_member1:Server1_Node01:SIBFAPInboundThreadPool:BoundedRangeStatistic:4:5.20254968E8:1283788583899:1:4:1.9989522093993581:PoolSize:1283528320064:N/A:50:2
CHECK|OK:PoolSize:Server1_Node01:server_member1:SIBFAPThreadPool:3/50:(6%)|4:7.60451275E8:1283788583899:1:4:2.9233778697579353:1283528456297:50:3:N/A
INFO|server_member1:Server1_Node01:SIBFAPThreadPool:BoundedRangeStatistic:4:7.60451275E8:1283788583899:1:4:2.9233778697579353:PoolSize:1283528456297:N/A:50:3
CHECK|OK:PoolSize:Server1_Node01:server_member1:SoapConnectorThreadPool:2/5:(40%)|3:5.20338761E8:1283788583899:1:3:1.999267779975942:1283528319233:5:2:N/A
INFO|server_member1:Server1_Node01:SoapConnectorThreadPool:BoundedRangeStatistic:3:5.20338761E8:1283788583899:1:3:1.999267779975942:PoolSize:1283528319233:N/A:5:2
CHECK|OK:PoolSize:Server1_Node01:server_member1:TCPChannel.DCS:4/20:(20%)|5:1.041063168E9:1283788583899:1:5:3.999951342641484:1283528314941:20:4:N/A
INFO|server_member1:Server1_Node01:TCPChannel.DCS:BoundedRangeStatistic:5:1.041063168E9:1283788583899:1:5:3.999951342641484:PoolSize:1283528314941:N/A:20:4
CHECK|OK:PoolSize:Server1_Node01:server_member1:WebContainer:8/100:(8%)|10:1.832855037E9:1283788583899:1:10:7.044842168064254:1283528414119:100:8:N/A
INFO|server_member1:Server1_Node01:WebContainer:BoundedRangeStatistic:10:1.832855037E9:1283788583899:1:10:7.044842168064254:PoolSize:1283528414119:N/A:100:8
No statistics found: “DB2 Universal JDBC Driver Provider”
No statistics found: “WebSphere embedded ConnectJDBC driver for MS SQL Server”

I will list here steps carried out to patch websphere 6.1. ND deployment (vertical cluster with 2 JVMs ) from 6.1.0.21 to 6.1.0.29

[spatil@~]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.5 (Tikanga)

[spatil@~]$ uname -a
Linux server1.domain.com 2.6.18-194.8.1.el5 #1 SMP Wed Jun 23 10:52:51 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

1. Locate current version of websphere using versioninfo.sh
2. Download upgrade patches certified for above release
3. Shutdown IHS,Websphere Services
4. If needed download & install “IBM Update Installer”
5. Use new installer to apply fix pack to IHS,WAS & PLUG-IN
6. Startup IHS,Websphere services
7. Locate version of websphere using versioninfo.sh, it should show the new release number

While installing AppServer components, you may see that installer freezes at 76% .  To see progress of an installer monitor “Install log”

[was61@Server1 tmp]$ pwd
/opt/IBM/WebSphere/UpdateInstaller/logs/tmp
[was61@Server1 tmp]$ tail -f updatelog.txt
(Aug 21, 2011 10:38:13 AM), Install, com.ibm.ws.install.ni.ismp.actions.InstallNIFMaintenance, msg1, Installing component: startupbean, percent complete: 75%
(Aug 21, 2011 10:38:15 AM), Install, com.ibm.ws.install.ni.ismp.actions.InstallNIFMaintenance, msg1, Installing component: transaction.impl, percent complete: 76%
(Aug 21, 2011 10:38:19 AM), Install, com.ibm.ws.install.ni.ismp.actions.InstallNIFMaintenance, msg1, Installing component: uddi.bundle, percent complete: 76%

1. Locate current version of Websphere components using $PROFILE/Node/bin/versioninfo.sh , $HTTP_HOME/bin/versioninfo.sh & $HTTP_HOME/Plugins/bin/versioninfo.sh command.

[was61@Server1 bin]$ ./versionInfo.sh
WVER0010I: Copyright (c) IBM Corporation 2002, 2005; All rights reserved.
WVER0012I: VersionInfo reporter version 1.15.4.2, dated 6/5/08
———————————————————————
IBM WebSphere Application Server Product Installation Status Report
———————————————————————
Report at date and time March 31, 2010 8:29:16 PM GMT
Installation
———————————————————————
Product Directory        /opt/IBM/WebSphere/AppServer
Version Directory        /opt/IBM/WebSphere/AppServer/properties/version
DTD Directory            /opt/IBM/WebSphere/AppServer/properties/version/dtd
Log Directory            /opt/IBM/WebSphere/AppServer/logs
Backup Directory         /opt/IBM/WebSphere/AppServer/properties/version/nif/backup
TMP Directory            /tmp
Product List
———————————————————————
ND                       installed
Installed Product
———————————————————————
Name IBM WebSphere Application Server – ND
Version                  6.1.0.21
ID                       ND
Build Level              cf210844.13
Build Date 11/6/08
———————————————————————
End Installation Status Report
———————————————————————

2. Download upgrade patches certified for above release

After download the patches looked like below
-rw-rw-r– 1 was61 was61 439113810 Mar 31 09:44 6.1.0-WS-WAS-LinuxX64-FP0000029.pak
-rw-rw-r– 1 was61 was61  18690237 Mar 31 09:44 6.1.0-WS-PLG-LinuxX64-FP0000029.pak
-rw-rw-r– 1 was61 was61  12795097 Mar 31 09:44 6.1.0-WS-IHS-LinuxX64-FP0000029.pak
-rwxrwxr-x 1 was61 was61 108644602 Mar 31 10:17 7.0.0.9-WS-UPDI-LinuxAMD64.zip

3. Shutdown IHS,Websphere Services

Stop JVM – /opt/IBM/WebSphere/AppServer/profiles/Profile01/Node/bin/stopServer.sh
Stop DMGR- /opt/IBM/WebSphere/AppServer/profiles/Profile01/Node/bin/stopNode.sh
Stop IHS – sudo /opt/IBM/HTTPServer/bin/apachectl -k stop

4. Intall “IBM Update Installer”  : Xwindows GUI Needed
Unzip 7.0.0.9-WS-UPDI-LinuxAMD64.zip
Navigate to UpdateInstaller and run “install”
Follow the screens as listed below

Please make sure “Lauch IBM Update Installer” is selected before you click on FINISH.

5. Use new installer to apply fix pack to IHS, WAS & PLUG-IN

Click on dropdown and then select HTTPServer

Please navigate to directory where fix pack patches have been downloaded.

I was looking for a NAGIOS monitoring script which will list the archive gap between Primary and 2 of my Standby databases.

http://www.oracledbasupport.co.uk/wp-content/uploads/2010/09/dg_archive_gap_listing-1.txt

When compiled the script will return results as below

[oracle@scripts]$ ./dg_archive_gap.sh
Primary_Arc_No=61077 but Stby_Arc_No=61066

[oracle@scripts]$ ./dg_archive_gap.sh
Primary_Arc_No=61088 but Stby_Arc_No=61047
Stby_Arc_No=61067 but Stby_Apply_No=61047
Dr_Arc_No=61067 but Dr_Apply_No=61047

If you have multiple standby databases , please make sure you Add SQL stmts with “Dest_Id=X” , here 1 (Primary), 2 (Standby) & 3 (DR standby)

#!/bin/bash
#set Oracle environment for Sql*Plus
export ORACLE_HOME=/u01/app/oracle/product/10.2.0
export ORACLE_SID=prod
export PATH=$PATH:$ORACLE_HOME/bin

#set working directory. script is located here..
cd /home/oracle/scripts

#Problem statement is constructed in message variable
MESSAGE=””

#hostname of the primary DB.. used in messages..
HOST_NAME=`hostname`

#Get Data guard information to Unix shell variables…

MESSAGE=`
sqlplus -s sys/sys as sysdba <<EOSQL
set pages 0 feed off
set serverout on size 1000000;
Var Rc Number;
Declare
Local_Arc             Number:=0;
Stby_Arc              Number:=0;
Dr_Arc                Number:=0;
Stby_Apply            Number:=0;
Dr_Apply              Number:=0;
Msg                   Varchar2(4000):=”;
Begin
:Rc:=0;
Select Archived_Seq# Into Local_Arc
From V\\$Archive_Dest_Status Where Dest_Id=1;
Select Archived_Seq#, Applied_Seq#
Into Stby_Arc, Stby_Apply
From V\\$Archive_Dest_Status Where Dest_Id=2;
Select Archived_Seq#, Applied_Seq#
Into Dr_Arc, Dr_Apply
From V\\$Archive_Dest_Status Where Dest_Id=3;

If Local_Arc > Stby_Arc + 10 Then
Msg:=Msg||’Primary_Arc_No=’||Local_Arc||’ but Stby_Arc_No=’||Stby_Arc||Chr(10) || ‘–‘;
:Rc:=1;
End If;

If Stby_Arc > Stby_Apply + 10 Then
Msg:=Msg||’Stby_Arc_No=’||Stby_Arc||’ but Stby_Apply_No=’||Stby_Apply||Chr(10) || ‘–‘;
:Rc:=1;
End If;

If :Rc != 0 Then
Dbms_Output.Put_Line(Msg);
End If;

If Dr_Arc > Dr_Apply + 10 Then
Msg:=Msg||’Dr_Arc_No=’||Dr_Arc||’ but Dr_Apply_No=’||Dr_Apply||Chr(10)|| ‘–‘;
:Rc:=1;
End If;

If :Rc != 0 Then
Dbms_Output.Put_Line(Msg);
End If;
End;
/
exit :rc
EOSQL`
echo $MESSAGE
if [ $? -ne 0 ]; then
MESSAGE=”Error on $HOST_NAME Standby -log APPLY- service!\n${MESSAGE}\nThis problem may cause the archive directories to get full!!!\n\n”
echo -e $MESSAGE
fi

I have many websphere 6.x ND clustered servers and growing every single week. Lots of them are upgraded from previous websphere installations to ND 6.x now.

We have admin shell scripts written but they are not portable.  There is diff in naming conventions for profile, Cluster name, Application Server name etc. We have some issues and they all point to diff in configuration settings.

Recently I came across tool called “Visual Configuration Explorer (VCE)”.

Here is a forum for discussions related to this tool:
http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1139

Getting Started with Visual Configuration Explorer

Download the Workbench component of ISA Version 4.0 for Windows via the following URL:
http://www.ibm.com/software/support/isa/

You may want to go thru this excellent ppt file about IBM Support assistant

http://www.oracledbasupport.co.uk/wp-content/uploads/2010/09/Headless_Help.txt
http://www.oracledbasupport.co.uk/wp-content/uploads/2010/09/VCEHeadlessRuntimeReadme-1.txt