Tweet
Change Display Name Of Targets In a Grid Control
If you carry Agent auto discovery, the display names cannot be configured.
To configure the display names of the targets in the GRID Control console, the targets must be manually discovered.
If the targets are already added, then they would have to be removed from the console and then added manually:
1) Remove the target from grid control.
Targets > All Targets > Search for the target in question > Select the radio button next to the target and choose Remove.
2) Wait for the delete to be completed.
3) Verify that the delete is complete by connecting to the repository database as the sysman user:
set pages 1000
select * from mgmt_targets_delete;
The DELETE_COMPLETE_TIME must be populated, do not continue until this is populated.
4) Add the new database manually in GC. This can be done in a couple of places depending on the target type.
Targets > All Targets > Add:
Setup link (Top right of console) > Agent subtab (blue bar across the top of the console) > click on the link for the agent that monitors the target (this goes to the agent homepage) > Add the target here.
If the target is not already in the console, manually using the same steps noted.
RMAN Offline Backup Script
rman> shutdown immediate;
rman> startup mount
1. Let’s validate if there is ample of space for backup at Target device type i.e DISK/TAPE
ALLOCATE CHANNEL c1 DEVICE TYPE sbt;
RMAN>
run {
ALLOCATE CHANNEL c1 DEVICE TYPE sbt;
BACKUP VALIDATE DATABASE;
}allocated channel: c1
channel c1: sid=18 devtype=SBT_TAPE
channel c1: VERITAS NetBackup for Oracle – Release 4.5MP6 (00000000)Starting backup at 14-JAN-09
channel c1: starting full datafile backupset
channel c1: specifying datafile(s) in backupsetinput datafile fno=00010 name=/oradata/oradataTEST/tspace/TESTO5TEST_TESTmes_idx_01.dbf
input datafile fno=00007 name=/oradata/oradataTEST/tspace/TESTO5TEST_TESTtrn_dat_01.dbf
input datafile fno=00009 name=/oradata/oradataTEST/tspace/TESTO5TEST_TESTmes_dat_01.dbf
input datafile fno=00017 name=/oradata/oradataTEST/tspace/TESTO5TEST_TESTtrn_idx_02.dbf
input datafile fno=00019 name=/oradata/oradataTEST/tspace/TESTO5TEST_TESTvch_idx_02.dbf
2. If validate doesn’t return any errors go head with following script to backup entire database.
rman target sys/manager nocatalog
RMAN> shutdown immediate
RMAN> startup mount
RMAN>
run { allocate channel c1 device type disk format ‘%Directory%\backup_db_%d_S_%s_P_%p_T_%t’;
# I need to create a compressed backup set due to lack of drive space.
# The entire database backup was compressed from 77GB to 5GB (10G new feature) so it does work well.
backup as compressed backupset full skip inaccessible
tag rman_backup_Offline_full
filesperset 10
(database include current controlfile);
delete obsolete;
crosscheck backup
}
Manually resolve archive log gap/ Recover Standby / Manual Standby recovery
The primary and standby database have the same filesystem layout, i.e. archive redo is at same place on both servers.
References: Metalink note 150214.1 – Synchronization of primary and standby database due to log transfer gap
Identify the missing log sequence numbers by running this SQL on the STANDBY database:
SQL> select thread#, low_sequence#, high_sequence# from v$archive_gap;
Identify the filenames of the missing logs by running this SQL on the PRIMARY database:
SQL>SELECT NAME
FROM V$ARCHIVED_LOG
WHERE DEST_ID = 1 AND SEQUENCE# BETWEEN &Low_Sequence AND &High_Sequence;
1. Cancel managed recovery in the STANDBY database
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
2. Copy the missing archive redo logs from primary to standby server.
This can be done using any available O/S utility. If you use ftp make sure the transfer is done in binary mode. We assume here that the redo files are going to the same location on the standby server.
3. Recover the STANDBY database
SQL> RECOVER AUTOMATIC STANDBY DATABASE;
When all logs have been applied Oracle will ask for the next, as yet unarchived, redo log. At this point cancel the recovery, e.g.
ORA-00279: change 2668131127884 generated at 12/16/2004 07:45:58 needed forthread 2
ORA-00289: suggestion : /oraarchive/standby/arc2_00000001992.log
ORA-00280: change 2668131127884 for thread 2 is in sequence #1992
ORA-00278: log file ‘/oraarchive/standby/arc2_00000001992.log’no longer needed for this recovery
ORA-00308: cannot open archived log ‘/oraarch01/smp/arch/arc2_00000001992.log’
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
Additional information: 3
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
CANCEL
4. Put the STANDBY database back into managed recovery mode:
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;
5. Check that the archive redo gaps have been resolved byt running thie SQL on the STANDBY database:
SQL> select thread#, low_sequence#, high_sequence# from v$archive_gap;
10g Grid : Start OMS /Stop OMS/Restart OMS
| Oracle Process Monitor and Notification Control utility
$ opmnctl startall |
Enterprise Manager Control Utility (emctl) $ emctl start oms $ emctl stop oms $ emctl status oms $ emctl start iasconsole $ emctl stop iasconsole |
Distributed Configuration Manager Control (dcmctl) $ dcmctl start $ dcmctl start –ct WebCache $ dcmctl stop $ dcmctl getstate $ dcmctl listcomponents |
Output of Commands
$ export ORACLE_HOME=/opt/oracle/product/oms10g/
$ echo $ORACLE_HOME
/opt/oracle/product/oms10g/
$ /opt/oracle/product/oms10g/opmn/bin/opmnctl status -l
Processes in Instance: EnterpriseManager0.test -------------------+--------------------+---------+----------+------------+----------+-----------+------ ias-component | process-type | pid | status | uid | memused | uptime | ports -------------------+--------------------+---------+----------+------------+----------+-----------+------ DSA | DSA | N/A | Down | N/A | N/A | N/A | N/A LogLoader | logloaderd | N/A | Down | N/A | N/A | N/A | N/A HTTP_Server | HTTP_Server | 749 | Alive | 1325924656 | 194208 | 123:54:07 | http1:7778,http2:7200,https1:4444,https2:1159,http3:4889 dcm-daemon | dcm-daemon | 629 | Alive | 1325924655 | 24956 | 123:54:39 | N/A OC4J | home | 23906 | Alive | 1325924658 | 33148 | 121:24:21 | ajp:12502,rmi:12402,jms:12602 OC4J | OC4J_EMPROV | 23907 | Alive | 1325924659 | 57724 | 121:24:21 | ajp:12503,rmi:12403,jms:12603 OC4J | OC4J_EM | 12150 | Alive | 1325924667 | 242044 | 1193:01:~ | ajp:12501,rmi:12401,jms:12601 WebCache | WebCache | 23908 | Alive | 1325924660 | 106924 | 121:24:21 | http:7777,invalidation:9401,statistics:9402 WebCache | WebCacheAdmin | 23909 | Alive | 1325924661 | 15652 | 121:24:21 | administration:9400
$ opmnctl status
Processes in Instance: EnterpriseManager0.test -------------------+--------------------+---------+--------- ias-component | process-type | pid | status -------------------+--------------------+---------+--------- DSA | DSA | N/A | Down LogLoader | logloaderd | N/A | Down HTTP_Server | HTTP_Server | 749 | Alive dcm-daemon | dcm-daemon | 629 | Alive OC4J | home | 23906 | Alive OC4J | OC4J_EMPROV | 23907 | Alive OC4J | OC4J_EM | 12150 | Alive WebCache | WebCache | 23908 | Alive WebCache | WebCacheAdmin | 23909 | Alive
To Start and stop individual ias components use : opmnctl startproc ias-component=OC4J (See IAS-Component Above)
To Start and stop individual processes use : opmnctl startproc process-type=OC4J_EMPROV (See Process-Type Above)
$ /opt/oracle/product/oms10g/opmn/bin/opmnctl stopall
opmnctl: stopping opmn and all managed processes...
================================================================================
opmn id=test:6201
5 of 6 processes stopped. ias-instance id=EnterpriseManager0.test
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
OC4J/OC4J_EM/default_island
Error
--> Process (pid=12150)
time out while waiting for a managed process to stop
Log:
/opt/oracle/product/oms10g/opmn/logs/OC4J~OC4J_EM~default_island~1
opmnctl: graceful stop of processes failed, trying forceful shutdown...
$ opmnctl status
Unable to connect to opmn. Opmn may not be up.
$ opmnctl startall
opmnctl: starting opmn and all managed processes...
================================================================================
opmn id=test:6201
5 of 6 processes started.
ias-instance id=EnterpriseManager0.Test
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
OC4J/OC4J_EM/default_island
Error
--> Process (pid=6900)
failed to start a managed process after the maximum retry limit
Log:
/opt/oracle/product/oms10g/opmn/logs/OC4J~OC4J_EM~default_island~1
If “opmnctl startall” fails use “ias-component” and “process-type” to start individual components
$ opmnctl startproc ias-component=OC4J
opmnctl: starting opmn managed processes...
================================================================================
opmn id=Test:6201
0 of 1 processes started.
ias-instance id=EnterpriseManager0.Test
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ias-component/process-type/process-set:
OC4J/OC4J_EM/default_island
Error
--> Process (pid=20563)
failed to start a managed process after the maximum retry limit
Log:
/opt/oracle/product/oms10g/opmn/logs/OC4J~OC4J_EM~default_island~1
$ opmnctl status
Processes in Instance: EnterpriseManager0.Test -------------------+--------------------+---------+--------- ias-component | process-type | pid | status -------------------+--------------------+---------+--------- DSA | DSA | N/A | Down LogLoader | logloaderd | N/A | Down HTTP_Server | HTTP_Server | 4750 | Alive dcm-daemon | dcm-daemon | N/A | Down OC4J | home | 4752 | Alive OC4J | OC4J_EMPROV | 4751 | Alive OC4J | OC4J_EM | N/A | Down WebCache | WebCache | 4754 | Alive WebCache | WebCacheAdmin | 4755 | Alive
In this case both options “ias-component” and “process-type” failed to start OMS. We were not inclined to restart database but there were rogue processes and at the end we had to restart database to get OMS working.
$ /opt/oracle/product/oms10g/opmn/bin/opmnctl status
Processes in Instance: EnterpriseManager0.o2hp127 -------------------+--------------------+---------+--------- ias-component | process-type | pid | status -------------------+--------------------+---------+--------- DSA | DSA | N/A | Down LogLoader | logloaderd | N/A | Down HTTP_Server | HTTP_Server | 27225 | Alive dcm-daemon | dcm-daemon | N/A | Down OC4J | home | 27230 | Alive OC4J | OC4J_EMPROV | 27226 | Alive OC4J | OC4J_EM | 27227 | Alive WebCache | WebCache | 27228 | Alive WebCache | WebCacheAdmin | 27229 | Alive
10g Grid : Troubleshooting Grid Control/ Grid Log files
When troubleshooting problems in the Grid Control framework, it is important to remember that Grid Control is a J2EE application deployed to the Oracle Application Server 10gJ2EE and Web Cache installation using an Oracle database as a repository.When faced with a problem, first localize the problem to the affected component
|
 |
1. Troubleshooting the OMR ( Repository/Database )
Connectivity: Access to the OMR is through the database listener. Please locate listener log file at ORACLE_HOME/network/log. The log file contains record of every connection and connection request received by the listener. Listener errors are recorded in the log file with a TNS-xxxxx format. If necessary, enable tracing for listener to record more diagnostic information. Trace files for the listener can be found in ORACLE_HOME/network/trace
Availability: If OMR database is unavailable for some other reason, check database alert log and trace files to determine the root cause.
Space usage: Space problems may occur in the OMR database if the OMR tablespaces are unable to accept new information due to lack of free space in the data files.
Performance: OMR database performance problems will normally trigger Grid Control alerts as metric threshold values are crossed.
2. Troubleshooting the OMS ( Management Service )
OMS log files are produced for – 1> Oracle Web Cache, 2> Oracle HTTP Server, 3> Oracle Application Server Containers for J2EE, and 4> the Oracle Process Monitor and Notification subcomponents.
1> Oracle Web Cache
Log files may be found in ORACLE_HOME/webcache/logs.
There are two key log files for Web Cache: the access_log (which records every connection to the Web Cache) and the event_log (which contains data about Web Cache availability and errors).
2> The Oracle HTTP Server
provides the access_log and error_log in ORACLE_HOME/Apache/Apache/logs. As with Web Cache, the access_log can grow very large and is therefore “rotated” every 12 hours. Older logs are preserved in ORACLE_HOME/Apache/Apache/logs.
3> Oracle Application Server Containers
For J2EE for generated log files for Grid Control may be found in ORACLE_HOME/j2ee/OC4J_EM/log/OC4J_EM_default_island_1. OC4J generates several log files that provide diagnostic information, including:
default-web-access.log:
Contains information about each request received by the component. Information includes the IP address of the requester, date and time of the request, the URL that was specified in the request, and the result code. All requests should come from the OHS; all result codes should indicate success (200). This information is valuable when troubleshooting connection difficulties between the OHS and OC4J.
em-application.log:
Contains information about all events, errors, and exceptions associated with the EM application. This is excellent information for troubleshooting Java errors.
global-application.log:
Contains information about events, errors, and exceptions relating to the OC4J JVM that are not specific to the EM application. This is also a good source of information for troubleshooting Java errors.
server.log:
Includes availability information for the OC4J_EM component, including start and stop times.
4> Oracle Process Monitor and Notification (OPMN) system provides logs for each of the OMS components in ORACLE_HOME/opmn/logs. Key log files include:
HTTP_Server: In cases where the OHS will not start, this log file often contains pertinent error messages that can help diagnose the problem.
OC4J~OC4J_EM~default_island~1: Contains any errors received while starting the OC4J_EM component. This can be helpful in troubleshooting Java errors and global configuration problems.
3. Troubleshooting the OMA ( Management Agent )
Connectivity between the OMA and OMS: When a single management agent is unable to connect to the OMS, the problem will normally be found on the OMA’s server. If multiple agents are unable to connect, the problem may lie with the OMS or underlying network. Check AGENT_HOME/sysman/config/emd.properties and verify that the repository URL is correct.
Ensure you can ping the host identified as the repository URL. attempt to telnet to the OMS host.
Upload throughput as the OMA reports information on metric targets through the OMS to the OMR: OMA logs may be found in AGENT_HOME/sysman/log. OMA upload errors are recorded in AGENT_HOME/sysman/log/emdctl.trc.
Target discovery as new targets are added to a server: Use emctl config listtargets or check AGENT_HOME/sysman/emd/targets.xml to determine which targets are monitored by the agent. Remember to make a backup copy of the targets.xml file prior to any modifications. Errors with target discovery will be reported in AGENT_HOME/sysman/log/agentca.log.
