10g Grid : Troubleshooting Grid Control/ Grid Log files

When troubleshooting problems in the Grid Control framework, it is important to remember that Grid Control is a J2EE application deployed to the Oracle Application Server 10gJ2EE and Web Cache installation using an Oracle database as a repository.When faced with a problem, first localize the problem to the affected component Troubleshooting the OMR Troubleshooting the OMS Troubleshooting the OMA

1. Troubleshooting the OMR ( Repository/Database )

Connectivity: Access to the OMR is through the database listener. Please locate listener log file at ORACLE_HOME/network/log. The log file contains record of every connection and connection request received by the listener. Listener errors are recorded in the log file with a TNS-xxxxx format. If necessary, enable tracing for listener to record more diagnostic information. Trace files for the listener can be found in ORACLE_HOME/network/trace

Availability: If OMR database is unavailable for some other reason, check database alert log and trace files to determine the root cause.

Space usage: Space problems may occur in the OMR database if the OMR tablespaces are unable to accept new information due to lack of free space in the data files.

Performance: OMR database performance problems will normally trigger Grid Control alerts as metric threshold values are crossed.

2. Troubleshooting the OMS ( Management Service )

OMS log files are produced for – 1> Oracle Web Cache, 2> Oracle HTTP Server, 3>  Oracle Application Server Containers for J2EE, and 4> the Oracle Process Monitor and Notification subcomponents.

1> Oracle Web Cache

Log files may be found in ORACLE_HOME/webcache/logs.

There are two key log files for Web Cache: the access_log (which records every connection to the Web Cache) and the event_log (which contains data about Web Cache availability and errors).

2> The Oracle HTTP Server

provides the access_log and error_log in ORACLE_HOME/Apache/Apache/logs. As with Web Cache, the access_log can grow very large and is therefore “rotated” every 12 hours. Older logs are preserved in ORACLE_HOME/Apache/Apache/logs.

3> Oracle Application Server Containers

For J2EE for generated log files for Grid Control may be found in ORACLE_HOME/j2ee/OC4J_EM/log/OC4J_EM_default_island_1. OC4J generates several log files that provide diagnostic information, including:

default-web-access.log:

Contains information about each request received by the component. Information includes the IP address of the requester, date and time of the request, the URL that was specified in the request, and the result code. All requests should come from the OHS; all result codes should indicate success (200). This information is valuable when troubleshooting connection difficulties between the OHS and OC4J.

em-application.log:

Contains information about all events, errors, and exceptions associated with the EM application. This is excellent information for troubleshooting Java errors.

global-application.log:

Contains information about events, errors, and exceptions relating to the OC4J JVM that are not specific to the EM application. This is also a good source of information for troubleshooting Java errors.

server.log:

Includes availability information for the OC4J_EM component, including start and stop times.

4> Oracle Process Monitor and Notification (OPMN) system provides logs for each of the OMS components in ORACLE_HOME/opmn/logs. Key log files include:

HTTP_Server: In cases where the OHS will not start, this log file often contains pertinent error messages that can help diagnose the problem.

OC4J~OC4J_EM~default_island~1: Contains any errors received while starting the OC4J_EM component. This can be helpful in troubleshooting Java errors and global configuration problems.

3.  Troubleshooting the OMA ( Management Agent )

Connectivity between the OMA and OMS: When a single management agent is unable to connect to the OMS, the problem will normally be found on the OMA’s server. If multiple agents are unable to connect, the problem may lie with the OMS or underlying network. Check AGENT_HOME/sysman/config/emd.properties and verify that the repository URL is correct.

Ensure you can ping the host identified as the repository URL. attempt to telnet to the OMS host.

Upload throughput as the OMA reports information on metric targets through the OMS to the OMR: OMA logs may be found in AGENT_HOME/sysman/log. OMA upload errors are recorded in AGENT_HOME/sysman/log/emdctl.trc.

Target discovery as new targets are added to a server: Use emctl config listtargets or check AGENT_HOME/sysman/emd/targets.xml to determine which targets are monitored by the agent. Remember to make a backup copy of the targets.xml file prior to any modifications. Errors with target discovery will be reported in AGENT_HOME/sysman/log/agentca.log.