I wanted to remove one my client node from OMS. I used following GUI option but it returned errors  as below

I clicked “Remove” tab which returned error message “There are other monitored targets on this Host. Before removing this Host, you must first remove all the other targets”

How to Fix this?

Run following SQL at OMR

select distinct target_name,target_type  from SYSMAN.MGMT$TARGET where target_name like ‘%HOSTNAME%’

TARGET_NAME,                  TARGET_TYPE
bmau0.uk.pri.o2.com,          host
bmau0.uk.pri.o2.com:1831,   oracle_emd
bmau0:1831,                     oracle_emd
LISTENER_TEST0_bmau0,    oracle_listener
LISTENER_TEST_bmau0,      oracle_listener
LISTENER_DBAS_bmau0,      oracle_listener

Now use following API to remove each of these targets from OMR

exec sysman.mgmt_admin.cleanup_agent(‘bmau0.uk.pri.o2.com:3872’);
exec sysman.mgmt_admin.cleanup_agent(‘bmau0.uk.pri.o2.com:1831’);

If you carry Agent auto discovery, the display names cannot be configured.

To configure the display names of the targets in the GRID Control console, the targets must be manually discovered.

If the targets are already added, then they would have to be removed from the console and then added manually:

1) Remove the target from grid control.

Targets > All Targets > Search for the target in question > Select the radio button next to the target and choose Remove.

2) Wait for the delete to be completed.

3) Verify that the delete is complete by connecting to the repository database as the sysman user:

set pages 1000
select * from mgmt_targets_delete;
The DELETE_COMPLETE_TIME must be populated, do not continue until this is populated.

4) Add the new database manually in GC.  This can be done in a couple of places depending on the target type.

Targets > All Targets > Add:

Setup link (Top right of console) > Agent subtab (blue bar across the top of the console) > click on the link for the agent that monitors the target (this goes to the agent homepage) > Add the target here.

If the target is not already in the console, manually using the same steps noted.

Output of Commands

$ export ORACLE_HOME=/opt/oracle/product/oms10g/
$ echo $ORACLE_HOME
/opt/oracle/product/oms10g/

$ /opt/oracle/product/oms10g/opmn/bin/opmnctl status -l

Processes in Instance: EnterpriseManager0.test
-------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component      | process-type       |     pid | status   |        uid |  memused |    uptime | ports
-------------------+--------------------+---------+----------+------------+----------+-----------+------
DSA                | DSA                |     N/A | Down     |        N/A |      N/A |       N/A | N/A
LogLoader          | logloaderd         |     N/A | Down     |        N/A |      N/A |       N/A | N/A
HTTP_Server        | HTTP_Server        |     749 | Alive    | 1325924656 |   194208 | 123:54:07 | http1:7778,http2:7200,https1:4444,https2:1159,http3:4889
dcm-daemon         | dcm-daemon         |     629 | Alive    | 1325924655 |    24956 | 123:54:39 | N/A
OC4J               | home               |   23906 | Alive    | 1325924658 |    33148 | 121:24:21 | ajp:12502,rmi:12402,jms:12602
OC4J               | OC4J_EMPROV        |   23907 | Alive    | 1325924659 |    57724 | 121:24:21 | ajp:12503,rmi:12403,jms:12603
OC4J               | OC4J_EM            |   12150 | Alive    | 1325924667 |   242044 | 1193:01:~ | ajp:12501,rmi:12401,jms:12601
WebCache           | WebCache           |   23908 | Alive    | 1325924660 |   106924 | 121:24:21 | http:7777,invalidation:9401,statistics:9402
WebCache           | WebCacheAdmin      |   23909 | Alive    | 1325924661 |    15652 | 121:24:21 | administration:9400

$ opmnctl status

Processes in Instance: EnterpriseManager0.test
-------------------+--------------------+---------+---------
ias-component      | process-type       |     pid | status
-------------------+--------------------+---------+---------
DSA                | DSA                |     N/A | Down
LogLoader          | logloaderd         |     N/A | Down
HTTP_Server        | HTTP_Server        |     749 | Alive
dcm-daemon         | dcm-daemon         |     629 | Alive
OC4J               | home               |   23906 | Alive
OC4J               | OC4J_EMPROV        |   23907 | Alive
OC4J               | OC4J_EM            |   12150 | Alive
WebCache           | WebCache           |   23908 | Alive
WebCache           | WebCacheAdmin      |   23909 | Alive

To Start and stop individual ias components use :  opmnctl startproc ias-component=OC4J (See IAS-Component Above)

To Start and stop individual processes use :  opmnctl startproc process-type=OC4J_EMPROV (See Process-Type Above)

$ /opt/oracle/product/oms10g/opmn/bin/opmnctl stopall

opmnctl: stopping opmn and all managed processes...
================================================================================
opmn id=test:6201
    5 of 6 processes stopped. ias-instance id=EnterpriseManager0.test
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
    OC4J/OC4J_EM/default_island

Error
--> Process (pid=12150)
    time out while waiting for a managed process to stop
    Log:
    /opt/oracle/product/oms10g/opmn/logs/OC4J~OC4J_EM~default_island~1
opmnctl: graceful stop of processes failed, trying forceful shutdown...

$ opmnctl status

Unable to connect to opmn.
Opmn may not be up.

$ opmnctl  startall

opmnctl: starting opmn and all managed processes...
================================================================================
opmn id=test:6201
    5 of 6 processes started.

ias-instance id=EnterpriseManager0.Test
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
    OC4J/OC4J_EM/default_island

Error
--> Process (pid=6900)
    failed to start a managed process after the maximum retry limit
    Log:
    /opt/oracle/product/oms10g/opmn/logs/OC4J~OC4J_EM~default_island~1

If “opmnctl startall” fails use “ias-component” and “process-type” to start individual components

$ opmnctl startproc ias-component=OC4J

opmnctl: starting opmn managed processes...
================================================================================
opmn id=Test:6201
    0 of 1 processes started.

ias-instance id=EnterpriseManager0.Test
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ias-component/process-type/process-set:
    OC4J/OC4J_EM/default_island

Error
--> Process (pid=20563)
    failed to start a managed process after the maximum retry limit
    Log:
    /opt/oracle/product/oms10g/opmn/logs/OC4J~OC4J_EM~default_island~1

$ opmnctl status

Processes in Instance: EnterpriseManager0.Test
-------------------+--------------------+---------+---------
ias-component      | process-type       |     pid | status
-------------------+--------------------+---------+---------
DSA                | DSA                |     N/A | Down
LogLoader          | logloaderd         |     N/A | Down
HTTP_Server        | HTTP_Server        |    4750 | Alive
dcm-daemon         | dcm-daemon         |     N/A | Down
OC4J               | home               |    4752 | Alive
OC4J               | OC4J_EMPROV        |    4751 | Alive
OC4J               | OC4J_EM            |     N/A | Down
WebCache           | WebCache           |    4754 | Alive
WebCache           | WebCacheAdmin      |    4755 | Alive

Processes in Instance: EnterpriseManager0.o2hp127
-------------------+--------------------+---------+---------
ias-component      | process-type       |     pid | status
-------------------+--------------------+---------+---------
DSA                | DSA                |     N/A | Down
LogLoader          | logloaderd         |     N/A | Down
HTTP_Server        | HTTP_Server        |   27225 | Alive
dcm-daemon         | dcm-daemon         |     N/A | Down
OC4J               | home               |   27230 | Alive
OC4J               | OC4J_EMPROV        |   27226 | Alive
OC4J               | OC4J_EM            |   27227 | Alive
WebCache           | WebCache           |   27228 | Alive
WebCache           | WebCacheAdmin      |   27229 | Alive

When troubleshooting problems in the Grid Control framework, it is important to remember that Grid Control is a J2EE application deployed to the Oracle Application Server 10gJ2EE and Web Cache installation using an Oracle database as a repository.When faced with a problem, first localize the problem to the affected component Troubleshooting the OMR Troubleshooting the OMS Troubleshooting the OMA

1. Troubleshooting the OMR ( Repository/Database )

Connectivity: Access to the OMR is through the database listener. Please locate listener log file at ORACLE_HOME/network/log. The log file contains record of every connection and connection request received by the listener. Listener errors are recorded in the log file with a TNS-xxxxx format. If necessary, enable tracing for listener to record more diagnostic information. Trace files for the listener can be found in ORACLE_HOME/network/trace

Availability: If OMR database is unavailable for some other reason, check database alert log and trace files to determine the root cause.

Space usage: Space problems may occur in the OMR database if the OMR tablespaces are unable to accept new information due to lack of free space in the data files.

Performance: OMR database performance problems will normally trigger Grid Control alerts as metric threshold values are crossed.

2. Troubleshooting the OMS ( Management Service )

OMS log files are produced for – 1> Oracle Web Cache, 2> Oracle HTTP Server, 3>  Oracle Application Server Containers for J2EE, and 4> the Oracle Process Monitor and Notification subcomponents.

1> Oracle Web Cache

Log files may be found in ORACLE_HOME/webcache/logs.

There are two key log files for Web Cache: the access_log (which records every connection to the Web Cache) and the event_log (which contains data about Web Cache availability and errors).

2> The Oracle HTTP Server

provides the access_log and error_log in ORACLE_HOME/Apache/Apache/logs. As with Web Cache, the access_log can grow very large and is therefore “rotated” every 12 hours. Older logs are preserved in ORACLE_HOME/Apache/Apache/logs.

3> Oracle Application Server Containers

For J2EE for generated log files for Grid Control may be found in ORACLE_HOME/j2ee/OC4J_EM/log/OC4J_EM_default_island_1. OC4J generates several log files that provide diagnostic information, including:

default-web-access.log:

Contains information about each request received by the component. Information includes the IP address of the requester, date and time of the request, the URL that was specified in the request, and the result code. All requests should come from the OHS; all result codes should indicate success (200). This information is valuable when troubleshooting connection difficulties between the OHS and OC4J.

em-application.log:

Contains information about all events, errors, and exceptions associated with the EM application. This is excellent information for troubleshooting Java errors.

global-application.log:

Contains information about events, errors, and exceptions relating to the OC4J JVM that are not specific to the EM application. This is also a good source of information for troubleshooting Java errors.

server.log:

Includes availability information for the OC4J_EM component, including start and stop times.

4> Oracle Process Monitor and Notification (OPMN) system provides logs for each of the OMS components in ORACLE_HOME/opmn/logs. Key log files include:

HTTP_Server: In cases where the OHS will not start, this log file often contains pertinent error messages that can help diagnose the problem.

OC4J~OC4J_EM~default_island~1: Contains any errors received while starting the OC4J_EM component. This can be helpful in troubleshooting Java errors and global configuration problems.

3.  Troubleshooting the OMA ( Management Agent )

Connectivity between the OMA and OMS: When a single management agent is unable to connect to the OMS, the problem will normally be found on the OMA’s server. If multiple agents are unable to connect, the problem may lie with the OMS or underlying network. Check AGENT_HOME/sysman/config/emd.properties and verify that the repository URL is correct.

Ensure you can ping the host identified as the repository URL. attempt to telnet to the OMS host.

Upload throughput as the OMA reports information on metric targets through the OMS to the OMR: OMA logs may be found in AGENT_HOME/sysman/log. OMA upload errors are recorded in AGENT_HOME/sysman/log/emdctl.trc.

Target discovery as new targets are added to a server: Use emctl config listtargets or check AGENT_HOME/sysman/emd/targets.xml to determine which targets are monitored by the agent. Remember to make a backup copy of the targets.xml file prior to any modifications. Errors with target discovery will be reported in AGENT_HOME/sysman/log/agentca.log.

For last many years I worked for number of clients using all sort of latest technologies. One of my duties was tuning and monitoring system performance which formed part of Capacity planning. I have been using number of tools and when I saw 10G grid first time I could see a real way forward where a database technology could work towards client needs to deliver end result.

There is an ever increasing trend on companies using managed service contract for Database and Application support. The Billing for these services is derived from time spent on fixing problems/issues. For every managed service contract you have to use case management tools like Remedy,Clarify, Hp Openview . These tools raise work cases on a specific event and could be tracked up and down different departments responsible for an activity.

Also a tool like 10G grid needs to be capable monitoring range of other systems like Sybase,SQL Server,Mysql , SAN file systems.

The initial version of 10g grid was not capable of working with case management tools or monitoring heterogeneous systems  and as a result 10g grid pick up was slow by most companies. But now Oracle have released following  connectors to liaise with these systems.

• Login to grid control using sysman or any admin user and click on deployments tab

• Scroll down at same screen and click on “Install Agent” under “Agent Installation” section

• Select “Fresh Install” at this screen

• At this screen select Client platform, Agent version, Installation Base Directory, OS Credentials

• Scroll down to Management Server Security section and specify registration password. Providing password will configure the agents to run in secure mode after installation.

• Click on Continue. Grid will start installing an agent at Target after initial checks.

Once installed make sure Agent is running at Target and locate it at grid control

Agent Deploy Log Files

Advantages of using grid control to schedule jobs

• Could be run against multiple targets at any given time
• Run a job immediately or repeatedly on a schedule
• Could be submitted against number of targets
• Could be shared using job library

1. View Jobs at Grid Control:

Login at Grid and click on Top tab –> Jobs –> Click on “Job Library” option to view number of jobs.

To see activity of job click on “Job Activity”

2. Create a new Job : I often see massive trace files consuming disk space. I will “create a dbms job to delete bdump trace files” (*.trc) older than 7 days i.e “find /opt/oracle/admin/repAAA/bdump -type f -mtime +7 -exec rm -f {} \;”

Select “Create Library Job”  -> “OS command” and hit <Go>

Add Description of a Job like “Delete udump trace files”

Add OS command at COMMAND box  “find /opt/oracle/admin/repAAA/bdump -type f -mtime +7 -exec rm -f {} \;”.  I selected “Single Operation” as it is a single UNIX command to delete files. If you have a UNIX script then please make sure you select Script at a dropdown box.

Enter OS credentials for a Target System

Add a schedule on what times you want to run “command/script”.

Now select “Save to Library” from right hand top menu.

Click on Database Tab at grid Control

Scroll down at database page and select “Metric and Policy Settings” under “Related Links”

Please select Metric you want to alter and click “OK”

Background : I have installed OMS/OMR and OMA successfully. In theory I should see OMA client successfully configured at OMS grid console but

$ ./emctl status agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version     : 10.2.0.3.0
OMS Version       : 10.2.0.2.0
Protocol Version  : 10.2.0.2.0
Agent Home        : /opt/oracle/product/oracle10gAgent/agent10g
Agent binaries    : /opt/oracle/product/oracle10gAgent/agent10g
Agent Process ID  : 23314
Parent Process ID : 23295
—————————————————————
Agent is Running but Not Ready

To locate an exact error look at OMS files $OMS_HOME\sysman\log\emoms.log and $OMS_HOME\sysman\log\emoms.trc

My  $OMS_HOME\sysman\log\emoms.trc show error at  Trigger ‘SYSMAN.EM_VIOLATION_CHECKS’

2008-12-02 15:00:37,180 [SeverityLoad https://bmhp01:3872/emd/main/] ERROR XMLLoader.Handler processInsertException.462 – java.sql.SQLException: ORA-20612: Internal error processing severity. Closing availability record with negative duration for :  (target_guid = 1C2A2131D1A40C621929D7CE63032001) (collection_ts = 2008-12-02 14:47:07) (current_status = AGENT DOWN)
ORA-06512: at “SYSMAN.EM_SEVERITY”, line 2110
ORA-06512: at “SYSMAN.EM_SEVERITY”, line 3287
ORA-06512: at “SYSMAN.EM_VIOLATION_CHECKS”, line 791
ORA-04088: error during execution of trigger ‘SYSMAN.EM_VIOLATION_CHECKS’

Check at Google or metalink for further details on it.

– Sometimes you may have to delete records from OMR tables to fix this issue see Note:603011.1

– For ORA-04088: during execution of trigger SYSMAN.TARGETS_INSERT_TRIGGER , locate TARGET_GUID/AGENT_GUID at emoms.log  for example CE6B5C2E232766E01D3803EB03D70128 and run “delete from MGMT_TARGET_ASSOC_INSTANCE where target_guid = HEXTORAW(‘TARGET_GUID’);”

How to fix it? For clustered install (RAC/OS cluster) please see details here

$ ./emctl stop agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
Stopping agent … stopped.

Delete all files in $AGENT_HOME/sysman/emd/upload and $AGENT_HOME/sysman/emd/state

$ ./emctl clearstate agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
EMD clearstate completed successfully

$ ./emctl unsecure agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
Checking Agent for HTTP…   Done.
Agent is already stopped…   Done.
Unsecuring agent…   Started.
OMS Upload URL – http://ASST105157:4889/em/upload/ is locked or unavailable.

$ ./emctl upload agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
—————————————————————
EMD upload completed successfully

$ ./emctl status agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version     : 10.2.0.3.0
OMS Version       : 10.2.0.2.0
Protocol Version  : 10.2.0.2.0
Agent Home        : /opt/oracle/product/oracle10gAgent/agent10g
Agent binaries    : /opt/oracle/product/oracle10gAgent/agent10g
Agent Process ID  : 23314
Parent Process ID : 23295
Agent URL         : https://bmhp01:3872/emd/main/
Repository URL    : https://ASST105157:1159/em/upload
Started at        : 2008-12-02 14:52:07
Started by user   : oracle
Last Reload       : 2008-12-02 14:52:07
Last successful upload                       : 2008-12-02 15:05:03
Total Megabytes of XML files uploaded so far :     5.25
Number of XML files pending upload           :        0
Size of XML files pending upload(MB)         :     0.00
Available disk space on upload filesystem    :    32.43%
Last successful heartbeat to OMS             : 2008-12-02 15:04:50
—————————————————————
Agent is Running and Ready

$ ./emctl secure agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
Enter Agent Registration password :
Agent successfully stopped…   Done.
Securing agent…   Started.
Requesting an HTTPS Upload URL from the OMS…   Done.
Requesting an Oracle Wallet and Agent Key from the OMS…   Done.
Check if HTTPS Upload URL is accessible from the agent…   Done.
Configuring Agent for HTTPS in CENTRAL_AGENT mode…   Done.
EMD_URL set in /opt/oracle/product/oracle10gAgent/agent10g/sysman/config/emd.properties
Securing agent…   Successful.

I can see new new target at grid console

Download OMS software from here

I am using Windows 10.2.x release for my OMS install. Due to limited hardware I am installing OMS (oracle management service or GRID console) on windows box while OMR (OMS database) configured at remote HP True 64 system. I have a separate database instance called “oms” for OMR.

Following diagram will give you an idea on OMS/OMR and OMA

Towards end of install, Oracle will configure default configuration and will take some time depending on underlying hardware. Please be patient until you see status=”SUCCEEDED” at GUI Installer.

Let go ahead and install Grid Agent at clients for monitoring.

How to start a GRID Console?

1.  OMR : Please start OMR (Oracle Management Repository)database as well a listener

2.  OMA : If you are using windows make sure “XXX10gAgent” service is started, for Unix/Linux use “emctl start agent” to start an agent

3. OMS : OMS could be controlled using number of methods

a. Use opmnctl to start and stop Oracle OMS

C:\oracle\product\10.2.0\oms10g\opmn\bin>opmnctl stopall
opmnctl: stopping opmn and all managed processes...

At this point http://asst105157:4889/em/console/ will return an error use following command to start all OMS components

C:\oracle\product\10.2.0\oms10g\opmn\bin>opmnctl startall
opmnctl: starting opmn and all managed processes...

C:\oracle\product\10.2.0\oms10g\opmn\bin>opmnctl status -l
Processes in Instance: EnterpriseManager0.asst105157

-ias-component     | process-type       |     pid | status   |        uid |  memused |    uptime | ports-------------------+--------------------+---------+----------+------------+----------+-----------+------DSA                | DSA                |     N/A | Down     |        N/A |      N/A |       N/A | N/AHTTP_Server        | HTTP_Server        |    4460 | Alive    |  607649821 |    43796 |  00:03:11 | http1:7778,http2:7200,https1:4444,https2:1159,http3:4889
LogLoader          | logloaderd         |     N/A | Down     |        N/A |      N/A |       N/A | N/Adcm-daemon         | dcm-daemon         |     N/A | Down     |        N/A |      N/A |       N/A | N/AOC4J               | home               |    5620 | Alive    |  607649822 |    30928 |  00:03:34 | ajp:12501,rmi:12401,jms:12601
OC4J               | OC4J_EM            |    5756 | Alive    |  607649823 |    30564 |  00:03:34 | ajp:12502,rmi:12402,jms:12602
OC4J               | OC4J_EMPROV        |    1260 | Alive    |  607649824 |    30796 |  00:03:34 | ajp:12503,rmi:12403,jms:12603
WebCache           | WebCache           |    4684 | Alive    |  607649825 |    31372 |  00:03:34 | http:80,invalidation:9401,statistics:9402
WebCache           | WebCacheAdmin      |     988 | Alive    |  607649826 |     9088 |  00:03:34 | administration:9400

b. Use emctl to start, stop, or check the status of Application Server Control

Start  iasconsole using “emctl start iasconsole”  and try http://<Host IP>:1810 to manage OMS components

c. Use dcmctl   to start and stop Oracle OMS

C:\oracle\product\10.2.0\oms10g\dcm\bin>dcmctl stop
Current State for Instance:EnterpriseManager0.asst105157
    Component               Type          Up Status     In Sync Status
================================================================================
1   home                    OC4J          Down          True
2   HTTP_Server             HTTP_Server   Down          True
3   OC4J_EM                 OC4J          Down          True
4   OC4J_EMPROV             OC4J          Down          True

At this point http://asst105157:4889/em/console/ will return an error use following command to start all OMS components

C:\oracle\product\10.2.0\oms10g\dcm\bin>dcmctl start
Current State for Instance:EnterpriseManager0.asst105157
    Component               Type          Up Status     In Sync Status
================================================================================
1   home                    OC4J          Up            True
2   HTTP_Server             HTTP_Server   Up            True
3   OC4J_EM                 OC4J          Up            True
4   OC4J_EMPROV             OC4J          Up            True

For listing OMS components and Status use “dcmctl getstate” else “dcmctl listcomponents”

See

1. Download Software and upload zip at server. You will need 10.2.0.1 else 10.2.0.2 base release before you upgrade to 10.2.03/10.2.0.4
2. You will also need an X windows environment like cygwin installed on your desktop for remote install.  You can also carry a silent install with no X windows interface.
3. Let’s setup cygwin on client window machine

Goto c:\cygwin folder and fire Cygwin.bat . At $ prompt enter “startx”

You will receive a new window like below. Now enter “xhost + <hostname/IP address>” to allow server connection.

I entered “xhost +” to allow connection from any machine

Start Command Prompt on your desktop and locate IP address using “ipconfig”

On Server export DISPLAY=<IP Address>:0.0  and start “runInstaller”

You will see Oracle Installer window like below. If there are errors returned at telnet console , fix them before you proceed.

If your OS is running a cluster file system then Oracle will prompt for a clustered node install

Wait until “Agent Configuration Assistant” return a SUCCESS

Run “emctl status agent” and make sure last line says “Agent is Running and Ready”

Login at grid control and click on “hosts” tab. Here I could locate my new host

Click on Targets to see list of databases at new Host

Another click at “Databases” should show number of active Oracle Instances at Node

See number displayed against “Targets Not Configured” . You will need to configure DBSNMP oracle user for Grid monitoring.

Problem Statement I am configuring Oracle Agent for a grid console. My system is HP TRUE64 and /opt/oracle/Agent10g is a clustered file system like QFS/GFS/OCFS. A clustered file system is shared between 2 nodes, in my case $ORACLE_HOME is shared between cluster1 and cluster2.

I could get number of standalone agents working except this one because it is on a clustered file system.

Agents are on a clustered file system so thinking each one of them need separate config files , I made 2 separate copies of most configuration files
– emd.properties
– /opt/oracle/product/agent10g/sysman/config/b64LocalCertificate.txt
-/opt/oracle/product/agent10g/sysman/config/b64InternetCertificate.txt
Also set a different directory on each server for
– emdWalletDest=/opt/oracle/product/agent10g/sysman/config/server
I could connect to Agent URL : http://Agent:3872/emd/main/ & Repository URL : http://oms:4889/em/upload/ successfully.

Result : When I start Agent on one machine , the agent running on other machine shuts down. I almost decided to reinstall individual agent but then came across oracle documentation  “Reconfiguring a Standalone Agent to an Oracle RAC Agent”
Reconfiguration of a standalone agent occurs when you want to configure this agent (with standalone configurations) as a Oracle RAC agent.
To reconfigure a standalone agent as a Oracle RAC agent, you must execute the agentca script with the following options:
<Agent_Home>/bin/agentca -f -c “node1,node2….” [-t -i oraInstloc -n CLUSTER_NAME ]

cluster1 (unknown)> ./agentca -f -c “cluster1 ,cluster2”

Stopping the agent using /opt/oracle/product/agent10g/bin/emctl  stop agent
Oracle Enterprise Manager 10g Release 10.2.0.2.0.
Copyright (c) 1996, 2006 Oracle Corporation.  All rights reserved.
Stopping agent … stopped.
Running agentca using /opt/oracle/product/agent10g/oui/bin/runConfig.sh ORACLE_HOME=/opt/oracle/product/agent10g ACTION=Configure MODE=Perform RESPONSE_FILE=/opt/oracle/product/agent10g/response_file RERUN=TRUE INV_PTR_LOC=/var/opt/oracle/oraInst.loc COMPONENT_XML={oracle.sysman.top.agent.10_2_0_1_0.xml}
Perform – mode is starting for action: Configure
You can see the log file: /opt/oracle/product/agent10g/cfgtoollogs/oui/configActions2008-11-27_04-54-03-PM.log
ERROR: Unable to convert from “UTF-8” to “ISO8859-1” for NLS!
ÿÿÿÿÿÿÿÿ
The inventory pointer is located at /var/opt/oracle/oraInst.loc
The inventory is located at /opt/oracle/oraInventory
‘UpdateNodeList’ was successful.
Starting the agent using /opt/oracle/product/agent10g/bin/emctl  start agent
Oracle Enterprise Manager 10g Release 10.2.0.2.0.
Copyright (c) 1996, 2006 Oracle Corporation.  All rights reserved.
Starting agent …………………………… started but not ready.

After creating clustered agent,  Cluster1 agent started OK. I could locate cluster1 under Targets on my grid control. But there is still an issue with cluster2 node

clsueter2(unknown)> ./emctl start agent
EM Configuration issue. /opt/oracle/product/agent10g/cluster2/sysman/config/emd.properties not found.

A closer look revealed that after converting agent into clustered config, default directories picked up for configuration are  different
For node 1 – $AGENT_HOME/cluster1/sysman/config/ & for node 2 $AGENT_HOME/cluster2/sysman/config/  changed from $AGENT_HOME/sysman/config/

I thought this could be resolved by 2 methods

• Copying the directory $AGENT_HOME/cluster1 as $AGENT_HOME/cluster2 , Edit “emd.properties“ and set EMD_URL as per machines

OK, I copied directories and started agent at cluster1, went OK. I then started agent at cluster2, went well as well but moments later Agent at Cluster 1 stopped. It revealed that they were sharing same port 3872 and only 1 agent could be active on that port. I then changed ports at emd.properties but still agents won’t start

cluster1/cluster2 > ./emctl start agent
Oracle Enterprise Manager 10g Release 10.2.0.2.0.
Copyright (c) 1996, 2006 Oracle Corporation.  All rights reserved.
Starting agent …………………………… failed.
Consult the log files in: /opt/oracle/product/agent10g/cluster1/sysman/log

I finally decided to : Run “agentca” at cluster2 node as well and oracle will create directory and files for node 2 at $AGENT_HOME/cluster2/sysman/config/

cluster1(unknown)> rm –Rf $AGENT_HOME/cluster2

Cluster2 (unknown)> ./agentca -f -c “cluster1, cluster2”

Cluster2 (unknown)> ./emctl status agent
Oracle Enterprise Manager 10g Release 10.2.0.2.0.
Copyright (c) 1996, 2006 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version     : 10.2.0.2.0
OMS Version       : 10.2.0.4.0
Protocol Version  : 10.2.0.2.0
—————————————————————
Agent is Running but Not Ready

Cluster2 (unknown)> ./emctl upload agent
Oracle Enterprise Manager 10g Release 10.2.0.2.0.
—————————————————————
upload failed with ret=-10

Run following steps to push agent in “Agent is Running and Ready” status

$ ./emctl stop agent

Delete all files at $AGENT_HOME/sysman/emd/upload and $AGENT_HOME/sysman/emd/state
export AGENT_HOME=/opt/oracle/product/agent10g/
echo $AGENT_HOME
rm –Rf $AGENT_HOME/sysman/emd/upload/*.*
rm –Rf $AGENT_HOME/sysman/emd/state/*.*

Since this is clustered install , directories will be different for each node
rm –Rf /opt/oracle/product/agent10g/<cluster 1>/sysman/emd/upload/*.*
rm –Rf /opt/oracle/product/agent10g/<cluster 1>/sysman/emd/state/*.*
rm –Rf /opt/oracle/product/agent10g/<cluster 2>/sysman/emd/upload/*.*
rm –Rf /opt/oracle/product/agent10g/<cluster 2>/sysman/emd/state/*.*

$ ./emctl clearstate agent
$ ./emctl unsecure agent
$ ./emctl start agent
$ ./emctl upload agent
$ ./emctl secure agent

cluster2(unknown)> ./emctl status agent
Oracle Enterprise Manager 10g Release 10.2.0.2.0.
Copyright (c) 1996, 2006 Oracle Corporation.  All rights reserved.
—————————————————————
Last Reload       : 2008-12-04 16:44:26
Last successful upload                       : 2008-12-04 16:46:11
Total Megabytes of XML files uploaded so far :     0.58
Number of XML files pending upload           :        2
Size of XML files pending upload(MB)         :     0.00
Available disk space on upload filesystem    :    19.55%
Last successful heartbeat to OMS             : 2008-12-04 16:45:33
—————————————————————
Agent is Running and Ready

Both systems are communicating with grid now

1. If you have a working EM grid console configuration, you can TAR a client configuration and clone it at other nodes.
2. Tar the working agent at node

$ cd /opt/oracle/product $ ls -l total 8 drwxr-xr-x  65 oracle9    oinstall      2048 Oct 10  2006 9.2.0 drwxrwx—   3 oracle9    oinstall        96 Nov 19 13:29 agent_10g $ tar –cvf agent.tar agent_10g

3. Set the ORACLE_HOME to /opt/oracle/product/agent10g< ?xml:namespace prefix = o />

$export ORACLE_HOME=/opt/oracle/product/agent10g

4. Run the installer as shown below –

$cd $ORACLE_HOME/oui/bin Or

$cd /opt/oracle/product/agent10g/oui/bin

$./runInstaller -clone -forceClone ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=agent10g -noconfig –silent

5. Run the root.sh as root user

/opt/oracle/product/agent10g/agent10g/root.sh

6. Amend the REPOSITORY_URL and emdWalletSrcUrl parameters in the <AGENT_HOME>/sysman/config/emp.properties configuration file to make the agent point to right grid server

7. Run the agent configuration assistant

$cd $ORACLE_HOME/bin or

$cd /opt/oracle/product/agent10g/bin

$./agentca -f

8. Secure the agent

$./emctl secure agent <password>

9. Start the agent, if not already

$./emctl start agent

I installed 10g grid console agent using  notes

Locate the Database SID from HPUNIX-/etc/oratab or SunOS-/var/opt/oratab file

Goto –>Main Grid Console  -> Click on Databases  -> Enter “Name of Database” in “Search” and click “GO”

Alternatively click on “Targets” to list available databases,listeners

Click on “Configure” button and enter details for DBSNMP user

Click on “TEST Connection” to see message “”The connection test was successful”