Click on relevant Tools to see working example

Read more…

This morning I noticed one of my JVM (managed Node) log file native_stderr.log was over grown to 4GB. “tail -f native_stderr.log” was scrolling pages continuously indicating some issue with JVM garbage collection.

In Web applications, memory utilization can impact system performance significantly. One of the most common memory problems is memory leak, which causes severe performance degradation. In theory, memory leaks should not happen in Java™ because it has Garbage Collection (GC). However, GC only cleans up unused objects that are not referenced anymore. Therefore, if an object is not used, but is still referenced, GC does not remove it, which leads to memory leaks. Beside memory leaks, other memory problems that you might encounter are memory fragmentation, large objects, and tuning problems. In many cases, these memory problems can cause the application server to crash. Many users first notice that application server performance gradually declines, and eventually crashes with OutOfMemory exceptions.

<af type=”nursery” id=”5465″ timestamp=”Aug 12 12:42:37 2010″ intervalms=”190.797″>
* type:
* id: The id represents how many times the gc was executed
* intervalms: The time in ms since last time gc was executed
* timestamp: time of gc

<minimum requested_bytes=”168″ />
The minimum represents the number of bytes that were requested and JVM couldnot allocate them so it had to trigger garbage collection cycle.

<time exclusiveaccessms=”0.116″ />
<nursery freebytes=”0″ totalbytes=”57302016″ percent=”0″ />
<tenured freebytes=”1327455816″ totalbytes=”1814672384″ percent=”73″ >
<soa freebytes=”1262687592″ totalbytes=”1742086144″ percent=”72″ />
<loa freebytes=”64768224″ totalbytes=”72586240″ percent=”89″ />
</tenured>
<gc type=”scavenger” id=”5465″ totalid=”5472″ intervalms=”192.634″>
<flipped objectcount=”50598″ bytes=”7341136″ />
<tenured objectcount=”109″ bytes=”12368″ />
<refs_cleared soft=”0″ weak=”0″ phantom=”0″ />
<finalization objectsqueued=”0″ />
<scavenger tiltratio=”85″ />
<nursery freebytes=”49737176″ totalbytes=”57366528″ percent=”86″ tenureage=”14″ />
<tenured freebytes=”1327434760″ totalbytes=”1814672384″ percent=”73″ >
<soa freebytes=”1262666536″ totalbytes=”1742086144″ percent=”72″ />
<loa freebytes=”64768224″ totalbytes=”72586240″ percent=”89″ />
</tenured>
<time totalms=”46.316″ />
</gc>
<nursery freebytes=”49735128″ totalbytes=”57366528″ percent=”86″ />
<tenured freebytes=”1327434760″ totalbytes=”1814672384″ percent=”73″ >
<soa freebytes=”1262666536″ totalbytes=”1742086144″ percent=”72″ /
<loa freebytes=”64768224″ totalbytes=”72586240″ percent=”89″ />
</tenured>
<time totalms=”48.323″ />
</af>

The af element has 3 main child elements first tenured element has data about the tenured memory position before gc then gc element represents data about what happened during gc, such as time spent in mark, sweep and compact phases, The second tenured element represents the position of tenured memory after gc.
The IBM Support assistance has IBM Pattern modeling and Analysis tool for Java Garbage collection tool that can be used to analyze the garbage collection.

In previous releases of WebSphere Application Server, when global security was enabled, both administrative and application security were enabled. In WebSphere Application Server V6.1, the concept of global security is split into administrative security and application security, of which each component can be enabled separately. Application security provides application isolation and requirements for authenticating users for the applications in your environment.

When global security is enabled, the application has to provide the right user name and password to be able to run the scripts. It could be achived by 2 ways , using 1)Remote Method Invocation (RMI) connector , or 2) a SOAP connector:

The sas.client.props (RMI) and the soap.client.props (SOAP) files are located in $PROFILE_ROOT/properties directory for each WebSphere Application Server profile:

1) Using RMI method with wsadmin

Amend following properties in sas.client.props file :

com.ibm.CORBA.loginSource=properties  — change from Prompt (Default)
com.ibm.CORBA.loginUserid=wsadmin
com.ibm.CORBA.loginPassword=wsadmin1234

wsadmin -conntype RMI -port 2809

2) Using SOAP Connector method

was61@properties]$ /opt/IBM/WebSphere/AppServer/profiles/Profile01/Node/bin/stopServer.sh  server_member1
ADMU0116I: Tool information is being logged in file
/opt/IBM/WebSphere/AppServer/profiles/Profile01/Node/logs/server_member1/stopServer.log
ADMU0128I: Starting tool with the Node profile
ADMU3100I: Reading configuration for server: server_member1
Realm/Cell Name: <default>
Username: wsadmin
Password:

To switch off username /password prompt while shutting down WAS services , edit $WAS_HOME/node/properties/soap.client.props

vi soap.client.props
com.ibm.SOAP.loginUserid=wsadmin
com.ibm.SOAP.loginPassword=wsadmin1234

Here is the result

[was61@properties]$ /opt/IBM/WebSphere/AppServer/profiles/Profile01/Node/bin/stopServer.sh server_member1
ADMU0116I: Tool information is being logged in file
/opt/IBM/WebSphere/AppServer/profiles/Profile01/Node/logs/server_member1/stopServer.log
ADMU0128I: Starting tool with the Node profile
ADMU3100I: Reading configuration for server: server_member1
ADMU3201I: Server stop request issued. Waiting for stop status.
ADMU4000I: Server server_member1 stop completed.

To switch off password prompt while shutting down dmgr you will need soap.client.props modified at $WAS_HOME/Dmgr/properties/

We have a Websphere 6.1 vertical cluster i.e 2 JVMs in a single physical machine with IBM HTTP server.

Problem : Users complained  Http server is not properly failing connections when one of the JVM is hung/not responding

Background : In clustered IBM WebSphere Application Server environments, the HTTP plug-in has the ability to provide failover in the event the HTTP plug-in is no longer able to send requests to a particular cluster member. By default, there are several conditions under which the HTTP plug-in will mark a particular cluster member down and failover client requests to another cluster member that is still able to receive connections. They are listed as follows:

• The HTTP plug-in is unable to establish a connection to a cluster member’s Application Server transport.
• The HTTP plug-in detects a newly connected socket that was prematurely closed by a cluster member during an active read or write.

There are several configurable settings in the plugin-cfg.xml that can be tuned to affect how quickly the HTTP plug-in will mark a cluster member down and failover to another cluster member.

Sample $HTTP_HOME/Plugins/config/Plugin-cfg.xml file

<ServerCluster CloneSeparatorChange=”false” GetDWLMTable=”false” IgnoreAffinityRequests=”true” LoadBalance=”Round Robin” Name=”server1_cluster” PostBufferSize=”64″ PostSizeLimit=”-1″ RemoveSpecialHeaders=”true” RetryInterval=”60″>
<Server CloneID=”14l574lkv” ConnectTimeout=”0″ ExtendedHandshake=”false” LoadBalanceWeight=”2″ MaxConnections=”-1″ Name=”Server1_Node01_ihs_member1″ ServerIOTimeout=”0″ WaitForContinue=”false”>
<Transport Hostname=”Server1″ Port=”9080″ Protocol=”http”/>
<Transport Hostname=”Server1″ Port=”9443″ Protocol=”https”>
<Property Name=”keyring” Value=”/opt/IBM/WebSphere/Plugins/config/ihs/plugin-key.kdb”/>
</Transport>
</Server>
<Server CloneID=”14l574mci” ConnectTimeout=”0″ ExtendedHandshake=”false” LoadBalanceWeight=”2″ MaxConnections=”-1″ Name=”Server1_Node01_ihs_member2″ ServerIOTimeout=”0″ WaitForContinue=”false”>
<Transport Hostname=”Server1″ Port=”9081″ Protocol=”http”/>
<Transport Hostname=”Server1″ Port=”9444″ Protocol=”https”>
<Property Name=”keyring” Value=”/opt/IBM/WebSphere/Plugins/config/ihs/plugin-key.kdb”/>
</Transport>
</Server>
<PrimaryServers>
<Server Name=”Server1_Node01_ihs_member1″/>
<Server Name=”Server1_Node01_ihs_member2″/>
</PrimaryServers>
</ServerCluster>

ConnectTimeout Setting:
It is possible to add an attribute to the Server element called ConnectTimeout, which makes the plug-in use a non-blocking connect. Setting ConnectTimeout to a value of 0 is equal to not specifying the ConnectTimeout attribute, that is, the plug-in performs a blocking connect and waits until the operating system times out. Set this attribute to an integer value greater than zero to determine how long the plug-in should wait for a response when attempting to connect to a server. A setting of 10 will mean that the plug-in waits for ten seconds to time out.

Without ConnectTimeout http server will have to wait for OS timeout interval which will run into several minutes. This setting is very similar to ORACLE RAC VIP settings where oracle uses VIPs to detect timeouts/failover quickly over using OS or network interval.

ServerIOTimeout:
The ServerIOTimeout attribute of a server element enables the HTTP plug-in to set a time out value, in seconds, for sending requests to and reading responses from a cluster member. If a value is not set for the ServerIOTimeout attribute, the HTTP plug-in, by default, uses blocked I/O to write request to and read responses from the cluster member until the TCP connection times out.

RetryInterval:
An integer specifying the length of time that should elapse from the time that a server is marked down to the time that the HTTP plug-in will retry a connection. The default is 60 seconds.

As you can see , both settings ConnectTimeout & ServerIOTimeout are not set in this environment. For additional details visit http://www-01.ibm.com/support/docview.wss?uid=swg21219808

Websphere writes formatted text log messages to SystemOut.log,SystemErr.log & startServer.log files.

Sample SystemOut.log

[14/08/11 07:06:54:204 BST] 0000000a ManagerAdmin  I   TRAS0017I: The startup trace state is *=info.
[14/08/11 07:06:54:683 BST] 0000000a ManagerAdmin  I   TRAS0111I: The message IDs that are in use are deprecated

[14/08/11 07:06:55:606 BST] 0000000a FileRepositor A   ADMR0010I: Document cells/server1_Cell/nodes/server1_Manager/node-metadata.properties is modified.

[14/08/11 07:06:57:823 BST] 0000000a ThreadPoolMgr W   WSVR0626W: The ThreadPool setting on the ObjectRequestBroker service is deprecated.

[21/05/10 10:02:56:240 BST] 00000012 TCPPort       E   TCPC0003E: TCP Channel TCP_5 initialization failed.  The socket bind failed for host * and port 9352.  The port may already be in use.
[21/05/10 10:02:56:244 BST] 00000012 TCPPort       E   TCPC0003E: TCP Channel TCP_5 initialization failed.  The socket bind failed for host * and port 9352.  The port may already be in use.

[15/08/11 03:49:59:333 BST] 0000003c SystemOut     O Debug options: file:/opt/IBM/WebSphere/AppServer/profiles/Profile01/dmgr/.options not found
[15/08/11 03:49:59:512 BST] 0000003c SystemOut     O Need to load org.eclipse.osgi.framework.internal.protocol.reference.Handler
[15/08/11 03:49:59:585 BST] 0000003c SystemOut     O Time to load bundles: 76

# Time Stamp: The first part of the log message in sample code is  [15/08/11 03:49:59:585 BST]. It is the time stamp when the message was written. The time stamp is formatted using the locale of the process and it is 24 hour time stamp with milli-second precision

# Thread ID : The next part in the log message is 00000012 /0000003c , which represents the thread id. The thread ID is an eight-character hexadecimal value that is generated from the hash code of the thread that issued the message

# Short name : The short name is the abbreviated name of the component that issued the message. This name is typically the class name of a WAS component and would be some other identifier for the application. here ThreadPoolMgr , FileRepositor , TCPPort etc

# Event Type: The event type is a one character field that indicates the type of the message. The possible values are
* F- Fatal message
* E- Error message
* W- Warning message
* A- Audit message
* I- Informational message
* C- Configuration message
* D- detail message
* O- Messages that are written directly to System.out by an application or server component
* R- Messages that are written directly to System.err by the user application or internal component.
* Z- Place holder to indicate type was not recognized

# Message identifier (TCPC0003E/WSVR0626W/ADMR0010I): The message identifier is a string that is nine characters in length and is in the form CCCC1234X. The first four characters indicate the WAS component that issues the message. The next four characters indicate the specific message that component is issuing. The last character indicates the severity of the message. Its value is either I- informational, W- Warning or E error.

# Message: The message is the data that is logged to the SystemOut.log by the component

Sample SystemErr.log

[21/05/10 10:06:22:772 BST] 0000002b SystemErr     R Warning: unable to load “pr3native” library (using path: /opt/IBM/WebSphere/AppServer/java/jre/bin:/opt/IBM/WebSphere/AppServer/bin::/usr/lib:/opt/IBM/WebSphere/AppServer/lib/WMQ/java/lib) Root cause: pr3native (Not found in java.library.path)
[21/05/10 10:07:43:844 BST] 00000036 SystemErr     R 21-May-2010 10:07:43 com.pega.pegarules.ra.priv
INFO: Attempting to connect to JMS queue for asynchronous task execution

Sample startServer.log

************* End Display Current Environment *************
[21/05/10 10:02:41:607 BST] 0000000a ManagerAdmin  I   TRAS0017I: The startup trace state is *=info.
[21/05/10 10:02:41:671 BST] 0000000a AdminTool     A   ADMU0128I: Starting tool with the dmgr profile
[21/05/10 10:02:41:673 BST] 0000000a AdminTool     A   ADMU3100I: Reading configuration for server: dmgr
[21/05/10 10:02:43:570 BST] 0000000a AdminTool     A   ADMU3200I: Server launched. Waiting for initialization status.
[21/05/10 10:03:04:866 BST] 0000000a AdminTool     A   ADMU3000I: Server dmgr open for e-business; process id is 902

The IBM service log
The IBM service log, referred to as the service log, is a binary file. It contains information written to System.out by the Application Server run time as well as special messages that contain extended service information. This extended service information has been useful to IBM WebSphere Service teams in solving complex problems. IBM has now created additional tools to decipher these binary service logs, such as the Log Analyzer in Application Server. However, it is best to solve as many problems as possible during the development phase, before deploying to a production Application Server environment.

The service logs (activity.log) are normally located at $WAS_HOME/Profiles/dmgr/logs & $WAS_HOME/Profile/Node/Logs

[was61@Profile01]$ pwd
/opt/IBM/WebSphere/AppServer/profiles/Profile01
[was61@Profile01]$ du -a | grep activity.log
724     ./dmgr/logs/activity.log
2052    ./Node/logs/activity.log

Using showlog to see contents of Service log (activity.log)
You can find the showlog tool inside the WebSphere/AppServer/bin folder. If you execute it without arguments it will print out help for that tool

[was61@Profile01]$ /opt/IBM/WebSphere/AppServer/bin/showlog ./dmgr/logs/activity.log | more
$LANG = en_GB
$CODESET = UTF-8
—————————————————————
ComponentId:     Application Server
ProcessId:       26053
ThreadId:        0000000a
ThreadName:      main
SourceId:        com.ibm.ws.management.AdminInitializer
ClassName:
MethodName:
Manufacturer:    IBM
Product:         WebSphere
Version:         Platform 6.1 [ND 6.1.0.31 cf311015.02]
ServerName:      Cell\_Manager\dmgr
TimeStamp:       2011-08-14 07:06:54.943000000
UnitOfWork:
Severity:        3
Category:        AUDIT
PrimaryMessage:  ADMN0015I: The administration service is initialized.
…………………..

1.  Edit the ${WAS_HOME}/systemApps/adminconsole.ear/deployment.xml file in a text editor.

2. Locate the xml statement <tuningParams xmi:id=”TuningParams_1088453565469″ maxInMemorySessionCount=”1000″ allowOverflow=”true” writeFrequency=”TIME_BASED_WRITE” writeInterval=”10″ writeContents=”ONLY_UPDATED_ATTRIBUTES” invalidationTimeout=”30″>

3. Change the invalidationTimeout value to the desired session timeout. The default is 30.

4. Save the ${WAS_HOME}/systemApps/adminconsole.ear/deployment.xml file.

5. Restart the deployment manager

In a distributed server environment, the administrative console is located in the deployment manager server, dmgr. In this case, the administrative console provides centralized administration of multiple nodes. Configuration changes are made to the master repository and pushed to the local repositories on the nodes by the deployment manager. In order for the administrative console to run, the dmgr server must be running. In order for the changes to the master repository to be pushed to the nodes, the node agents must also be running.

The file synchronization service is the administrative service responsible for keeping up to date the configuration and application data files that are distributed
across the cell. The service runs in the deployment manager and node agents, and ensures that changes made to the master repository will be propagated out to the nodes, as necessary. The file transfer system application is used for the synchronization process. File synchronization can be forced from an administration client, or can be scheduled to happen automatically.

During the synchronization operation, the node agent checks with the deployment manager to see if any files that apply to the node have been updated in the master repository. New or updated files are sent to the node, while any deleted files are also deleted from the node.

Synchronization is one-way. The changes are sent from the deployment manager to the node agent. No changes are sent from the node agent back to the deployment manager

You can configure the File Synchronization service from the WAS Admin Console like this->

System administration -> Node Agents -> $NodeAgent->Configuration -> File synchronization service -> Automatic synchronization

Options Available

Automatic Synchronization : Specifies whether to synchronize files automatically after a designated interval. When this setting is enabled, the node agent automatically contacts the deployment manager every synchronization interval to attempt to synchronize the node’s configuration repository with the master repository owned by the deployment manager. If the Automatic synchronization setting is enabled, the node agent attempts file synchronization when it establishes contact with the deployment manager. The node agent waits the synchronization interval before it attempts the next synchronization.

Startup Synchronization : Specifies whether the node agent attempts to synchronize the node configuration with the latest configurations in the master repository prior to starting an application server. The default is to not synchronize files prior to starting an application server. Enabling the setting ensures that the node agent has the latest configuration but increases the amount of time it takes to start the application server.

Use this feature if you want to log or capture “wsadmin” commands excuted by WAS internally. It can be very useful when you want to automate your tasks and don’t know exact wsadmin syntaxes to use.

A logging could be enabled by 2 ways:

View Administrative scripting command for last action:
* Every time you perform an action a “View Administrative scripting command for last action” link shows up on the right hand side, if you click on the link a dialog box will open with the wasadmin command that is executed to perform the action

I navigated to DMGR-> “Applications” ->  Enterprise Applications -> At right hand Side I can see “View administrative scripting command for last action”
A click at View administrative scripting command for last action shows:
Administrative Scripting Command
AdminApp.list()

Log command assistance commands:
* You can enable the option “Log command assistance commands” at DMGR -> System administration -> preference. Once you do that every time you perform action in WAS Admin command the equivalent command will get saved in the log file.

Enable command assistance notifications
Specifies whether to send Java Management Extensions (JMX) notifications that contain command assistance data from the administrative console. Enablement of the notifications allows integration with product tools such as the WebSphere Application Server Toolkit (AST) Jython editor. Enablement of this option is recommended for non-production environments only.
Default     false (cleared)

Log command assistance commands
Specifies whether to log all the command assistance wsadmin data to a file. This file is saved to ${LOG_ROOT}/server/commandAssistanceJythonCommands_user name.log , where:
* server is the server process where the console runs, such as dmgr or server1.
* user name is the administrative console user name.
Occasionally clean out the file to manage its growth.

After i turned the preference on, i went to WAS Admin console and stopped my JVM. Then i went to dmgr_profile_root/logs directory and i could see commandAssistanceJythonCommands_patilsa.log file like this.

[was61@dmgr]$ ls -lrt
-rw-r–r– 1 was61 was61     5 Aug 14 07:07 dmgr.pid
-rw-r–r– 1 was61 was61   138 Aug 15 03:51 commandAssistanceJythonCommands_patilsa.log
-rw-r–r– 1 was61 was61 38373 Aug 15 03:52 SystemOut.log

vi commandAssistanceJythonCommands_patilsa.log

# [15/08/11 03:56:33:300 BST] ApplicationDeployment
AdminApp.list()

# [15/08/11 03:57:22:059 BST] Server status feedback
AdminControl.invoke(‘WebSphere:name=server_member2,process=server_member2,platform=proxy,node=Server1_Node01,j2eeType=J2EEServer,version=6.1.0.31,type=Server,mbeanIdentifier=cells/Server1_Cell/nodes/Server1_Node01/servers/server_member2/server.xml#Server_1259674426040,cell=Server1_Cell,spec=1.0,processType=ManagedProcess’, ‘stop’)

# [15/08/11 03:59:50:263 BST] Application servers
AdminControl.invoke(‘WebSphere:name=NodeAgent,process=nodeagent,platform=common,node=Server1_Node01,diagnosticProvider=true,version=6.1.0.31,type=NodeAgent,mbeanIdentifier=NodeAgent,cell=Server1_Cell,spec=1.0’, ‘launchProcess’, ‘[server_member2]’, ‘[java.lang.String]’)

Use profileRegistry.xml file to locate name of profiles on system. File normally located at /opt/IBM/WebSphere/AppServer/properties

<?xml version=”1.0″ encoding=”UTF-8″?>
<profiles>
<profile isAReservationTicket=”false” isDefault=”true” name=”Dmgr” path=”/opt/IBM/WebSphere/AppServer/profiles/Profile61/Dmgr” template=”/opt/IBM/WebSphere/AppServer/profileTemplates/cell/dmgr”/>
<profile isAReservationTicket=”false” isDefault=”false” name=”Node” path=”/opt/IBM/WebSphere/AppServer/profiles/Profile61/Node” template=”/opt/IBM/WebSphere/AppServer/profileTemplates/cell/default”/>
</profiles>

The Web Server plug-in uses an XML configuration file to determine whether a request is for the Web Server of the application server. When a request reaches the Web Server, the URL is compared to those managed by the plug-in. If a match is found, the plug-in configuration file contains the information needed to forward the request to the web container using the web container inbound chain.

For example lets say you make a request to http://localhost/help/SessionAffinityServlet URL, so the Web Server Plugin will check /prhelp URL to find out how it is should be handled.  It will check if there is matching UriGroup element in plugin-cfg.xml

<UriGroup Name=”default_host_test_cluster_URIs”>
<Uri AffinityCookie=”JSESSIONID” AffinityURLIdentifier=”jsessionid” Name=”/util/*”/>
<Uri AffinityCookie=”JSESSIONID” AffinityURLIdentifier=”jsessionid” Name=”/web/*”/>
<Uri AffinityCookie=”JSESSIONID” AffinityURLIdentifier=”jsessionid” Name=”/help/*”/>
<Uri AffinityCookie=”JSESSIONID” AffinityURLIdentifier=”jsessionid” Name=”/sysmgmt/*”/>
</UriGroup>

In this case it knows that /help/* URL is for dynamic content, so next part is how to route it to correct server.?

It will now read value of Name attribute for UriGroup which is default_host_cluster1_URIs, and name of cluster “cluster1”. It will use these values find out virtual host and the cluster.

It can see the test_cluster has two servers  Server1_Node01_test_server_member1  and   Server1_Node01_test_server_member2 . From the cluster definition it can locate http and https port to forward request to either of the server. The cluster definition also says that the Load balancing algorithm is Round robin.

Sample complete Plugin-cfg.xml

<Config ASDisableNagle=”false” AcceptAllContent=”false” AppServerPortPreference=”HostHeader” ChunkedResponse=”false” FIPSEnable=”false” IISDisableNagle=”false” IISPluginPriority=”High” IgnoreDNSFailures=”false” RefreshInterval=”60″ ResponseChunkSize=”64″ VHostMatchingCompat=”false”>
<Log LogLevel=”debug” Name=”/opt/IBM/HTTPServer/Plugins/logs/ihs-prpc/http_plugin.log”/>
<Property Name=”ESIEnable” Value=”true”/>
<Property Name=”ESIMaxCacheSize” Value=”1024″/>
<Property Name=”ESIInvalidationMonitor” Value=”false”/>
<Property Name=”ESIEnableToPassCookies” Value=”false”/>
<VirtualHostGroup Name=”default_host”>
<VirtualHost Name=”*:9080″/>
<VirtualHost Name=”*:80″/>
<VirtualHost Name=”*:9443″/>
<VirtualHost Name=”*:5060″/>
<VirtualHost Name=”*:5061″/>
<VirtualHost Name=”*:443″/>
<VirtualHost Name=”*:9081″/>
<VirtualHost Name=”*:9444″/>
</VirtualHostGroup>
<ServerCluster CloneSeparatorChange=”false” GetDWLMTable=”false” IgnoreAffinityRequests=”true” LoadBalance=”Round Robin” Name=”test_cluster” PostBufferSize=”64″ PostSizeLimit=”-1″ RemoveSpecialHeaders=”true” RetryInterval=”60″>
<Server CloneID=”1411a9de8″ ConnectTimeout=”0″ ExtendedHandshake=”false” LoadBalanceWeight=”2″ MaxConnections=”-1″ Name=”Server1_Node01_test_server_member1″ ServerIOTimeout=”0″ WaitForContinue=”false”>
<Transport Hostname=”10.0.0.100″ Port=”9080″ Protocol=”http”/>
<Transport Hostname=”10.0.0.100″ Port=”9443″ Protocol=”https”>
<Property Name=”keyring” Value=”/opt/IBM/WebSphere/Plugins/config/ihs-prpc/plugin-key.kdb”/>
<Property Name=”stashfile” Value=”/opt/IBM/WebSphere/Plugins/config/ihs-prpc/plugin-key.sth”/>
</Transport>
</Server>
<Server CloneID=”1411a9dt5″ ConnectTimeout=”0″ ExtendedHandshake=”false” LoadBalanceWeight=”2″ MaxConnections=”-1″ Name=”Server1_Node01_test_server_member2″ ServerIOTimeout=”0″ WaitForContinue=”false”>
<Transport Hostname=”10.0.0.100″ Port=”9081″ Protocol=”http”/>
<Transport Hostname=”10.0.0.100″ Port=”9444″ Protocol=”https”>
<Property Name=”keyring” Value=”/opt/IBM/WebSphere/Plugins/config/ihs-prpc/plugin-key.kdb”/>
<Property Name=”stashfile” Value=”/opt/IBM/WebSphere/Plugins/config/ihs-prpc/plugin-key.sth”/>
</Transport>
</Server>
<PrimaryServers>
<Server Name=”Server1_Node01_test_server_member1″/>
<Server Name=”Server1_Node01_test_server_member2″/>
</PrimaryServers>
</ServerCluster>
<UriGroup Name=”default_host_test_cluster_URIs”>
<Uri AffinityCookie=”JSESSIONID” AffinityURLIdentifier=”jsessionid” Name=”/prdbutil/*”/>
<Uri AffinityCookie=”JSESSIONID” AffinityURLIdentifier=”jsessionid” Name=”/prweb/*”/>
<Uri AffinityCookie=”JSESSIONID” AffinityURLIdentifier=”jsessionid” Name=”/prhelp/*”/>
<Uri AffinityCookie=”JSESSIONID” AffinityURLIdentifier=”jsessionid” Name=”/prsysmgmt/*”/>
</UriGroup>
<Route ServerCluster=”test_cluster” UriGroup=”default_host_test_cluster_URIs” VirtualHostGroup=”default_host”/>
<RequestMetrics armEnabled=”false” loggingEnabled=”false” rmEnabled=”false” traceLevel=”HOPS”>
<filters enable=”false” type=”URI”>
<filterValues enable=”false” value=”/snoop”/>
<filterValues enable=”false” value=”/hitcount”/>
</filters>
<filters enable=”false” type=”SOURCE_IP”>
<filterValues enable=”false” value=”255.255.255.255″/>
<filterValues enable=”false” value=”254.254.254.254″/>
</filters>
<filters enable=”false” type=”JMS”>         <filterValues enable=”false” value=”destination=aaa”/>
</filters>
<filters enable=”false” type=”WEB_SERVICES”>
<filterValues enable=”false” value=”wsdlPort=aaa:op=bbb:nameSpace=ccc”/>
</filters>
</RequestMetrics>
</Config>

Stop the application

Click on application and select UPDATE Scroll Down and hit NEXT Keep default options and hit next Select where you want top deploy this application. I have one Cluster and 2 HTTP servers so I have selected respective nodes in screen above. Click on finish to see the progress We can select Save here and then start an application but I will select ROLLOUT update here. Let’s see what happens next I used “ptree –a” to locate if I can see any STOP /START server commands but it didn’t show relevant details. The Application was also down during this process though HTTP server was up and running. Please wait until you see message “Application Rollout succeeded” Click on continue and navigate to application servers Once servers are up validate application is also up & running before releasing system to users.

1. Navigate to the /bin-directory of your IHS-Installation $IHS_HOME

2. execute ./ikeyman to open Key Management Tool

3. Use “Key Database File > Open” to open your password-protected Key-Database

4. After Key-Database is loaded switch to “Personal Certificate Requests” (under “Key database content”).

5. Click New and fill out the certificate request dialog. Depending on your CA-Provider (RapidSSL here) you may need to fill out the dialog in a special way (VeriSign demands the common name to be the domain)

6. Click “OK” to save the certificate request in a file

7. Now you need to provide content of certificate request file to your Ceritifcate Authority (e.g.: RapidSSL). You will receiving a new certificate file from them.

8. Once you have received certificate switch back to ikeyman->”Personal Certificates” (under “Key database content”)

9. Click Receive and navigate to certificate file. Click Ok to import certificate file.

10. Open httdp.conf File of your IHS and replace SSL-Cert-Name (name will be displayed after the import of new certificate in iKeyman).
Usally a SSL-Cert is definded within a virtual host:
Example:
<VirtualHost “ip-adress”:443>
ServerName www.test.com
SSLEnable
SSLClientAuth 0
SSLServerCert ihssslcert
AllowEncodedSlashes On

DocumentRoot /usr/IBM/HTTPServer/www-doc-root/
</VirtualHost>

11. Restart the IHS-Server
$/opt/IBM/HTTPServer/bin/apachectl stop
$ps -ef | grep httpd (Should return NO results)

/opt/IBM/HTTPServer/bin/apachectl start

$ ps -ef | grep httpd (Should return results like below)
root 13608 1 0 16:06 ? 00:00:00 /opt/IBM/HTTPServer/bin/httpd -d /opt/IBM/HTTPServer -k start
nobody 13610 13608 0 16:06 ? 00:00:00 /opt/IBM/HTTPServer/bin/httpd -d /opt/IBM/HTTPServer -k start
nobody 13611 13608 0 16:06 ? 00:00:00 /opt/IBM/HTTPServer/bin/httpd -d /opt/IBM/HTTPServer -k start
nobody 13612 13608 0 16:06 ? 00:00:00 /opt/IBM/HTTPServer/bin/httpd -d /opt/IBM/HTTPServer -k start

Check the $IHS_HOME/logs/error_log file to see any SSL handshake errors

Clusters

Logical grouping of application servers is called clusters. Instead of installing an application on individual server we can install it on a cluster to automatically deploy application on each application cluster.

Vertical cluster: When cluster members are defined on the same physical machine
Horizontal cluster: When the cluster members are defined on different physical machines

Plugin-cfg.xml

This file contains the necessary information to help the plugin module on  how to work load manage the requests.

Here we will look at a Vertical cluster:

Connect to the deployment manager admin console. Servers -> application servers -> server1.

We are using this application server1 as the template for the cluster we  create. All the cluster members will be identical to this application server.

The following applications will be deployed on the cluster and load balanced by cluster for performance and availability.

Click servers -> clusters -> new ->

Prefer local; it will improve performance in the clustered environment.

Now select the cluster members.

Note: select the node on which the previous cluster you have created.

Select generate unique http ports.

And click add member.

See the summary details.

Click finish, save the configuration when prompted. When you save, the following screen is

To verify cluster configuration and plugin work load management:

Deployment manager admin console -> servers -> clusters

Click on servers -> application servers ->

Now you see the cluster members you have created.

Check the cluster member and the names of the cluster and understand.

Select the cluster and click the start button to start the cluster.

Select servers -> application servers -> and check whether the cluster servers are started.

Now expand servers -> select web servers -> and click generate plug-in to generate the plugin representing the cluster environment.

And bounce the server once.

Now issue the following command in the browser and see which server is supplying the request.

For this you need to check the ServletContextAttributes

Websphere Silent installation consists oof 3 stages. You need to successfully complete all of these.

1. Base installation
2. IBM Http server
3. Plugin Software installation

The command “netstat –a” is used to find out the port numbers which are being used.

Silent Installation of Base

Go to the software 6.0 base/WAS. In the WAS console there will be a file named responsefile.base. Open it using editor.

Hints: node name can be any name

Check the port numbers, it should not be repeated.

The host name happens to be the name of the computer

-W nodehostandcellnamepanelInstallWizardBean.hostName=”rk”,

Prepare the silent response file as follows.

Now go to the command prompt and to the software where home is installed and type the following command:

WAS >install –options “the response file address.txt” –silent

Now in the log file of the logs folder, the following message should appear:

com.ibm.ws.install.ni.ismp.actions.ISMPLogSuccessMessageAction, msg1, INSTCONFSUCCESS

Also check that in the program files, entry of the Application server base is also Added V6.

Silent install of IBM HTTP SERVER:

Locate response file .6.0/base/response file. Now open  response file, and  copy the required commands from the response file, and save the response file.

Here, I have changed the port numbers, so be careful with the port numbers. If the port numbers are being used, then there will be a problem in the installation.

\Go to the command prompt and issue the following command:

As per location mentioned in silent response file , check whether the server is being installed or not. In install folder, check for the following file “insv_install”.

In the insv_install file, the following message should appear: (%DATE%), install, VerifyInstall, msg1, INSTCONFSUCCESS

Also check in the program files whether the new entry of IBM http server is made.

Plugin Software Silent Installation

Go to the plugin folder, find the response file, and create your response file similarly.

For the location of the Application server, be careful while selecting the location. For example in the silent installation, I had used this following location of the server, so I am providing the same for the plugin software installation.

-W websphereLocationWizardBean.wasExistingLocation=”C:\Program Files\IBM\Silent\WebSphere\AppServer”

WAS MACHINE host name is the system name.

-P pluginSettings.wasMachineHostName=”rk”

Also check the port number that is being used here. I am using the same port number that I have used earlier for the silent installation of the server. The response file is as follows:

Go to the command prompt and go to the location of the software, and type the command as follows:

Correspondingly check at the location mentioned in the response file whether the plugin is being installed or not. Now go to the install folder in the specified location as follows:

Check the text file install-HSPlugin. It must say the install complete.

By federating nodes to the deployment manager we can administer all the application servers through one deployment manager console.

1. You can add a new node through the admin console (explained below) &
   
2. Using addNode command at command prompt

TO FEDERATE:

Log on to the admin console of the application server Go to servers -> Application Servers -> server -> ports And note the soap connector address (here 8880).

Communications between the node agent and deployment manager takes place through this port.

Log on to the WAS admin:

JUST FOR your information to check what is present in the cell, Go to admin console -> cell -> Local Topology

In the cell you will note that only dmgr is part of the cell as of now

[Before federation]

System administration -> nodes -> add node -> managed node

Check the message whether the node is successfully federated.

Now go to the appserver profile root/logs

And check the log file addNode.txt

See the message which the node federated successfully.

Go to application server profile root/logs/nodeagent/logs

And check for systemOut.log and the message.

The configuration synchronization completed successfully.

To verify federation from the admin console:

System administration -> nodes -> check the node whether it is synchronized.

Start the application server.

Check the applications if they are running.

Go to node agents; Verify that a node agent process is created on the node you federated.

Now you can see the node added.

Note: Till now we have seen federating a node from deployment manager’s admin console. We can also federate a node from the command prompt of application server’s command prompt.

Go to console Enterprise Applications -> install -> location of the application

Here, we are using the lab files (An J2ee Application) that are located in our system. Click on next. Click on next. NOW, select the modules and select the application servers where you want to install the particular application so that the corresponding modules are installed on that particular servers. Click on next Now select the database that you want to use. Step 4 Each none message driven bean in the application should be bound to the Java naming and directory interface. Step 5 In the step 5 same screen only, You are mapping the modules to the entity beans. Now you have to specify the jndi name STEP 6 In this step, you have to map the modules to the cmp beans. Step 7 Step 8 Step 9 Now there are two roles that were specified in the application. There is a tab to look up users. Now you can select the tab and loop up for the users and you can assign the particular users for that module. When you click on the tab for look up users: Here in my system there are no users, but it will show a list of users where you can select. I have selected every one in the previous screen. Step 10 Select the role. This is the summary of the settings that you have made. And click Finish.

---

Now select the application and start.

---

For command prompt installation:

---

Go to wsadmin. Now at wsadmin bin Wsadmin>bin Note: If you place the .war file at the bin, then it is OK. OR ELSE, you have to give the location of the war file. Wsadmin>bin>$AdminApp installInteractive webspherebank.ear And follow the suggestions as per your requirement. And after successful installation save by $AdminConfig save

---

To configure the database

---

Jdbc providers -> select serverscope Select new. And select as follows: Give the name: Select data sources -> new Click new, and provide the address of the database and name of the jndi.

Authentication – is the act of proving a certain user’s identity.
Authorization – is a process of granting access or giving permission to a user to perform certain tasks.

To perform these operations of Authentication and Authorization, Websphere needs ‘REGISTRY’.

Websphere supports three kinds of registries.

• Custom
• Operating System
• LDAP

Custom
A user provided class is used to implement the registry API

Operating System
User and group registry used by the host operating system.

LDAP
A registry that supports the Light Weight Directory Access Protocol.

---

Custom registry :  1. Create a registry file in the appropriate location. Ex: c:\fileregistry\

For users create: usersfile.registry

For groups create: groupfile.registry

To configure Websphere security on your WAS you have to perform 3 tasks.

1. Configure user registry
2. Configure LTPA [light weight third party authentication]
3. Enable security.

---

Note: LTPA is like a token, which passes between the browser request and the application server. It is used to provide an identity to the remote server, that the particular request is coming from a particular user.

The following are screen shots from the base edition.

Security -> Global security ->

In the user registries select custom registries; give the user id and password.

Here I have given admin/admin

Now click on custom properties, and give the location of the users.registry file, click apply and ok.

Similarly add groups.registry.

To configure the LTPA authentication mechanism, go to console security -> global security -> Authentication Mechanism ->

Here I have given administrator as the password.

You should apply and save the changes.

Now you are ready to enable the global security.

After saving it will lead to the console as follows:

Select to enable global security, uncheck java 2 security.

In the active protocol list there are two options:

1. CSI
2. CSI and SAS

In the active protocol list select CSI (Common Secure Interoperability protocol).

If you need back ward compatibility with the other versions of WAS select CSI and SAS.

For active authentication mechanism select LTPA.

For active user registry Select custom user registry and click apply.

Click Apply, then OK.

It’s OK if you received warnings.

Log out.

Stop the server.

then login again.

On the address bar, you will observe that you were redirected to a secured Http: environment:

To create a group of administrators who should login.

Go to System Administration -> console -> console groups and add

Work Load Management [WLM]: WLM means share the requests across multiple application servers.

Important terms

• Scalability
• Load Balancing
• High Availability
• Failover

WLM is implemented by using the clusters of application servers.

Uses of WLM

1. WLM provides fail over so that application availability is increased.
2. WLM optimizes the distribution of requests.

Logical grouping of applications is called a cluster.

Instead of installing an application on an individual server, Install it on a cluster so that the application is automatically deployed on each application server which is a member of the cluster.

Vertical cluster: when the cluster members are defined on the same machine.

Horizontal cluster: when the cluster members are defined on different machines. Also, a combination of vertical and horizontal clusters can be made within the same machine.

Vertical cluster architecture:

The application app1 is installed on the cluster. This application was installed on both the application servers automatically when we deploy the app1 on the cluster through the Deployment manager’s admin console.

The deployment manager and http server can be in the same machine as the cluster member or in different machines.

[For production it is recommended to be in different machines]

We need to generate the plugin-cfg.xml file for the cluster environment.
This file contains the necessary information to work load manage requests for the cluster members.

It depends on the factors like …

• HTTP session creation
• Load balance weight value in plug-in file.

Note: Cluster members will have only applications in common. Other attributes of the application servers in a cluster may differ.

JVM Logs:The JVM logs are created by redirecting the System.out and System.err streams of the JVM to independent log files. The System.out log is used to monitor the health of the running application server. The System.err log contains exception stack trace information that is used to perform problem analysis. One set of JVM logs exists for each application server and all of its applications. JVM logs are also created for the deployment manager and each node manager

Application servers -> click on server -> Troubleshooting -> Logging and tracing -> jvmlogs

• Log file Rotation tells about the size of the log file
• Maximum no. of historical files tell how many log files can be maintained.

Process Logs: The process logs are created by redirecting the standard out and standard error streams of a process to independent log files. Native code writes to the process logs. These logs can also contain information that relates to problems in native code or diagnostic information written by the JVM. One set of process logs is created for each application server and all of its applications. Process logs are also created for the deployment manager and each node manager.

IBM Service Logs:The IBM service log contains both the application server messages that are written to the System.out stream and special messages that contain extended service information that you can use to analyze problems. One service log exists for all Java virtual machines (JVMs) on a node, including all application servers and their node agent, if present. A separate activity log is created for a deployment manager in its own logs directory. The IBM Service log is maintained in a binary format. Use the Log Analyzer or Showlog tool to view the IBM service log.

Red Hat Linux provides a standardized interface to allow users to add scripts to start various processes during system initialization without requiring a user to log in to the system. This process consists of three steps:

1.  Create the start_server1 script.

Navigate to /opt/WebSphere/AppServer/bin directory and run following command. This command generates a script named start_server1.sh which is referenced by the actual initialization script.

./startServer.sh server1 -script -background  [ Note here server1 is name of a JVM]

2. Create the ibmhttpd and was scripts using the scripts below for reference and place them in the /etc/init.d directory. Make sure to set the executable flag in the scripts. The ibmhttpd script is the system initialization script that is used to automatically start the IBM HTTP Server at system initialization time. The was script is the system initialization script that is used to automatically start WebSphere Application Server .

Example of ibmhttpd script:

#!/bin/bash
#
# apache
#
# chkconfig: 345 85 15 — line says run this script in run level 5, with “start” at 85 and “stop” at 15 (so for startups, it’s done near the end of the startup process, and for shutdown, it’s done quite early
# description: Start up the Apache web server.
RETVAL=$?
APACHE_HOME=”/opt/IBMHttpServer”
case “$1″ in
start)
if [ -f $APACHE_HOME/bin/apachectl ]; then
echo $”Starting IBM Http Server”
$APACHE_HOME/bin/apachectl start
fi
;;
stop)
if [ -f $APACHE_HOME/bin/apachectl ]; then
echo $”Stopping IBM Http Server”
$APACHE_HOME/bin/apachectl stop
fi
;;
status)
if [ -f $APACHE_HOME/bin/apachectl ]; then
echo $”Show status of IBM Http Server”
$APACHE_HOME/bin/apachectl status
fi
;;
*)
echo $”Usage: $0 {start|stop|status}”
exit 1
;;
esac
exit $RETVAL

Example of a was autostart script:

#!/bin/bash
#
# apache
#
# chkconfig: 345 90 10
# description: Start up the WebSphere Application Server.
RETVAL=$?
WAS_HOME=”/opt/IBM/WebSphere/AppServer/profiles/Profile01″

case “$1″ in
start)
if [ -f $WAS_HOME/Node/bin/start_nodeagent.sh ]; then
echo $”Starting IBM WebSphere Node Agent and Application Server”
$WAS_HOME/dmgr/bin/startManager.sh
$WAS_HOME/Node/bin/start_nodeagent.sh
$WAS_HOME/Node/bin/start_server_member1.sh
$WAS_HOME/Node/bin/start_server_member2.sh
fi
;;
stop)
if [ -f $WAS_HOME/bin/stopServer.sh ]; then
echo $”Stop IBM WebSphere Application Server”
$WAS_HOME/Node/bin/stop_server_member1.sh
$WAS_HOME/Node/bin/stop_server_member2.sh
$WAS_HOME/Node/bin/stop_nodeagent.sh
$WAS_HOME/dmgr/bin/stopManager.sh
fi
;;
status)
if [ -f $WAS_HOME/bin/serverStatus.sh ]; then
echo $”Show status of IBM WebSphere Application Server”
$WAS_HOME/bin/serverStatus.sh server_member1
$WAS_HOME/bin/serverStatus.sh server_member2
$WAS_HOME/bin/serverStatus.sh nodeagent
fi
;;
*)
echo $”Usage: $0 {start|stop|status}”
exit 1
;;
esac
exit $RETVAL

3. Establish the ibmhttpd and was scripts as services in order to run them in the system initialization process. To do this, enter the following commands as root user:

chkconfig –add was
chkconfig –level 5 was on
chkconfig –add ibmhttpd
chkconfig –level 5 ibmhttpd

Please check service details using “chkconfig –list | grep httpd/was”

The chkconfig –add commands add the script entries into the services table and the chkconfig –level commands indicate the runlevels at which Red Hat should automatically run the scripts.

4. Test the autostart scripts by rebooting your system and verifying that the desired processes have started.

1. If you configure Apache for mod_status you can view how many connections are open, the bandwidth being used, and a bunch of other neat statistics.

Example  http://httpd.apache.org/server-status

2. If you’re using Apache2, then apache-top would be useful as it’s interactive and would obviously update in real time:

Example : http://www.fr3nd.net/projects/apache-top/

3. To see number of IP connections and IPs connected to port 80, use the following command.

$netstat -plan|grep :80|awk {‘print $5’}|cut -d: -f 1|sort|uniq -c|sort -nk 1

$ netstat -plan|grep :80 | wc -l  (Number of connections on http port 80)
17

$ netstat -plan|grep :9080 | wc -l  (Number of connections on JVM port 9080)
10

Same command could be used to locate connections on any TCP port

backupConfig.sh will create backup of your websphere configuration while restoeConfig.sh could be used to restore backup taken by backupConfig.sh.

$WAS_HOME/dmgr/bin/backupConfig.sh $HOME/websphere_backup.zip -nostop -logfile $HOME/backupConfig.lst

Parameters

In a Network Deployment environment, the deployment manager maintains the master repository for all of the WebSphere Application Server nodes and servers that it manages in the cell. Copies of the files that each node needs are replicated to that node by a process known as synchronization.

In a Network Deployment environment with two nodes,  All of the configuration files relevant to both Node01 and Node02 are kept in the master repository along with the  configuration files that are relevant to the deployment manager. Only those files that are relevant to Node01 are replicated to Node01, and only those files that are relevant to Node02 are replicated to Node02. Each node gets a copy of the serverindex.xml file for every other node because this contains connection information for the other nodes, that is host names and ports.

We  can set each node agent to perform automatic or manual synchronization and the interval at which each node agent will perform the synchronization. To set this in the administrative console,

select System administration → Node agents → nodeagent → File synchronization service.

We can manually initiate synchronization using the administrative console by selecting

System administration → Nodes, putting a check by the node that you wish to synchronize, and then clicking either Synchronize or Full Resynchronize.

You can also perform synchronization from the node agent using the syncNode.bat|sh script. You must stop the node agent to use this tool.

You should check if application server process is running to determine a crash. To do this, you need to know process ID of application server. You can find process ID at server name.pid file in:

<WAS_install_root>/profiles/<profile>/logs/<server> For Exa : /opt/IBM/WebSphere/AppServer/profiles/Profile01/dmgr/logs/dmgr/dmgr.pid

Open the <server_name>.pid file in a text editor. The four-digit number is a process ID. You can use appropriate operating system command to check if process is actively running. If it’s not running, then you have a crash.

What is a Thread Dump? (Java Core Dumps) javacore.<PID><TIME>.txt

A thread dump is a dump of the stacks of all live threads. Thus useful for analysing what an app is up to at some point in time, and if done at intervals handy in diagnosing some kinds of ‘execution’ problems (e.g. thread deadlock).

When to generate ? : If you get unexplained server hangs under WebSphere, you can obtain, from the WebSphere server, a thread dump to help diagnose the problem.
In the case of a server hang, you can force an application to create a thread dump.

If an application server spontaneously dies, look for a file. The JVM creates the file in the product directory structure, with a name like javacore[number].txt

What is a heap dump? heapdump.<PID><TIME>.phd

A heap dump is a “binary dump” of the full memory the JVM is using, and is for example useful if you need to know why you are running out of memory – in the heap dump you could for example see that you have one billion User objects, even though you should only have a thousand, which points to a memory retention problem.

When to generate? : Memory leaks in the Java heap produce java.lang.OutOfMemoryError exceptions in log files. However, not all out-of-memory errors are caused by Java heap memory leaks. Out-of-memory errors can also be caused by the following conditions:
Java heap fragmentation. This fragmentation occurs when no contiguous chunk of free Java heap space is available from which to allocate Java objects. Various causes for this problem exist, including the presence of pinned or dosed objects or because of the repeated allocation of large objects.
Memory leaks in native heap. This problem occurs when a native component, like DB2 connections, is leaking.

How to create Thread Dumps (Java Core Dumps)/Heap Dumps using wsadmin.sh

1. Navigate to cd <WAS_ROOT>/profiles/<PROFILE_NAME>/bin/

2. Connect to deployment manager using wsadmin script
wsadmin.sh <DMGR_HOST> <PORT> -conntype SOAP -username <USERNAME> -password <PASSWORD>

3. Set object variable
wsadmin> set jvm [$AdminControl completeObjectName type=JVM,process=<JVM_NAME>,node=<NODE_NAME>,*]

4. Create HeapDump:

wsadmin>$AdminControl invoke $jvm generateHeapDump
/opt/IBM/WebSphere/AppServer/profiles/Profile01/Node01/./heapdump.20100202.121506.27816.0001.phd

5. Create ThreadDump:

wsadmin>set jvm [$AdminControl completeObjectName type=JVM,process=member2,*]

wsadmin>$AdminControl invoke $jvm dumpThreads

6. Heap or thread dump will be saved to <WAS_ROOT>/profiles/<PROFILE_NAME>/ directory with with respective naming convention

Create Thread dumps using “kill -3” command

Add following settings:
Navigated to: Servers > Application Servers > Server1 (or the name of  the server to get a heap dump) > Process Definition > Environment Entries

Then set following properties:
IBM_HEAPDUMP = true
IBM_HEAP_DUMP = true
IBM_JAVA_HEAPDUMP_TEXT=true
IBM_HEAPDUMP_OUTOFMEMORY=false
JAVA_DUMP_OPTS=ONANYSIGNAL(JAVADUMP[5],HEAPDUMP[5])

Here export JAVA_DUMP_OPTS=”ONANYSIGNAL(JAVADUMP[n],HEAPDUMP[m])”
– n is the maximum number of javacores that can be generated, and
– m is the maximum number of heapdumps that can be generated

export JAVA_DUMP_OPTS=”ONANYSIGNAL(JAVADUMP[5],HEAPDUMP[5])”
A kill -3 to the java process will generate a maximum of 5 javacore and 5 heapdump files.

Now using “kill -3 <AppServer PID>” should create a HeapDump & ThreadDump