Use profileRegistry.xml file to locate name of profiles on system. File normally located at /opt/IBM/WebSphere/AppServer/properties

<?xml version=”1.0″ encoding=”UTF-8″?>
<profiles>
<profile isAReservationTicket=”false” isDefault=”true” name=”Dmgr” path=”/opt/IBM/WebSphere/AppServer/profiles/Profile61/Dmgr” template=”/opt/IBM/WebSphere/AppServer/profileTemplates/cell/dmgr”/>
<profile isAReservationTicket=”false” isDefault=”false” name=”Node” path=”/opt/IBM/WebSphere/AppServer/profiles/Profile61/Node” template=”/opt/IBM/WebSphere/AppServer/profileTemplates/cell/default”/>
</profiles>

Stop the application

Click on application and select UPDATE Scroll Down and hit NEXT Keep default options and hit next Select where you want top deploy this application. I have one Cluster and 2 HTTP servers so I have selected respective nodes in screen above. Click on finish to see the progress We can select Save here and then start an application but I will select ROLLOUT update here. Let’s see what happens next I used “ptree –a” to locate if I can see any STOP /START server commands but it didn’t show relevant details. The Application was also down during this process though HTTP server was up and running. Please wait until you see message “Application Rollout succeeded” Click on continue and navigate to application servers Once servers are up validate application is also up & running before releasing system to users.

Authentication – is the act of proving a certain user’s identity.
Authorization – is a process of granting access or giving permission to a user to perform certain tasks.

To perform these operations of Authentication and Authorization, Websphere needs ‘REGISTRY’.

Websphere supports three kinds of registries.

• Custom
• Operating System
• LDAP

Custom
A user provided class is used to implement the registry API

Operating System
User and group registry used by the host operating system.

LDAP
A registry that supports the Light Weight Directory Access Protocol.

---

Custom registry :  1. Create a registry file in the appropriate location. Ex: c:\fileregistry\

For users create: usersfile.registry

For groups create: groupfile.registry

To configure Websphere security on your WAS you have to perform 3 tasks.

1. Configure user registry
2. Configure LTPA [light weight third party authentication]
3. Enable security.

---

Note: LTPA is like a token, which passes between the browser request and the application server. It is used to provide an identity to the remote server, that the particular request is coming from a particular user.

The following are screen shots from the base edition.

Security -> Global security ->

In the user registries select custom registries; give the user id and password.

Here I have given admin/admin

Now click on custom properties, and give the location of the users.registry file, click apply and ok.

Similarly add groups.registry.

To configure the LTPA authentication mechanism, go to console security -> global security -> Authentication Mechanism ->

Here I have given administrator as the password.

You should apply and save the changes.

Now you are ready to enable the global security.

After saving it will lead to the console as follows:

Select to enable global security, uncheck java 2 security.

In the active protocol list there are two options:

1. CSI
2. CSI and SAS

In the active protocol list select CSI (Common Secure Interoperability protocol).

If you need back ward compatibility with the other versions of WAS select CSI and SAS.

For active authentication mechanism select LTPA.

For active user registry Select custom user registry and click apply.

Click Apply, then OK.

It’s OK if you received warnings.

Log out.

Stop the server.

then login again.

On the address bar, you will observe that you were redirected to a secured Http: environment:

To create a group of administrators who should login.

Go to System Administration -> console -> console groups and add

1. If you configure Apache for mod_status you can view how many connections are open, the bandwidth being used, and a bunch of other neat statistics.

Example  http://httpd.apache.org/server-status

2. If you’re using Apache2, then apache-top would be useful as it’s interactive and would obviously update in real time:

Example : http://www.fr3nd.net/projects/apache-top/

3. To see number of IP connections and IPs connected to port 80, use the following command.

$netstat -plan|grep :80|awk {‘print $5’}|cut -d: -f 1|sort|uniq -c|sort -nk 1

$ netstat -plan|grep :80 | wc -l  (Number of connections on http port 80)
17

$ netstat -plan|grep :9080 | wc -l  (Number of connections on JVM port 9080)
10

Same command could be used to locate connections on any TCP port

backupConfig.sh will create backup of your websphere configuration while restoeConfig.sh could be used to restore backup taken by backupConfig.sh.

$WAS_HOME/dmgr/bin/backupConfig.sh $HOME/websphere_backup.zip -nostop -logfile $HOME/backupConfig.lst

Parameters

Comparing Files is one of very common task as a DBA, System Administrator. There are tonnes of Oracle,Websphere,linux configuration files. Often I have to compare one server to another and locate changes between environments.

Recently one of my websphere server broke down. Despite my good efforts I couldn’t revive it so I had to restore it from a backup.  Then came the task to compare the websphere confiuration between good and bad. When I looked at $WAS_HOME/bin/backupconfig , it backed up more than 400 files and carrying one to one comparison is no way possible.  I used following script to locate the difference.

#!/usr/bin/perl
# file_compare.pl
# Purpose: compare two files and show differences
# usage: file_compare.pl filename1 filename2

use strict;
use warnings;

my $file1 = shift or die “filename missing \n”;
my $file2 = shift or die “filename missing \n”;

open (FILE1, “< $file1”) or die “Can not read file $file1: $! \n”;
my @file1_contents = <FILE1>; # read entire contents of file
close (FILE1);

open (FILE2, “< $file2”) or die “Can not read file $file2: $! \n”;
my @file2_contents = <FILE2>; # read entire contents of file
close (FILE2);

my $length1 = $#file1_contents; # number of lines in first file
my $length2 = $#file2_contents; # number of lines in second file

if ($length1 > $length2) {
# first file contains more lines than second file
my $counter2 = 0;
foreach my $line_file1 (@file1_contents) {
chomp ($line_file1);

if (defined ($file2_contents[$counter2])) {
# line exists in second file
chomp (my $line_file2 = $file2_contents[$counter2]);

if ($line_file1 ne $line_file2) {
print “\nline ” . ($counter2 + 1) . ” \n”;
print “< $line_file1 \n” if ($line_file1 ne “”);
print “— \n”;
print “> $line_file2 \n\n” if ($line_file2 ne “”);
}
}
else {
# there is no line in second file
print “\nline ” . ($counter2 + 1) . ” \n”;
print “< $line_file1 \n” if ($line_file1 ne “”);
print “— \n”;
print “> \n”; # this line does not exist in file2
}
$counter2++; # point to the next line in file2
}
}
else {
# second file contains more lines than first file
# or both have equal number of lines
my $counter1 = 0;
foreach my $line_file2 (@file2_contents) {
chomp ($line_file2);

if (defined ($file1_contents[$counter1])) {
# line exists in first file
chomp (my $line_file1 = $file1_contents[$counter1]);

if ($line_file1 ne $line_file2) {
print “\nline ” . ($counter1 + 1) . ” \n”;
print “< $line_file1 \n” if ($line_file1 ne “”);
print “— \n”;
print “> $line_file2 \n” if ($line_file2 ne “”);
}
}
else {
# there is no line in first file
print “\nline ” . ($counter1 + 1) . ” \n”;
print “< \n”; # this line does not exist in file1
print “— \n”;
print “> $line_file2 \n” if ($line_file2 ne “”);
}
$counter1++; # point to next line in file1
}
}

Output

$perl compare_files.pl notworking.lst working.lst  | more

line 1
< 4     notworking/Cell/pmirm.xml
—
> 4     working/Cell/pmirm.xml
line 2
< 4     notworking/Cell/resources-pme.xml
—
> 4     working/Cell/resources-pme.xml
line 3
< 32    notworking/Cell/resources.xml
—
> 32    working/Cell/resources.xml

You should check if application server process is running to determine a crash. To do this, you need to know process ID of application server. You can find process ID at server name.pid file in:

<WAS_install_root>/profiles/<profile>/logs/<server> For Exa : /opt/IBM/WebSphere/AppServer/profiles/Profile01/dmgr/logs/dmgr/dmgr.pid

Open the <server_name>.pid file in a text editor. The four-digit number is a process ID. You can use appropriate operating system command to check if process is actively running. If it’s not running, then you have a crash.

What is a Thread Dump? (Java Core Dumps) javacore.<PID><TIME>.txt

A thread dump is a dump of the stacks of all live threads. Thus useful for analysing what an app is up to at some point in time, and if done at intervals handy in diagnosing some kinds of ‘execution’ problems (e.g. thread deadlock).

When to generate ? : If you get unexplained server hangs under WebSphere, you can obtain, from the WebSphere server, a thread dump to help diagnose the problem.
In the case of a server hang, you can force an application to create a thread dump.

If an application server spontaneously dies, look for a file. The JVM creates the file in the product directory structure, with a name like javacore[number].txt

What is a heap dump? heapdump.<PID><TIME>.phd

A heap dump is a “binary dump” of the full memory the JVM is using, and is for example useful if you need to know why you are running out of memory – in the heap dump you could for example see that you have one billion User objects, even though you should only have a thousand, which points to a memory retention problem.

When to generate? : Memory leaks in the Java heap produce java.lang.OutOfMemoryError exceptions in log files. However, not all out-of-memory errors are caused by Java heap memory leaks. Out-of-memory errors can also be caused by the following conditions:
Java heap fragmentation. This fragmentation occurs when no contiguous chunk of free Java heap space is available from which to allocate Java objects. Various causes for this problem exist, including the presence of pinned or dosed objects or because of the repeated allocation of large objects.
Memory leaks in native heap. This problem occurs when a native component, like DB2 connections, is leaking.

How to create Thread Dumps (Java Core Dumps)/Heap Dumps using wsadmin.sh

1. Navigate to cd <WAS_ROOT>/profiles/<PROFILE_NAME>/bin/

2. Connect to deployment manager using wsadmin script
wsadmin.sh <DMGR_HOST> <PORT> -conntype SOAP -username <USERNAME> -password <PASSWORD>

3. Set object variable
wsadmin> set jvm [$AdminControl completeObjectName type=JVM,process=<JVM_NAME>,node=<NODE_NAME>,*]

4. Create HeapDump:

wsadmin>$AdminControl invoke $jvm generateHeapDump
/opt/IBM/WebSphere/AppServer/profiles/Profile01/Node01/./heapdump.20100202.121506.27816.0001.phd

5. Create ThreadDump:

wsadmin>set jvm [$AdminControl completeObjectName type=JVM,process=member2,*]

wsadmin>$AdminControl invoke $jvm dumpThreads

6. Heap or thread dump will be saved to <WAS_ROOT>/profiles/<PROFILE_NAME>/ directory with with respective naming convention

Create Thread dumps using “kill -3” command

Add following settings:
Navigated to: Servers > Application Servers > Server1 (or the name of  the server to get a heap dump) > Process Definition > Environment Entries

Then set following properties:
IBM_HEAPDUMP = true
IBM_HEAP_DUMP = true
IBM_JAVA_HEAPDUMP_TEXT=true
IBM_HEAPDUMP_OUTOFMEMORY=false
JAVA_DUMP_OPTS=ONANYSIGNAL(JAVADUMP[5],HEAPDUMP[5])

Here export JAVA_DUMP_OPTS=”ONANYSIGNAL(JAVADUMP[n],HEAPDUMP[m])”
– n is the maximum number of javacores that can be generated, and
– m is the maximum number of heapdumps that can be generated

export JAVA_DUMP_OPTS=”ONANYSIGNAL(JAVADUMP[5],HEAPDUMP[5])”
A kill -3 to the java process will generate a maximum of 5 javacore and 5 heapdump files.

Now using “kill -3 <AppServer PID>” should create a HeapDump & ThreadDump

Basic Definitions:

WebSphere architectures contain one or more computer systems, which are referred to in WebSphere terminology as nodes. Nodes exist within a WebSphere cell. A WebSphere cell can contain one node on which all software components are installed or multiple nodes on which the software components are distributed.

A typical WebSphere cell contains software components that may be installed on one node or distributed over multiple nodes for scalability and reliability purposes. These include the following:

• A Web server that provides HTTP services
• A database server for storing application data
• WebSphere Application Server (WAS) V5

HTTP server
The HTTP server, more typically known as the Web server, accepts page requests from Web browsers and returns Web page content to Web browsers using the HTTP protocol. Requests for Java servlets and JavaServer Pages (JSPs) are passed by the Web server to WebSphere for execution. WebSphere executes the servlet or JSP and returns the response to the Web server, which in turn forwards the response to the Web browser for display.

WebSphere V5 supports numerous Web servers such as Apache, Microsoft IIS, Netscape and Domino. However, WebSphere has the tightest integration with Domino because IBM provides single sign-on capabilities between WebSphere and Domino.

WebSphere plug-in
The WebSphere plug-in integrates with the HTTP Server and directs requests for WebSphere resources (servlets, JSPs, etc.) to the embedded HTTP server (see below). The WebSphere plug-in uses a configuration file called plugin-cfg.xml file to determine which requests are to be handled by WebSphere. As applications are deployed to the WebSphere configuration, this file must be regenerated (typically using the Administration Console) and distributed to all Web servers, so that they know which URL requests to direct to WebSphere. This is one of the few manual processes that a WebSphere administrator must do to maintain the WebSphere environment.

Application server
The application server provides a run-time environment for J2EE applications (supporting servlets, JSPs, Enterprise JavaBeans, etc.). A node can have one or more application server processes. Each application server runs in its own runtime environment called a Java Virtual Machine (JVM). The JVM provides complete isolation (crash protection) for individual applications servers.

Application database
WebSphere applications such as IBM’s commerce and portal products, as well as applications you create yourself, use a relational database for storing configuration information and data. WebSphere V5 ships with the Cloudscape database and supports a wide range of database product, including the following:

• IBM DB2
• Informix
• Oracle
• SQL Server
• Sybase

Administration console
The administration console provides a Web-based interface for managing a WebSphere cell from a central location. The administration console can be used to change the configuration of any node within the cell at run-time. Configuration changes are automatically distributed to other nodes in the cell.

Cell:

A Cell is a virtual unit that is built of a Deployment Manager and one or more nodes.

The Deployment Manager is a process (in fact it is an special WebSphere instance) responsible for managing the installation and maintenance of Applications, Connection Pools and other resources related to a J2EE environment. It is also responsible for centralizing user repositories for application and also for WebSphere authentication and authorization.

The Deployment Manager communicates with the Nodes through another special WebSphere process, the Node Agent.

The Node is another virtual unit that is built of a Node Agent and one or more Server instances.

The Node Agent it the process responsible for spawning and killing server processes and also responsible for configuration synchronization between the Deployment Manager and the Node. Extra care must be taken when changing security configurations for the cell, since communication between Deployment Manager and Node Agent is ciphered and secured when security is enabled, Node Agent needs to have configuration fully resynchronized when impacting changes are made to Cell security configuration.

Servers are regular Java process responsible for serving J2EE requests (eg.: serving JSP/JSF pages, serving EJB calls, consuming JMS queues, etc).

Clusters

And to finish, Clusters are also virtual units that groups Servers so resources added to the Cluster are propagated to every Server that makes up the cluster, this will in fact affect usually more than a single Node instance.