Generating Dump []

Moved to the page http://3rdstage.wikia.com/wiki/J2EE_Application_Tuning in my wiki as of 5th Oct. 2012.

Generating Heap Dump

• Getting a Heap Dump from getting started guide of Eclipse MAT

For IBM Java SDK, you can trigger dumps including.... For more refer : http://publib.boulder.ibm.com/infocenter/hctool/v1r0/topic/com.ibm.java.diagnostics.healthcenter.doc/topics/triggering.html

Generating Thread Dump

Sun JDK

The following is quoted from J2SE 5.0 Troubleshooting and Diagnostic Guide.

1.16 Ctrl-Break Handler

On Solaris or Linux the HotSpot VM will print a thread dump to the application's standard output if the Ctrl and \ keys are pressed. The thread dump consists of the thread stack for all Java threads in the VM. On Windows the equivalent key sequence is the Ctrl and Break keys. The thread dump doesn't terminate the application – it continues after the thread information is printed.
On Solaris and Linux a thread dump is also printed if the J2SE process receives a QUIT signal. So kill -QUIT <pid> causes the process with id <pid> to print a thread dump to its standard output.
In addition to the thread stacks, the ctrl-break handler also executes a deadlock detection algorithm. If any deadlocks are detected then it prints out additional information on each deadlocked thread.

IBM Java SDK

The following is quoted from IBM Java SDK Diagnostics Guide 6.

Chapter 25. Using Javadump

...

The signal for Linux, AIX and z/OS is SIGQUIT. Use the command kill -QUIT n to send the signal to a process with process id (PID) n. Alternatively, press CTRL+\ in the shell window that started Java.(CTRL+V on z/OS.)
The signal for AIX is SIGQUIT. Use the command kill -QUIT n to send the signal to a process with process id (PID) n. Alternatively, press CTRL+\ in the shell window that started Java.
In Windows, the dump is initiated by using CTRL+Break in the command window that started Java.
The signal for i5/OS is SIGQUIT. Use the command kill -QUIT n to send the signal to a process with process id (PID) n. The PID for a particular JVM can be found in the joblog for the job (JVAB56D: Java Virtual Machine is IBM Technology for Java. PID(x)), or using the "ps" command from qsh or an i5/OS PASE shell.
The JVM will continue operation after the signal has been handled.

Obtaining Clue on OOM(OutOfMemory) Error

OutOfMemoryError is one of the most terrible problems. To prevent it from occurring again, it is very important to identify the cause. The heap dump at the time of error maybe provide the most exact cause to the error. With JDK 1.5 or higher, you can use the following options with JVM to obtain the heap dump at the time when OutOfMemoryError ocurrs.

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=path-of-dump-to-generate

One thing that needs to be considered more is that the above options derive how much overhead or side effects. I don't have any idea about it, and so I have no confidence where the above options can be safely used in production environment. To confirm using in production environment, more thorough research is necessary.

HotSpot VM Tuning []

Moved to the page http://3rdstage.wikia.com/wiki/J2EE_Application_Tuning in my wiki as of 5th Oct. 2012.

Articles

• Tuning HotSpot Java virtual machines (Solaris & HP-UX) (from documentation of WebSphere 7.0)
• Sun HotSpot JVM tuning parameters (Solaris and HP-UX) (from documentation of WebSphere 7.0)
• My Favorite Hotspot JVM Flags

Identifying Current/Min/Max Size of Young and Old Generation

You can identify details of memory usage such as min./max. size of old generation, current size of eden space and current size young generation on a running VM instance using jstat command.

For example, the VM launched with the following option.

-Xms128m -Xmx256m -XX:MaxPermSize=256M -server -XX:+UseParallelGC -XX:NewRatio=2 -XX:SurvivorRatio=8

Shows the following information with jstat -gccapacity -h5 3680 5s.

C:\lang\jdk1.6\bin>jstat -gccapacity -h5 3680 5s
 NGCMN    NGCMX     NGC     S0C   S1C       EC      OGCMN      OGCMX       OGC         OC      PGCMN    PGCMX     PGC
    PC     YGC    FGC
 43648.0  87360.0  43648.0 4480.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
 103936.0     48     3
 43648.0  87360.0  43648.0 4480.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
 103936.0     48     3
 43648.0  87360.0  43648.0 4480.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
 103936.0     48     3
 43648.0  87360.0  43648.0 4480.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
 103936.0     48     3
 43648.0  87360.0  43648.0 4480.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
 103936.0     48     3
 NGCMN    NGCMX     NGC     S0C   S1C       EC      OGCMN      OGCMX       OGC         OC      PGCMN    PGCMX     PGC
    PC     YGC    FGC
 43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
 103936.0     49     3
 43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
 103936.0     49     3
 43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
 103936.0     49     3
 43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
 103936.0     49     3
 43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
 103936.0     49     3
 NGCMN    NGCMX     NGC     S0C   S1C       EC      OGCMN      OGCMX       OGC         OC      PGCMN    PGCMX     PGC
    PC     YGC    FGC
 43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
 103936.0     49     3
 43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
 103936.0     49     3
 43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
 103936.0     49     3
 43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
 103936.0     49     3
 43648.0  87360.0  43648.0 4352.0 4416.0  34752.0    87424.0   174784.0   116608.0   116608.0  16384.0 262144.0 103936.0
 103936.0     49     3

You can identify the above statistics in visual form using visualgc. visualgc is provided as a plugin of VisualVM.

References

• jvmstat home
• jstat reference
• visualgc reference
• Plugins of VisualVM

Using VisualVM with IBM JDK []

Moved to the page http://3rdstage.wikia.com/wiki/J2EE_Application_Tuning in my wiki as of 5th Oct. 2012.

You can use JMX to monitoring IBM JDK with VisualVM.
Java SE 5.0 introduced platform MBeanServer and IBM Java SDK 5.0 or higher naturally provide it. Platform MBeanServer includes various basic managed beans about memory, thread, garbage collection, class loading and so on^[1]. VisualVM can show such information with visual views.

Basically remote access of platform MBeanServer is not activated. You can enable it in several ways.
If you can control the start-up of the VM, you can enable remote access of platform MBeanServer by setting com.sun.management.jmxremote.por system property in command line to launch VM. If you are in position where you can control the system properties, you can programmatically load JMX connector server and expose it using RMI registry.

Method 1. Enabling built-in JMX agent using system properties

You can enable built-in JMX agent of Java runtime with the following system properties explicitly in start-up command of JVM.

-Dcom.sun.management.jmxremote.port=3333
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

With VisualVM, you should add new local or remote application using [File > Add JMX Connection] menu item. The connection string has the form of hostname:port such as localhost:3333 or 203.201.71.31:3333.

For more about built-in JMX agent, refer readings 2 below. If more secure setup for remote access to the platform MBeanServer is necessary, it would be really helpful.

Method 2. Exposing platform MBeanServer using JMXConnectorServer programmatically.

When you can not control the start-up command of the JVM, you can expose platform MBeanServer through JMXConnectorServer instance in your code. The JMXConnectorServer should be exposed using RMI registry.

The following code sample shows core steps, acquiring platform MBeanServer, creating JMXConnectorServer instance, creating RMI registry and starting JMXConnectorServer.

   ...

   //acquiring platform MBeanServer
   MBeanServer mbs = ManagementFactory.getPlatformMBeanServer(); 

   //creating JMXConnectorServer instance
   JMXServiceURL url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" 
         + InetAddress.getLocalHost().getHostAddress() + ":9999/server");
   JMXConnectorServer cs = JMXConnectorServerFactory.newJMXConnectorServer(url, null, mbs);
  
   //creating RMI registry
   Registry reg = LocateRegistry.createRegistry(9999);
  
   //starting JMXConnectorServer
   cs.start();

   ...

With VisualVM, you should add new local or remote application using [File > Add JMX Connection] menu item. The connection string has the form of service:jmx:rmi:///jndi/rmi://host-address:port/service such as service:jmx:rmi:///jndi/rmi://203.201.71.31:9999/server.
Now you can identify JVM argument or system properties, monitor memory or garbage collection, class loading statistics, thread in timeline, and so on with VisualVM.

If you want stop the remote access to the platform MBeanServer programmatically, just stop the JMXConnectorSever.

   ...

   //stoping JMXConnectorServer
   cs.stop();

   //unload RMI registry
   java.rmi.server.UnicastRemoteObject.unexportObject(reg,true);

   ...

As for me, the above code worked with IBM Java SDK 6.0 on Windows and IBM Java SDK 6.0 64bit on AIX 6.2.

Readings

1. java.lang.management package API
2. Monitoring and Management Using JMX
3. Connecting to JMX Agents Explicitly
4. Support for the Java Attach API of IBM Java v6 on AIX

JVM Tuning Foundation []

Moved to the page http://3rdstage.wikia.com/wiki/J2EE_Application_Tuning in my wiki as of 5th Oct. 2012.

Basic Glossary or Definition

Profiling

a form of dynamic program analysis that measures, for example, the usage of memory, the usage of particular instructions, or frequency and duration of function calls.

from Wikipedia

Debugging

a methodical process of finding and reducing the number of bugs, or defects, in a computer program or a piece of electronic hardware, thus making it behave as expected.

from Wikipedia

Tuning

the improvement of system performance.

from Wikipedia

Optimization

improving software to make it work more efficiently or use fewer resources.

from Wikipedia

Diagnosis

the identification of the nature and cause of anything.

from Wikipedia

Types of Diagnostic Information in IBM JVM

Javadump (Thread dump)

The Javadump is sometimes referred to as a Javacore or thread dump in some JVMs. This dump is in a human-readable format produced by default when the JVM terminates unexpectedly because of an operating system signal, an OutOfMemoryError exception, or when the user enters a reserved key combination (for example, Ctrl-Break on Windows®). It can also be generated by calling com.ibm.jvm.Dump.JavaDump() from inside the application. A Javadump summarizes the state of the JVM at the instant the signal occurred. Much of the content of the Javadump is specific to the IBM JVM.

Heapdump

The JVM can generate a Heapdump at the request of the user (for example by calling com.ibm.jvm.Dump.HeapDump() from inside the application) or (by default) when the JVM terminates because of an OutOfMemoryError exception. You can specify finer control of the timing of a Heapdump with the -Xdump:heap option. For example, you could request a Heapdump after a certain number of full garbage collections have occurred. The default Heapdump format (phd files) is not human-readable and you process it using available tools such as Heaproots.

System dump (Core dump)

System dumps (also known as core dumps on Linux® platforms) are platform-specific files that contain information about the active processes, threads, and system memory. System dumps are usually large. By default, system dumps are produced by the JVM only when the JVM fails unexpectedly because of a GPF (general protection fault) or a major JVM or system error. You can also request a system dump by calling com.ibm.jvm.Dump.SystemDump() from your application. You can use the -Xdump:system option to produce system dumps when other events occur

Garbage collection data

A JVM started with the -verbose:gc option produces output in XML format that can be used to analyze problems in the Garbage Collector itself or problems in the design of user applications. Numerous other options affect the nature and amount of Garbage Collector diagnostic information produced. See Garbage Collector diagnostics for more information.

Trace data

The IBM JVM tracing allows execution points in the Java code and the internal JVM code to be logged. The -Xtrace option allows the number and areas of trace points to be controlled, as well as the size and nature of the trace buffers maintained. The internal trace buffers at a time of failure are also available in a system dump and tools are available to extract them from a system dump. Generally, trace data is written to a file in an encoded format and then a trace formatter converts the data into a readable format. However, if small amounts of trace are to be produced and performance is not an issue, trace can be routed to STDERR and will be pre-formatted.

The above are cited from the diagnostic guide of IBM Java SDK which you can find from here.

Readings on Tuning []

Moved to the page http://3rdstage.wikia.com/wiki/J2EE_Application_Tuning in my wiki as of 5th Oct. 2012.

Readings on Tuning General with Java

• Java command line debug options
• JBossProfiler Guide

• Debugging with the Maven Jetty Plugin in Eclipse
• Uncovering Memory Leaks Using NetBeans Profiler
• Java memory leaks -- Catch me if you can (2006)
• How to Analyze Java Thread Dumps

Tuning HotSpot VM

• Tuning HotSpot Java virtual machines (Solaris & HP-UX) (from documentation of WebSphere 7.0)
• Sun HotSpot JVM tuning parameters (Solaris and HP-UX) (from documentation of WebSphere 7.0)
• My Favorite Hotspot JVM Flags

Tuning IBM Java SDK

• IBM User Guides for Java v6 on 64-bit AIX
• Diagnostics Guide for IBM SDK and Runtime Environment Java Technology Edition, Version 6

• IBM Monitoring and Diagnostic Tools for Java Manual
• IBM Monitoring and Diagnostic Tools for Java - Health Center
• Maximizing Java performance on AIX: Part 1: The basics
• Maximizing Java performance on AIX: Part 2: The need for speed
• Maximizing Java performance on AIX: Part 3: More is better
• Maximizing Java Performance on AIX: Part 4: What goes in
• Maximizing Java Performance on AIX: Part 5: References and Conclusion
• Disabling the JIT
• Enabling the JIT

Oracle JDBC Driver Tuning

When tuning application which connect database using Oracle JDBC driver, you should refer the following documents.

• Oracle 10.2 JDBC Developer's Guide and Reference/Statement Caching
• Oracle 10.2 JDBC Developer's Guide and Reference/Implicit Connection Caching
• Oracle 10.2 JDBC Developer's Guide and Reference/Performance Extensions
• Oracle 10.2 JDBC Developer's Guide and Reference/OCI Connection Pooling
• Oracle 10.2 JDBC Developer's Guide and Reference/Coding Tips

JBoss Tuning

• Tuning and Slimming JBossAS
• JBoss Performance Tuning part 1
• JBoss Performance Tuning part 2

WebSphere Tuning

• WebSphere AS 7.0 Performance Tuning
• OS Tuning for WebSphere AS 7.0

WebLogic Tuning

• WebLogic 10.3 Performance and Tuning
• WebLogic 9.2 Performance and Tuning

Linux Tuning

• System Tuning Info for Linux Servers

Windows Tuning

• Windows 2000 Performance Tuning