There are several causes for such an exception, but the most obvious is that there is memory leak somewhere in the application.
The situation
At one of my recent projects we were having some memory related issues on one of the production machines. The web application was running in Tomcat 6 and it appeared that at a certain point in time, the JVM tried to allocate twice the average memory it was using. Let's say from 2 Gb the memory footprint went up to 4GB. You might have guessed it, because only a couple of minutes later we were represented with the OOME message in the server log. It turned out that this was related to the amount and type of requests being handled by the application server. I was glad to find out it was not a memory leak, but more a warning that the total size of requests could allocate a lot of memory. If your application is running, it's hard to see what objects allocate memory. To get an insight on what the application was doing at the moment the OOME occurred, we configured the JVM to create a memory(heap) dump/snapshot at the moment that the OOME was thrown.Generating a heap dump
Most of my clients run on the Sun(Oracle) JVM, so I will try to keep this post focussed on the instructions for the Sun JVM. In case your application server runs out of memory you can instruct the JVM to generate a heap dump when an OOME occurs. This heap dump will be generated in the HPROF binary format.You can do this in by:
- Manually: by using 'jmap', which is available since JDK 1.5.
- Automatically by providing the following JVM command line parameter:
-XX:+HeapDumpOnOutOfMemoryError
The size of the heap dump is around the same size as the configured max heap size JVM parameter.
So if you have set your max heap size to -Xmx512m , your heap dump will be around that size.
Analyzing the heap dump
Now you have the heap dump and want to figure out what was inside the heap at the moment the OOME occurred. There are several Java heap dump analyzers out there, where most of them can do more then just heap analysis. The products range from commercial to open source and these are the ones that I tried with my 4Gb .hprof file:I was surprised to see that most of the above applications were unable to handle a file of this size. Eclipse Memory Analyzer was actually the only heap dump analyzer that was able to handle a heap dump of this size on my MacBookPro. All the other analyzers were unable to handle a file of this size. Apparently some of these tools tried to load the entire file into memory. On a laptop with only 4 Gb this will not fit. Eclipse MAT was able to analyze this file within 5-10 minutes.
About Eclipse Memory Analyzer
One of the great things about Eclipse Memory Analyzer is that it starts indexing the heapdumps on first load. This makes the processing of the heapdump very fast and once you've parsed the entire heapdump, reopening it is a piece of cake, because it does not have to process it all over again.
Once you have the heapdump on your screen the dominator tree view is the most useful view and can give you a very good insight on what was loaded when the server ran out of memory.
Next to the statistical views there is also an automatic leak hunter available to help you figure out the problem as fast as possible.
Summary
If you ever have to analyze a heap dump I would recommend to use Eclipse Memory Analyzer. It's fast, easy to use and free.
Really sorry, Eclipse Memory Analyzer wouldn't parse my 4G head dump. Allocated 512mb to the JVM, but it ran out of heap.
ReplyDeleteAdmittedly it got further than Netbeans/jhat, but on my 2GB RAM PC, and it seems that it requires less memory - 14% complete used 512MB.
Seems stupid that a dump file from a server, can't be inspected by a developer's PC. So annoyed :(
Hi I've only seen this once. It means that if you have configured the Max heap size (Xmx) to 512m, the biggest object tree inside the memory dump is larger then this max size. Try 1 Gb for the Xmx and close all other applications. See if that works out for you.
ReplyDeleteHey,
ReplyDeleteI faced a similar problem with my 2.5 gb heap dump file.
This solved my problem.
http://wiki.eclipse.org/index.php/MemoryAnalyzer/FAQ#Out_of_Memory_Error_while_Running_the_Memory_Analyzer
I allocated 2.7 gb on my system for MAT tool and it worked really fine.
No problem to analyze large heaps with any of the above tools you can see how to do this in the this blog
ReplyDeletehttp://performanceandjava.blogspot.com/
When analyzing large java heap dumps and trying to do memory leak detection, you should also take a look at BHeapSampler:
ReplyDeletehttp://dr-brenschede.de/bheapsampler/
You may be amazed by the clarity and simplicity of the infotmation it presents.
It does not work:
Deleteroot@ads8-1: ~ # java -Xmx800m -jar bheapsampler.jar memdump
BHeapSampler 1.0 / 2011 / Dr. Arndt Brenschede
* Sampling from: memdump in pass 0
version = JAVA PROFILE 1.0.1 id-size=8 date=Mon May 14 20:23:15 UTC 2012
Thread-ID:0
tag = heap_dump size=-374919899 date=Mon May 14 20:23:15 UTC 2012
heap size:-374919899 date=Mon May 14 20:23:15 UTC 2012
tag = size=-109726256 date=Mon May 14 20:23:15 UTC 2012
tag = size=2041 date=Mon May 14 20:23:15 UTC 2012
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: -8
at BHeapSampler.a(Unknown Source)
at BHeapSampler.main(Unknown Source)
Check for the latest version at:
Deletehttp://dr-brenschede.de/bheapsampler/revisions.html
There had been problems with parser robustness in the 1.0 version,
if with the current version (1.3) you still have problems, please
drop me a note per email (see the README for email adress)
heap dump size depends on your max heap size [ IF you are experiencing a OOM]. your best bet to open a heapdump (hprof or .phd) > 1.5 gigs is to get a 64 bit machine and install a 64 bit JRE AND a 64 bit version of MAT WITH a lot of RAM.
ReplyDeleteI was finally able to open those huge heapdumps > 1.5 g {generated on solaris64bit}
on a Windows 64 bit machine with 8gb ram. I started MAT (.ini file) with -Xmx3g.
Hope this helps.
I am trying to analyze a heap dump {phd format}, the value of -Xmx parameter is set to 1024mb but when i got the heap dump and opened it in Eclipse Memory Analyzer th e "beggest object by retained size" graph show the total memory of 485mb, where is the rest of the memory.
ReplyDeleteHi - I have >2g heap dump file, pls suggest what would be the best place to analyze(like online service upload heap dump file, response should be downloadable report format). I tried from my win xp not enough memory.
ReplyDeleteany help would be greatly appreciated...!
Hello, I'm running SAP Business Objects 4.0 with Tomcat6..
ReplyDeleteI got a massive .hprof file 9-Gig's.. I can open this file with Eclipse Memory Analyzer. Had to set my "Xmx" to 6-Gig..
So I can open it, But how do I read it to find the source of the problem... The issue happens when a particular reports is run, so I know there is an issue with that report/query.. But I'd like to Learn/Use Eclipse Memory Analyzer see the error in the .hprof file..
Thanks in advance for any help in learning how to use this tool.
Steve
Memory Analyzer comes with some default reports like, Dominator Tree, etc. I would first start with those, since they can lead you into a direction that eventually might help you figure things out.
ReplyDeleteThere is no standard sequence of steps to figure out what is going on. You know your application best.