Devoxx 2009: A Year of Monitoring with Java-Monitor

17/11/2009, Tools in Action, Kees Jan Koster

Slides are available at: http://java-monitor.com/forum/showthread.php?t=646

Case-by-case approach from experiences reported on java-monitor.com, from an Operations' perspective.

Memory Leak:

• diagnose:
  
• Scavenger gc gives up and stop the world gc (= mark and sweep gc) takes over
  
• monitor # scavenger gc's versus mark and sweep gc's
• solution:
• profile your app and fix your app
• workaround: increase heap

Misbehaving GC

• problem: System.gc() in code freezes the JVM!
• diagnose:
  
• many stop the world gc's (= mark and sweep gc)
• low heap usage (e.g. 20%)
  
• solution:
• workaround: -XX-DisableExplicitGC
  
• use findbugs and fix code
  

Native threads in 32 bit environments

• problem:
  
• OutOfMemoryException: Couldn't allocate native Thread
• 32bit Systems share 4GB with OS Kernel, JVM (heap + perm space) and thread stack space.
  
• solution:
• workaround: -Xss limitThreadStack
  
• migrate to 64 bit!
  

In verboseGC: "promotion failed"

• problem:
  
• verboseGC logs "promotion failed" --> stop-the-world GC (mark and sweep) is triggered
  
• CMS is non-compacting: the heap can end up to be fragmented. Fragmentation can prevent promotion of young heap (eden spaces) to old heap.
  
• solution:
• bigger heap
  
• no real solution
  

File Descriptors

• problem:
  
• "too many open files"
• files not properly closed on exceptions --> file description leakage
• GlassFish opens file-descriptors for each static file served (efficient usage of FileChannels)
  
• solution:
• check ulimit / limits of your OS
  
• fix your application (findbugs)
  

others

• hyperthreading doesn't make any significant difference
• use the magic "-server" switch