Devoxx 2009: A Year of Monitoring with Java-Monitor

17/11/2009, Tools in Action, Kees Jan Koster

Slides are available at: http://java-monitor.com/forum/showthread.php?t=646

Case-by-case approach from experiences reported on java-monitor.com, from an Operations' perspective.

Memory Leak:
  • diagnose:
    • Scavenger gc gives up and stop the world gc (= mark and sweep gc) takes over
    • monitor # scavenger gc's versus mark and sweep gc's
  • solution:
    • profile your app and fix your app
    • workaround: increase heap
Misbehaving GC
  • problem: System.gc() in code freezes the JVM!
  • diagnose:
    • many stop the world gc's (= mark and sweep gc)
    • low heap usage (e.g. 20%)
  • solution:
    • workaround: -XX-DisableExplicitGC
    • use findbugs and fix code
Native threads in 32 bit environments
  • problem:
    • OutOfMemoryException: Couldn't allocate native Thread
    • 32bit Systems share 4GB with OS Kernel, JVM (heap + perm space) and thread stack space.
  • solution:
    • workaround: -Xss limitThreadStack
    • migrate to 64 bit!
In verboseGC: "promotion failed"
  • problem:
    • verboseGC logs "promotion failed" --> stop-the-world GC (mark and sweep) is triggered
    • CMS is non-compacting: the heap can end up to be fragmented. Fragmentation can prevent promotion of young heap (eden spaces) to old heap.
  • solution:
    • bigger heap
    • no real solution
File Descriptors
  • problem:
    • "too many open files"
    • files not properly closed on exceptions --> file description leakage
    • GlassFish opens file-descriptors for each static file served (efficient usage of FileChannels)
  • solution:
    • check ulimit / limits of your OS
    • fix your application (findbugs)
  • hyperthreading doesn't make any significant difference
  • use the magic "-server" switch


Unknown said...

Dear Michaël ,

Nice summary. Hope you liked the talk?

Here are the actual slides, with a bit more details.

Kees Jan

Michaël said...

I did enjoy the talk, both from personal and professional interest. The only downside was that one hour was a bit too short for the presentation and you couldn't delve into the details.

I added the url of your slids

Unknown said...

Dear Michaël,

The tools in action talk was only 30 minutes, so I really had to rush. On Monday night I presided a BOF on more or less the same subject. We had an hour then.

I'll ask for more time next year. :-)

Kees Jan