Devoxx 2009: A Year of Monitoring with Java-Monitor

17/11/2009, Tools in Action, Kees Jan Koster

Slides are available at: http://java-monitor.com/forum/showthread.php?t=646

Case-by-case approach from experiences reported on java-monitor.com, from an Operations' perspective.

Memory Leak:
  • diagnose:
    • Scavenger gc gives up and stop the world gc (= mark and sweep gc) takes over
    • monitor # scavenger gc's versus mark and sweep gc's
  • solution:
    • profile your app and fix your app
    • workaround: increase heap
Misbehaving GC
  • problem: System.gc() in code freezes the JVM!
  • diagnose:
    • many stop the world gc's (= mark and sweep gc)
    • low heap usage (e.g. 20%)
  • solution:
    • workaround: -XX-DisableExplicitGC
    • use findbugs and fix code
Native threads in 32 bit environments
  • problem:
    • OutOfMemoryException: Couldn't allocate native Thread
    • 32bit Systems share 4GB with OS Kernel, JVM (heap + perm space) and thread stack space.
  • solution:
    • workaround: -Xss limitThreadStack
    • migrate to 64 bit!
In verboseGC: "promotion failed"
  • problem:
    • verboseGC logs "promotion failed" --> stop-the-world GC (mark and sweep) is triggered
    • CMS is non-compacting: the heap can end up to be fragmented. Fragmentation can prevent promotion of young heap (eden spaces) to old heap.
  • solution:
    • bigger heap
    • no real solution
File Descriptors
  • problem:
    • "too many open files"
    • files not properly closed on exceptions --> file description leakage
    • GlassFish opens file-descriptors for each static file served (efficient usage of FileChannels)
  • solution:
    • check ulimit / limits of your OS
    • fix your application (findbugs)
  • hyperthreading doesn't make any significant difference
  • use the magic "-server" switch
Post a Comment