Devoxx 2013 - Java Microbenchmark Harness: The Lesser of Two Evils

Aleksey Shipilev

micro-benchmarks:

• depends on usage
• aim: explore performance models

java Microbenchmark Harness

• http://openjdk.java.net/projects/code-tools/jmh/
• specific to HotSpot / OpenJDK

issues when running micro-benchmarks

• warm-up all benchmarks
  • JIT
  • GC config
• hyperthreading
• power vs performance logic from cpu -- can change between cores (cpufreq, speedstep,...)
• OS Scheduler logic
• time sharing of OS
  • e.g. System.nanoTime() calcs -> de-schedule by OS influences measuring
  • do not overload system
• avoid dead code (eliminated by the compiler)
  • JMH "BlackHoles"
• loop unrolling
• replay / do multiple JVM runs
• check inlining: -XX:+PrintInlining
• check memory footprint (SRAM cache vs slow DRAM mem)
• cpu branch prediction (out-of-order engines) -> realistic data

conclusion: micro-benchmarks are not simple :-)

• check everything
• retest multiple times
• use JMH...