Devoxx 2013 - Conference day 3: Random stuff

Hadoop data mining swiss army knife

Alexandre Dutra Cançado, Bertrand Dechoux, Pablo Lopez

Map/reduce: Hadoop

• record (key/value) based
• map/group-by/reduce
• distributed batch

Hive

• load data in table
• sql-like syntax to split lines / do processing (sql can become complex)
• metastore / HCatalog
• external tooling (Tableau)
• partitionng not trivial

Pig Latin: http://developer.yahoo.com/blogs/hadoop/comparing-pig-latin-sql-constructing-data-processing-pipelines-444.html

• new language for map/reduce
• procedural

Cascading:

• java api
• procedural

---

Enterprise Concurrency - practical solutions with JSR236

Alex Heusingfeld refs

• http://www.slideshare.net/FredRowe/jsr-236-concurrency-utils-for-ee-presentation-for-java-onesf2013-con7948jsr236
• http://jcp.org/en/jsr/detail?id=236
• https://blogs.oracle.com/arungupta/entry/concurrency_utilities_for_java_ee

---

Taming Drones: How Java Controls the Uprising of the Drone Force

Eva Veenstra-Kazakov, Timon Veenstra

"specialists at low cost" e.g.

• rainfall damage - aerial overview
• biomass estimation -> AgroSense tool
• track spraying

selled as toy: quadcopter Parrot AR.Drone 2.0

• library for lowlevel access
• javafx app (wifi): instructions / data
• inside / outside options
• sensors: cameras, sonar, speed ...
• API AgroSense builds on top of ARDRoneForP5

autopilot through java: very experimental

• extra raspberry pi (reuse existing usb connector)
• gps + big antenna - accurate to 2m -> correct with gsm signal (RTK-GPS)
• challenges: wind, 30 min fly time, mandatory wifi connection, battery, charging, follow tractor...

refs:

• http://agrosense.eu
• http://agrosense.java.net
• https://bitbucket.org/limetri/drone-gui
• https://github.com/shigeodayo/ARDroneForP5
• http://learn.adafruit.com/adafruit-ultimate-gps-on-the-raspberry-pi/

Devoxx 2013 - Batch Applications for the Java Platform (JSR 352)

Scott Kurz

JSR352 spec: Batch Applications for the Java Platform

• JEE7
  • CDI: not required, but used in the ref. spec
• java6 pre-req
• SE-friendly (minus transactional req.); but not part of SE...
  • can use other dependency injection frameworks.

implementations (under development!):

• ref impl: developed exclusively within IBM: readonly for public -- not for production use
• Spring Batch
• RedHat
• Batch EE project Apache (tomee) ???

sequential batch reasons

• transactional (ACID)
• efficiency gains
• massive parallelization not needed
• facilitate restarts
  • reasons:
    • invalid data
    • lock contentions
    • batch window closed
  • restart:
    • within job: skip completed steps
    • within step: skip processed data-chunks

key concepts (roles)

• implementor: programming model
  • batchlet: free form
  • chunk: ETL pattern (Extract Transform Load
• orchestrator:
  • job specification: batch.xml
    • step
    • batchlet
    • support for EL
  • can change depending on implementations!
  • flow control:
    • define exception handling (e.g. skip)
    • interpret and act on return codes
    • retry config (retry with/without rollback)
    • define parallel jobs or steps (aka. partitioning) + support merging (config + programming model (Partion* classes)
• executor: runtime environment
  • basic components
    • Job Repository
    • JobOperator
  • implementation-specific (not defined in the JSR):
    • clustering, security, ...
    • performance

execution of Job

• JobInstance
• JobExecution
• StepExecution - chunk loop step: one global transaction:
  • ItemReader
    • data access / deserialization of records
    • checkpoint function (positioning)
  • ItemProcessor:
    • core business logic
    • Context gotchas
      • Jobcontext / StepContext: not reliable; consider partitions as running on separate jvms
      • transient data: careful on restart
      • -> design "context" as part of your batch
  • ItemWriter
    • similar to ItemReader, but for output
    • accepts "chunk" of output objects
    • no restrictions on data access

nice first step, looks a bit like EJB 2.x, but, hey, this is only 1.0...:

• xml config with vendor-specific constructs
• no really "open" / "usable" ref implementation
• no "old" and a lot of vendor-specific aspects.
• no annotations

Devoxx 2013 - Conference day 2: Random stuff

Go see all the ships in the world

Francesc Campoy Flores

Check out the original talk from Google I/O:

• www.youtube.com/watch?v=MT7cd4M9vzs
• https://developers.google.com/events/io/sessions/327696300

Nice overview of Google Cloud offering and usage

---

Thalmic Labs - Myo API from init to flush

Gord Tanner, Scott Greenberg

Myo: Gesture Control Armband

• https://www.thalmic.com/en/myo/
• https://www.youtube.com/watch?v=oV0PbKZyqg8
• track muscular activities in arms -> control computer, phone ...
• Talmic Labs, Waterloo, CA
• scheduled for 2014

details

• Bluetooth 4.0 Low Energy
• 9 axis tracking
• tactile feedback
• muscle activity sensor
• Myo API will target multiple languages
• demo: node.js

---

Security test automation in software development using open source tools

David Tillemans

ZAProxy -- web application security scanner developed by OWASP

ref: https://www.owasp.org/index.php/ZAP

---

AsciiDoctor

Andres Almiray

http://asciidoctor.org/

• ruby
• ascii formatting syntax
• cfr. txt2html http://txt2html.sourceforge.net/

---

NFC + Chrome = Awesomeness

Francois Beaufort

chrome apps:

• html5 + css3 + js
• immersive
• secure
• offline by default
• chrome APIs

NFC enabled through USB Chrome API (chrome.usb)

• USB NFC devices: ACR122U, SCL3711
• chrome.nfc api
  • findDevices()
  • write()
• next:
  • bugfix
  • open-source the lib
  • improve performance

Unfortunately not a mainstream browser option for now.

---

Browser-side security: Mitigate the risk of XSS

Mike West

"lock down applications and securely communicate crossorigin"

• Web Cryptography API
• MessagePorts
• on Parleys: http://www.parleys.com/play/5148922b0364bc17fc56c937/chapter0/about

Devoxx 2013 - Nuts and Bolts of Java EE 7 Interceptors

Emmanuel Bernard

Interceptor 1.2

• update of 1.1, part of JEE7
• common rules
• used by developers and container providers (e.g. bean validation)
• AroundConstruct lifecycle
• priority ranges

Bean Validation and Interceptors

• define / declare constraints on objects
• annotation based
• extensible
• (new) validation of parameters and return values of methods
• no proprietary hook
• ordering of validators -> new @Priority

details

• @AroundConstruct -> dedicated interceptor class
• ordering of interceptors:
  • @Priority: global ordering (smaller number = first - cfr. Interceptor.Priority...)
  • or: ordering with @Interceptors list
  • @ExcludeClassInterceptors: remove interceptors defined on classlevel
• Lyfecycle: simplified method signatures for checked exceptions

Devoxx 2013 - Java Microbenchmark Harness: The Lesser of Two Evils

Aleksey Shipilev

micro-benchmarks:

• depends on usage
• aim: explore performance models

java Microbenchmark Harness

• http://openjdk.java.net/projects/code-tools/jmh/
• specific to HotSpot / OpenJDK

issues when running micro-benchmarks

• warm-up all benchmarks
  • JIT
  • GC config
• hyperthreading
• power vs performance logic from cpu -- can change between cores (cpufreq, speedstep,...)
• OS Scheduler logic
• time sharing of OS
  • e.g. System.nanoTime() calcs -> de-schedule by OS influences measuring
  • do not overload system
• avoid dead code (eliminated by the compiler)
  • JMH "BlackHoles"
• loop unrolling
• replay / do multiple JVM runs
• check inlining: -XX:+PrintInlining
• check memory footprint (SRAM cache vs slow DRAM mem)
• cpu branch prediction (out-of-order engines) -> realistic data

conclusion: micro-benchmarks are not simple :-)

• check everything
• retest multiple times
• use JMH...

Devoxx 2013 - How to make your Android apps suck. A rant.

Litrik De Roy

Android problems to avoid

• crazy permissions. E.g.
  • directly call phone numbers
  • phone status & identity: tracking
  • retrieve other apps running
  • mock location for testing -> test-only settings...
• splash screens :-(
• "menu button of shame" lower right corner of app
• Skeumorphic design
  • try to use iOS design on Android
  • tabs at the bottom of the screen
• spammy notifications: games
• exit confirmation "are you sure you want to exit?" -> doesn't exist..
• portrait only mode
• tabs without swiping
• Jerky scrolling
• themed buttons without selectors -> pressed, selected, active , disabled
• app size (=feature creep)
• flooding logcat (too many debug messages)

Devoxx 2013 keynote: Shaping the future of web development

Lars Bak, Kasper Lund

Dart is out of beta: 1.0

• open sourced: www.dartlang.org (BSD License)
• structured programming language
• reliable libs
• IDE (based on eclipse)
• dart2js translator
• ongoing development (speed, more compact, standardized etc.)

Dart

• abstraction mechanisms
• optional static types (type annotations)
• can be combined with existing Javascript: https://github.com/dart-lang/js-interop

IDE -- Dart SDK1.0

• special Chromium with DartVM integrated
• integrated debugger
• automatic update on code changes
• code completion

Real Dart Projects:

• Montage (Mixbook): nice photo book tool
• Blossom - agile project management tool
• GreanTea Project - internal Google project
  • Angular ported to Dart: AngularDart https://github.com/angular/angular.dart
  • subprojects:
    • shadowdom http://www.html5rocks.com/en/tutorials/webcomponents/shadowdom/
    • Streamy https://github.com/google/streamy-dart
    • route https://github.com/dart-lang/route
    • Polymer.dart

Performance of Dart:

• best on native Dart VM
• converted to JS: comparable to JS
• startup performance imporved because of snapshot mechanism (Dart VM only)

Dart outside the browser:

• standalone
• IO libs (networking, file, ...)
• cpus: ARM, MIPS, x64

Devoxx 2013 - EJB 3.2 and Beyond

David Blevins

deprecated stuff: (aka. 'optional')

• Entity Beans (CMP & BMP)
• JAX-RPC -> replaced by JAX WS and JAX-RS
  • XML-RPC
  • Soap Encoding

API Groups: EJB spec sliced in groups -- check if your JEE server supports your requirements:

• EJB lite
• MDB
• EJB remote
• persistent Timers
• JAX-WS
• Embeddable EJB Container
• EJB 2.x
• Entity beans

small fixes:

• lifted restriction from EJB: "do not touch the file system" (wasn't enforced)
• business interfaces: allow multiple interface implementation
• interceptors: difficult method signatures -> 1 method can intercept all events
• transactional callbacks: now allowed for @Stateful
• passivation capable (EJB with CDI scopes)
• timer service: more global, usable outside bean code (javax.ejb.Timer & TimerHandle)
• security: implicit '**' role (= allow any authenticated user) and no more xml needed

Bigger fix: Message-Driven Beans (MDB)

• basics: (ongoing!)
  • not specific to JMS! (= Connector-Driven Beans)
  • configuration outdated
    • loosely typed
    • bad documentation
  • static interface (text)
• learning from JAX-RS
  • no interfaces
  • fluid method signatures
  • annotation-based config
    • strongly typed
    • self-documenting
    • targeted (class, method or param)
• api for creating your own connector:
  • implement ResourceAdapter: bean invocation
  • MessageEndPoint
  • annotation in MessageDriven bean: @MessageDriven
  • demo: implementation accessible with telnet: custom command prompt / commands.
  • -> MDB is basis for accessing any legacy system
  • -> can replace JMX; portable / standard
• ref: https://github.com/dblevins/mdb-improvements

future JEE8 (under development):

• CDI-EJB Alignment
• scopes are similar
• fundamentally the same: both return a "fake" Proxy object as interface
• difference is services: EJB has extra security and transaction -> allow EJB services on CDI beans
• life-cycle lineup: @Singleton, @Stateful
• deprecate EJB (will take years...)

Devoxx 2013 - Cryptographic operations in the browser

Nick Van den Bleeken

why

• easy and secure authentication
• sign/verify messages & docs
• encrypt docs

existing crypto frameworks (Dojo, CryptoJS, etc.)

• completely implemented in JavaScript
• hard to protect against timing attacks (optimizations in JS engine change)
• Math.random() is not secure
• hard to get correct
• performance issues (e.g. number is 64bit float)

alternatives:

• java plugins
  • disabled in many browsers (recommendation US DHS)
  • not on mobile
• native extensions
  • separate install
  • browser specific (some browsers drop support for extension!)
  • not on mobile

solution: Web Cryptography API

• W3C standard: http://www.w3.org/TR/WebCryptoAPI/ (under construction...)
• JS API for web developers
• still requires TLS to prevent sniffing communication
• still relatively complex (requires correct usage of encryption algorithms)

low level API

• Basic building block: window.crypto.subtle
  • .generateKey(algorithmKeyGen ...)
    • callback when key is generated
  • importKey: add existing keys (e.g. public key of bank)
  • verify() signature
  • decrypt() message
  • sign() message
  • ...
• use and combine it for more complex operation

high level API:

• what framework to choose NaCl, KeyCzar, SJCL or ...
• hard to define
• existing APIs should address this: JQuery, Dojo, Prototype etc.
• -> Not for now...

implemented by

• MS IE11 (old version of spec)
• chromium (flags)
• mozilla (under development)
• missing: Safari...
• polycrypt (old version of spec)
• Netflix: webcrypto key discovery

 future need: smartcard support (but: not a focus for US-based companies...)

Devoxx 2013 - Conference day 1: Random stuff

keynote: Java, Chess, and the Internet of Things

Stephen Chin, Richard Bair

Dukepad

• http://j.mp/DukePad
• raspberry pi --> very slow cpu; very good GPU
• java se embedded (ARM)
• javafx

demo Chessrobot

---

Fluentlenium

Mathilde Lemee

• https://github.com/FluentLenium/FluentLenium
• selenium-based
• CSS selector api
• in assertion-framework

---

WebJars - The Web in JARs

James Ward 
http://www.webjars.org/

---

The Crazyflie Nano Quadcopter development platform

Tobias Antonsson, Arnaud Taffanel, Marcus Eliasson
http://www.bitcraze.se/

---

Taming Functional Web Testing (with Spock & Geb)

Luke Daley, Peter Niederwieser

to check...

• https://code.google.com/p/spock/
• http://www.gebish.org/
• http://www.infoq.com/presentations/testing-spock-geb

---

Arquillian testing framework

Devoxx 2013: The Modular Java Platform and Project Jigsaw

Mark Reinhold

Java platform layers; total of 54 mb:

• compact 1: 11mb: core
• compact 2: 16mb (Jdbc, rmi, jaxp,...)
• compact 3: 30mb (rowset, jdbc, prefs, xmldsig)
• 54mb: javafx, crypto, httpservers, corba...
• + extra tools (jaxws, devtools,...)

Not a perfect solution -> better to make a dependency model of each component (directed graph).

Performance solution needed for:

• fast class lookup
• aggressive inlining
• ahead of time compilation (startup speed)
• JVM-specific memory images
• removal of unused fields/methods/classes

Solution:Jlink (optional tool)

• input: *.jar, *.class, *.jmod
• output: jmod, jar, JVM image
• e.g.:
  • jlink -mods javafx,rmi,nashorn -d $JRE
  • javac -mods javafx,rmi,nashorn -d classes *.java
  • java -mods javafx,rmi,nashorn -jar ...
• requires changes to the Java Runtime: no rt.jar & tools.jar anymore (bad representation for JVM)

Security & modularity

• current setup can't hide some parts. E.g.: sun.misc.Unsafe
• historic checks: java.lang.SecurityManager::checkPackageAccess: no static checks possible
• modules & dependency graph can help validate access to sensitive packages
• some internal packages will be removed -> must be fixed!:
  • sun.*
  • *.internal.*

Devoxx 2013: HTTP Caching in Practice

Xavier Coulon

types of cache:

• local cache
• proxy cache
• gateway cache
• (app cache -- custom built in the app)

http1.1 caching GET requests

• expiration Last-Modified + Expires headers
• If-Modified-Since header -> 304 not modified
• ETag header
• If-Match & 412 Precondition failed.

prevent inconsistencies: use explicit directives

• Cache-Control
  • request header: no-cache, max-age, no-store, max-stale, min-fresh, no-transform & only ..
  • response: public, private, no-cache, proxy-revalidate, max-age,...
• pragma header, request & response: no-cache: only for http 1.0 backwards compatiblity.
• Refresh button in brower adds Cache-Control and Pragma headers to request. (not in Ajax requests!

java helpers:

• Servlet Filter
• JAX-RS 2.0: interceptors: Ajax-specific (e.g. add ETag + send Status.NOT_MODIFIED)
• testing:
  • Arquillian (+ REST extension)
  • REST-assured

Https:

• proxy caching doesn't work
• decide on using http for static resources...

caching strategy:

• static content:
  • max cache usage
  • enable gateway caching: Cache-Control: public, max-age
• dynamic content:
  • timeout or ETag

RFC 2616 spec (http 1.1): http://www.ietf.org/rfc/rfc2616.txt

Devoxx 2013 - Lambda: A Peek Under the Hood

Brian Goetz

lambda expression

• "anonymous method"
• type inference compiler
• capture values from context

why lambdas for java

• path to multicore support in libraries
• library developers
• competition other languages
• inner classes: clunky
• @runtime support

typing of lambda expressions

• function types? -> VM / bytecode issues -> too complex
• functional interfaces:
  • single method interfaces (cfr Runnable, Comparator)
  • compiler identifies functional interface
  • compiler infers lambda
    representation
• inner classes? -> performance issues, error-prone, -> "obvious but wrong"
• MethodHandle in Java7 (tool for compiler writers) -> performance issues, conflates binary representation with impl. -> wrong
• invokedynamic
  • lets some language logic determine call logic
  • performant impl. in jvm

lambda & invokedynamic

• code generation strategy (metafactory)
  • dynamic inner classes (=current Java 8 startegy)
  • wrapper class per interface
  • dynamic proxies (or MethodHandleProxy
• vm-private APIs
  • build object from scratch
• lazy evaluation
  • non-capturing lambda's: create single instance
  • defer initialization cost to first use
  • VM optimizes call
• faster than inner classes
• usable by other languages
• serialization support: SerializeLambda
• future VM optimizations
  • escape analysis

ref: http://openjdk.java.net/projects/lamda

Devoxx 2013 Keynote: Java 8 and beyond

Mark Reinhold, Brian Goetz

Java: 18 years old

• static language, but many dynamic features
• many great features: gc, JIT ..
• principles Java Language
  • reading code more important than writing
  • simplicity
  • one language, same everywhere

Java 8 features:

• lambda expressions (+ type inference)
• default methods in interfaces
• java.util.stream: aggregate / data-parallel operations

lambda expressions:

• inspiration: How to think about Parallel Programming -- Not (Guy Steele) (http://www.infoq.com/presentations/Thinking-Parallel-Programming)
• more expressive code
  • instead of boilerplate of anonymous inner classes
  • syntax inspired from C#
  • single-method interfaces
  • ´´default´´ methods
    • default implementation
    • fully virtual
    • how stream() method was added
• more powerful:
  • treat behavior as data
  • "gentle turn for the functional"
  • push away from mutative / imperative
  • huge impact on API design
• more parallel
  • aggregate operations (filter, conversions, max, etc.)
  • stream() -> parallelStream() (single parallel pass on datasets)
  • retrofitted Collections

next:

• value types: key enabler for
  • tuples,
  • user-defined primitive types