Tech stuff

Performance Anti-Patterns in Hibernate

2011-11-16T18:27:00.001+01:00

Patrycja Wegrzynowicz
Snippets:

"premature optimization is the root of all evil"
"expect unexpected"...
avoid temporary changes
careful with Collections: inadequate collection mappings can impact resulting # of queries
OneToMany on owning side: lots of unnecessary locking / querying
bulk processing through Hibernate --> use database operations instead

Conclusion

big data = big problems -- my addition: with hibernate :-)
- standard mappings don't handle large datasets well
- smart model, bulk processing, projections
read the manual
- hibernate docs
- think of consequences
hibernate
- smart policies / data structures

Nothing was said of my personal performance finding with Hibernate: maximally use readonly access whenever possible. The support for 'dirty' objects adds a lot of performance overhead if some model-parts are readonly.

Bleeding Edge HTML5

2011-11-16T18:26:00.001+01:00

Paul Kinlan
This presentation demonstrated some ongoing - bleeding edge - development in html5:

: section that can be collapsed: no extra javascript required
: highlight portions of text
- less javascript
- easier for to handle
input-tag adds support for speech-to-text input
video & animations: check if the current tab is visible. Allows to stop a video when the tab is inactive
a link can suggest the browser to pre-fetch the contents of links the user will likely click --> the page is pre-loaded
intents: loosely coupling/integration of services from different websites. Cfr Android intents: an application asks a photo viewer and Android lets the user choose which one he wants.
check online / offline status in javascript or use a callback. --> clean handling of unconnected status.
paste of images in browser
camera / microphone support
full-screen support, callable from javascript (user must grant permission)
WebRTC: real-time communication: Real time Video / audio for communication
much better audio support

check out http://html5hacks.com

JDK 7, 8 and 9

2011-11-16T18:24:00.001+01:00

Mark Reinhold

Java7

Project Coin: small language improvements
- diamond <> operator: easier generics
- multi-catch clause
- ...
invokedynamic: bytecode opperation for dynamic languages (e.g. JRuby) and will be used for Lamba expressions in Java 8.
fork/join framework

Java8

Lambda expressions (aka Closures)
Jigsaw: modularity
- improve performance:
  - only load what's needed
  - more efficient jmod package type
- dependencies are defined in the module configuration --> better than the current classpath hell
- jmod modules will be easily packaged into rpm or deb packages ⁻ JavaFX 3.0
Type annotations
new Date/Time library (cfr. JodaTime)
support for Sensors (accelerometer, etc.)

Java9 (not decided yet)

self-tuning JVM's
better JNI
support for Big Data
reified generics
unified types: int <> Integer etc.
optimize tail recursion... finally?
resource management / accounting?

Devoxx11: Java EE Keynote

2011-11-16T11:57:00.001+01:00

Cameron Purdy
Focus on next Java EE versions 7 & 8:

Html 5
Modularity
- jigsaw
- profiles:JEE providers can choose to not implement part of the SPEC. Eg. some older specs like EJB CMP & BMP will become optional.
Cloud
- elastic
- auto provisioning
- Paas

Platform as a Service (Paas) details

JEE7 is the service
auto-provisioning:
- DB (incl. init-script)
- LDAP
- loadbalancer / clustering settings
extensible for frameworks (e.g. Spring, Seam)
Quality of Service (QOS)
API's for the Cloud: Caching ...
Multi-tenant: deploy the same app for multiple / isolated users.
Cloud descriptions in meta in the ear

Other:

Timeboxed strategy for providing JEE 7 / 8: : everything not ready for JEE7 will shift to JEE8.
Glassfish provides the reference implementation of latest JEE developments.

Devoxx11: Java SE Keynote

2011-11-16T11:56:00.001+01:00

Henrik Ståhl
Java Standard Edition (SE) 7 achievements:

project coin: small language improvements
concurrency: fork/join framework
JavaFx as part of the java language

Java SE 8 focus:

Jigsaw: modularity
lambda (aka closures): opens the door for easier to use concurrency api's
javaFx 3.0
new devices: multi-touch etc.

Devoxx11: Jackpot 3.0 - Large Custom Refactorings

2011-11-15T17:58:00.001+01:00

Jan Lahoda
Jackpot:

originally from Sun labs
uses the Abstract Syntax Tree (AST) --> knows java-language / syntax: smarter than simple regexp's
ideal to refactor deprecated API's
remove anti-patterns

Jackpot is now included in Netbeans 7.1

use special 'Hint' syntax: Class.oldMethod => NewClass.newMethod ;;
completely integrated in standard Netbeans hints / refactoring logic
- hints are show in the IDE, directly to the developer
- batch refactorings are allowed: completely integrated in the "Inspect and refactor" dialog (incl. managing hints)
- run headless from the command line
can be highly fine-tuned:
- use instanceof checks
- match on specific Java constructs: e.g. only look for Fields
- check on target java version
- combine multiple checks
- write repeatable tests in ".test" file
definitely worthwhile looking at for large refactorings of legace Java projects!

Devoxx11: Java 7 toolbox

2011-11-15T17:55:00.001+01:00

Project coin brought some minor improvements to the Java language. Most notably:

easier generics:
- diamond operator <>: less copy/paste thanks to smarter compiler
- less warnings on vararg's
easier error-handling:
- multicatch: less copy/paste
- 'try-with-resource' contstruct: automatic & correct closing and error handling
consistency / clarity
- Strings in Switch
- numbers improvements:
  - direct binary format: 0b0101001
  - logical grouping of large number in sub-chunks: 123_435_576L --> java ignores the underscores.

IDE support

Netbeans & Intellij have excellent support for Java 7
- inline hints and refactorings are offered
- batch refactorings throughout the 'inspect and transform' dialog
Eclipse is somewhat lagging behind.

Devoxx11: Real-world deep-dive into Infinispan - the open source data grid platform

2011-11-15T14:46:00.000+01:00

Sanne Grinovero, Pete Muir and Mircea Markus
Infinispan details

developed & provided by JBoss
Open Source license
distributed, in-memory data structure
High Available: configuration options depending on your safety / performance considerations
- Total replication
- Partial replication ("distribution mode")
Elastic:
- easy to scale up / down
- deal with node stops / crashes
- accept additional nodes
- managed through JGroups

Capabilities:

Transactions
Map / Reduce
Query
JMX access

Two access modes to inifinispan:

embedded mode: in the JVM
client / server mode: different protocols supported:
- REST
- memcached
- hotrod

Possible setups:

local cache: Simple Map-like interface, with advanced cache-features (eviction, etc.)
cluster cache: all caches are kept in sync between JVM's, the developer is responsible for accessing the persistence layer (DB)
grid cache: the cache itself handles the persisting of the data to the DB.

JGroups

reliable cluster communication
developed by JBoss, and used by Infinispan
Node / cluster config (incl. auto-discover setup)
focus on performance: e.g. UDP, but with reliability handled by JGroups
support for reliable multicast messages

Devoxx11: Building Web Applications with MongoDB

2011-11-15T14:44:00.000+01:00

Brendan McAdams

mongoDB is a nosql DB, ie. a "non-relational" DB

no joins
no referential integrity
no complex transactions --> not suitable for higly transactional apps
but: focus on horizontal scalability
- sharding
- replica's
a scheme-less DB
- unique ObjectID is automatically generated (based on time, machine, pid & counter)
- flexible structures: adding/omitting elements is easy
native JSON support
- hierarchical 'documents'
- arrays
- internally, an optimized format is used: BSON:http://bsonspec.org
support for indexes (+ 'multi-indexes' on arrays)

technical specifics:

based on memory mapped files:
- OS maps the files in the RAM.
- pagefaults are expensive --> keep working set in RAM
  - provide sufficient RAM
  - prewarm the DB
- only use 64bit OS & binaries
Intel-only (little endian)
filesystem
- Ext4 & Xfs: only filesystems with support for posix_fallocate()
- set noatime, noadirtime in /etc/fstab
'documents' (i.e. objects) have a max. size of 16mb
accessible over tcp/ip: Mongo Wire Protocol
UTF-8
the data is stored in fysical files / 'extents'

modeling for mongoDB:

think of 'documents' instead of 'rows'
favor embedding over referencing -- mongoDB offers no referential integrity (aka foreign keys)
think of the access patterns of your data

other specifics:

support for MapReduce
geospatial indexing
indexes (internally btrees)
query optimization
advanced queries ($lt, $all, $in...)
gridfs:
- large 'chunked' files
- distributed (sharded)
- accessible through a regular java.io.File

Devoxx11: Codereview with GIT / Gerrit

2011-11-14T18:43:00.001+01:00

Matthias Sohn

Git:

decentralized source version control
good at merging
cryptographic hashes to prevent modifications to history
works well off-line
personal branches
JGit: written in Java
eGit: eclipse plugin for Git

Adopters of GIT:

Linux
CVS will be deprecated in 2012
eclipse.org: converting from CVS to Git

Gerrit:

Code-review based on Git
parallel work-flow based on branches
web based interface

Devoxx11: Android awesomeness

2011-11-14T18:39:00.003+01:00

Chet Haase, Romain Guy and Philip Milne

Android 4.0: Tablet (3.0) & Phone (2.3) reunited

GridLayout: powerful & flexible lay-outing -- way too complex explanation.

Tools for profiling / debugging Android apps. "Experience matters"

Memory
- dumpsys: dump info of processes running on the device
  - adb shell dumpsys meminfocom.android.Launcher: dumps heap usage details
  - dumpsys gfxinfo: graphics chip
  - dumpsys window: decompose window objects used
- DDMS: heap dump
- jhat:
  - generate hprof
  - convert it with hprof-conv
  - in eclipse
  - check out "rootset references" for potentially memory leaks
- MAT: Eclipse Allocation Tracker - http://eclipse.org/mat
Performance
- "use your eyes"
- DDMS
- track frames per second -> "trackFPS()" call in a PreDrawListener
UI
- 'crazy' stuff like high speed cams: Casio Ex-F1: replay the animation in slow motion
- HierarchyViewer:
  - ViewTree
  - PixelPerfectTree
- ViewServer

Devoxx11: Android Jumpstart

2011-11-14T14:54:00.002+01:00

Lars Vogel

Excellent introduction to the Android development experience.

tutorials on Android: www.vogella.de
java --> class -- through dx-tool --> dex
development environment: ADT for eclipse.
- emulator: QEMU
- DDMS: very powerful device debug mode
API Components:
- View for single views & ViewGroup for combining
- Activity: app lifecycle - onPause() is the guaranteed last call to persist data
- Intent: transitions to other apps
  - Implicit: Android decides
  - explicit: hard coded coupling to called app
- Toast class: display quick temporary message
- Preferences
- DIY Views:
  - onDraw() method
  - Path class
  - selective invalidate() call to improve performance
- ListActivity & ListAdapter
- Threading & Async behaviour:
  - Threads: use runOnUiThread for async updates of the UI
  - Services
  - Broadcast & Receiver

Fixing the RTL8191SE wifi under Ubuntu 11.10

2011-10-28T10:28:00.005+02:00

The RTL8191SEvB wifi chipset is still not working under Ubuntu 11.10. Luckily, it's quite easy to add support for it. Please note that the procedure is slightly different from the previous 11.4 steps.

Download the latest Linux driver from Realtek. Choose the Linux driver for kernel 2.6.35 and above.
Extract all files
In a terminal, execute the following commands:
sudo su
cd
make
make install
modprobe rtl8192se
Wifi should be working now. In order to keep the wifi working after a reboot, add one line with "rtl8192se" to the /etc/modules file.
gksudo gedit /etc/modules

Fixing the RTL8191SEvB wifi under Ubuntu 11.4

2011-06-22T22:45:00.006+02:00

Remark: If you're running Ubuntu 11.10, please use this new procedure.

Out of the box, Ubuntu 11.4 has no support for the RTL8191SEvB chipset that ships with my Medion Akoya S5612 laptop. Luckily, the manufacturer provides drivers for Linux.

Follow the next steps:

First, make sure you uninstall ndiswrapper if you experimented with it (check in synaptic)
Download the Linux driver from http://www.wireless-driver.com/realtek-rtl8191se-rtl8192se-wireless-linux-driver-ver0019/ and untar it in your home directory.
In a Terminal, execute the following commands (*)
cd ~/rtl8192se_linux_2.6.0019.1207.2010 sudo su make make install modprobe r8192se_pci
After the modprobe, wifi should work. If everything is OK, you can persist the modprobe command by appending a line with "r8192se_pci" to /etc/modules:
gksudo gedit /etc/modules

(*) Don't try to sudo every single command as the 'make install' will give strange compilation errors. Those errors vanish with the recommended (in this case) sudo su approach.

background:

http://forum.ubuntu-nl.org/internet-en-draadloos/hoe-de-driver-te-installeren-voor-medion-akoya-voor-draadloos/
http://ubuntuforums.org/showthread.php?t=1635892

Fixing the rt2860 wifi chipset under Ubuntu 11.4

2011-06-12T21:30:00.011+02:00

I have a Medion Akoya MD96888 that ran ubuntu just fine from day 1. Unfortunately, with the latest Ubuntu versions the wifi connection is really unstable, unreliable and very slow. This post explains how to identify the problem and how to resolve it.

Diagnosis

The following commands will identify which driver ubuntu uses for your wifi card:

nm-tool
...
- Device: wlan0 ----------------------------------------------------------------
Type:           802.11 WiFi
Driver:         rt2800pci
State:          disconnected
Default:        no
HW Address:     00:22:43:13:7F:7B

Capabilities:

Wireless Properties
WEP Encryption:  yes
WPA Encryption:  yes
WPA2 Encryption: yes ...

 lsmod | grep rt2800pci
rt2800pci           18159  0
rt2800lib           43824  1 rt2800pci
rt2x00pci           13986  1 rt2800pci
rt2x00lib           39075  3 rt2800pci,rt2800lib,rt2x00pci
eeprom_93cx6        12653  1 rt2800pci

sudo lshw -C network
*-network          
 description: Ethernet interface
 product: RTL8101E/RTL8102E PCI Express Fast Ethernet controller
 vendor: Realtek Semiconductor Co., Ltd.
 physical id: 0
 bus info: pci@0000:01:00.0
 logical name: eth0
 version: 02
 serial: 00:21:85:4c:75:cc
 size: 100Mbit/s
 capacity: 100Mbit/s
 width: 64 bits
 clock: 33MHz
 capabilities: pm msi pciexpress msix vpd bus_master cap_list rom ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd autonegotiation
 configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=2.3LK-NAPI duplex=full ip=192.168.1.60 latency=0 link=yes multicast=yes port=MII speed=100Mbit/s
 resources: irq:42 ioport:d000(size=256) memory:ffd10000-ffd10fff memory:ffd00000-ffd0ffff memory:dfd00000-dfd0ffff
*-network
 description: Wireless interface
 product: RT2860
 vendor: Ralink corp.
 physical id: 0
 bus info: pci@0000:02:00.0
 logical name: wlan0
Conclusion: ubuntu uses the rt2800pci driver for accessing the RT2860 wifi chipset. This driver causes the instability.

Solution
Currently the only solution is to download the driver directly from Ralink and manually compiling and installing it. Although the command line steps are quite tedious, I did get a stable wifi back. The following steps are copied and slightly updated from http://ubuntuforums.org/showthread.php?t=1592731.
Download latest RT2860 driver source code from Ralink at http://www.ralinktech.com/support.php?s=2. The driver is tagged RT2860PCI/mPCI/CB/PCIe(RT2760/RT2790/RT2860/RT2890). Put the tar into your home directory.
Extract the tar file and go to the expanded directory (ignore the bz2 extensions, bunzip2 won't work):tar -xvf ~/2010* cd ~/2010*
Edit the config.mk file:
gedit ./os/linux/config.mk 
Find the following lines (by default set to "n") and set them to be "y":
HAS_WPA_SUPPLICANT=y
HAS_NATIVE_WPA_SUPPLICANT_SUPPORT=y
Save and close the file.
Back in the terminal window perform the following command
gedit ./common/cmm_wpa.c
For the warning message choose "Western" then "Retry".

Use the find command to locate "MIX_CIPHER_NOTUSE". Replace the entire line with this code:
WPA_MIX_PAIR_CIPHER FlexibleCipher = WPA_TKIPAES_WPA2_TKIPAES;
Save and close the file.
Make sure gcc is installed (usually is by default but if not install it via synaptic) then from the terminal window perform the following commands:
sudo make
sudo make install
sudo ifconfig wlan0 down    (on some hardware it is ra0)
sudo rmmod rt2860sta
sudo mv /lib/modules/$(uname -r)/kernel/drivers/staging/rt2860/rt2860sta.ko /lib/modules/$(uname -r)/kernel/drivers/staging/rt2860/rt2860sta.ko.dist
Still in the terminal window perform the following commands:
sudo depmod -a
sudo modprobe rt2860sta
sudo ifconfig wlan0 up        (on some hardware it is ra0)
Check that network manager can now select networks and connect to them.
Back in the terminal window perform the following command:
sudo cp ./os/linux/rt2860sta.ko /lib/modules/$(uname -r)/kernel/drivers/staging/rt2860/
In the terminal window perform the following command:
gksudo gedit /etc/modules
Add "rt2860sta" on a line at the end of the file then save and close it.
Reboot and check that wifi still works

Fixing Ubuntu One

2010-01-27T23:00:00.004+01:00

Ubuntu One is Ubuntu's first step into Cloud Computing. Like dropbox, it provides automatic file-syncing of a dedicated "ubuntu one" folder over the internet between several computers. Both solutions are running on Amazon EC2 virtual servers and both solution have a free starting plan that provides 2 GB of synced storage. The magic syncing and free 2GB provides a convenient alternative to USB sticks for exchanging documents.

As an Ubuntu 9.10 user, I played a bit with ubuntu one, but I had problems with the syncing on one computer. If you have trouble with ubuntu one, you can experiment with the command line tools. Some useful commands:

install command line tools:
$ sudo apt-get install ubuntuone-client-tools
$ u1sync --authorize
$ u1sync --init ~/Ubuntu\ One
$ u1sync --diff ~/Ubuntu\ One

At first, I had to uncheck the "limit bandwidth" in the ubuntu one config: ~/.config/ubuntuone

[bandwidth_throttling]
read_limit = -1
write_limit = -1
on = False

This didn't help much, however. These are the relevant ubuntu one directories:

python sources:
/usr/lib/ubuntuone-client/ubuntuone-syncdaemoncd
/usr/share/pyshared/ubuntuone/*
data dir: ~/.local/share/ubuntuone/syncdaemon
shares dir: ~/.local/share/ubuntuone/shares
logging ~/.cache/ubuntuone/log

As I couldn't find any relevant errors and get Ubuntu one continue its syncing, I finally re-initialized Ubuntu One:

I killed all ubuntu one processes:
$ pkill ubuntuone
I removed all ubuntu one configuration files:
$ rm -rf ~/.*/ubuntuone
$ rm -rf ~/.*/*/ubuntuone

After logging out / logging back in, ubuntu one just worked fine. I had to remove the conflict files, but apart from that, the syncing is now OK.

As an additional service, ubuntu one also provides excellent syncing for Tomboy and for Evolution. Simply follow the steps in https://wiki.ubuntu.com/UbuntuOne/Tutorials/Notes.

Ubuntu: Changing the Computer Name

2010-01-04T20:23:00.004+01:00

When installing Ubuntu, the setup wizard asks for a computer name. By default the computername is based on the first username. It is easy to change the computer name afterwards:

gksudo gedit /etc/hosts /etc/hostname

Modify and save both files at the same time

The Realtek 8191/8192SE under Ubuntu

2009-12-14T18:01:00.007+01:00

I have a Realtek 8191/8192SE wifi card embedded in my Medion Akoya S5612. As usual for Realtek, this card has only support for Windows. I had to use ndiswrapper to get the card working under Ubuntu 9.10. Steps:

download the Realtek 8192SE driver
from Synaptic, install the ndisgtk package (dependencies are automatically selected)
now, in System / Administration / Window Wireless Drivers you can install the windows driver. Select the Win2K driver.

That's all. :-)

Accessing GSM SIM-Cards under Ubuntu

2009-11-29T20:38:00.014+01:00

I borrowed a cheap GSM SIM-card reader and tried to use it under Ubuntu. The "Zolid" card reader is both a memory-card reader as well as a SIM card reader. The device presents itself to Ubuntu as follows (including the typo):

$lsusb
Bus 001 Device 005: ID 0bda:0169 Realtek Semiconductor Corp. Mass Stroage Device

The 'Mass Storage Driver' that mounts SD cards just works fine out-of-the-box. I concentrated my experiments on the SIM-card reader. I installed the 'MonoSIM' application to access the device.

http://www.integrazioneweb.com/monosim/
install the debian package. After some google-ing, I found a package at http://linux.softpedia.com/progDownload/monosim-Download-27949.html
under ubuntu, the package installer is automatically triggered and all associated packages (like libmono) are installed if needed
make sure the 'pcscd" smartcard daemon package is installed - check with Synaptic

In Applications/Office you can run the MonoSIM application and manage all phone numbers on it.

Some scripts that might be useful:

combine the 'monosim' file records to 1 line per record
$ cat *mono* | while read a; do read b; echo "$a;$b"; done | sort -u > gsm.csv
I used OpenOffice to filter out old lines and save the file back to csv
converting the csv back to the 'monosim' format:
$ while read a
do
echo $a | cut -d '"' -f 2
echo $a | cut -d '"' -f 4
done < gsm.csv > gsm.csv.monosim

The MonoSIM application works fine; the trick is to reconnect whenever a menu-item is greyed out.

see also: http://www.barrydegraaff.tk/index.php?page=files/Archive/Linux%20Sim%20Card%20reader/index.html

Fixing Auto-mount Problems of a USB Device in Ubuntu

2009-11-24T22:36:00.018+01:00

On one of my laptops with Ubuntu 9.10, USB memory sticks would only mount after several plug/unplug attempts. I investigated the problem and found a solution.

Diagnosis:

$ tail -f /var/log/messages (or call dmesg)
...
Nov 24 21:33:41 michael-laptop2 kernel: [ 2592.748078] usb 2-1: new full speed USB device using uhci_hcd and address 2
Nov 24 21:33:44 michael-laptop2 kernel: [ 2595.113326] usb 2-1: configuration #1 chosen from 1 choice
→ only after several attempts, the log would display "Initializing USB Mass Storage" and proceed with the auto-mount.
$ sudo lsusb
Bus 002 Device 002: ID 041e:200c Creative Technology, Ltd
→ in this case, the USB stick is an mp3 player
$ sudo lshal
...
udi = '/org/freedesktop/Hal/devices/usb_device_41e_200c_4002FA76125D8F8A_if0'
info.linux.driver = 'ndiswrapper'
(string) info.parent = '/org/freedesktop/Hal/devices/usb_device_41e_200c_4002FA76125D8F8A'
(string) info.product = 'USB Mass Storage Interface'
(string) info.subsystem = 'usb' (string) ...
→ the "41e_200c" USB device has been wrongly associated to the ndiswrapper module.

Conclusion: my old experiments with ndiswrapper and the rtl8187b wifi device still left traces on my Ubuntu system I upgraded to 9.10.

Removing ndiswrapper was a bit tricky:

$ sudo ndiswrapper -l
net8187b : driver installed
device (0BDA:8189) present (alternate driver: rtl8187)
$ sudo ndiswrapper -r rtl8187
in synaptic: remove all the ndiswrapper-* packages
$ sudo rmmod ndiswrapper
$ sudo gedit /etc/modules
→ remove ndiswrapper line
$ sudo rm /lib/modules/$(uname -r)/kernel/ubuntu/ndiswrapper/ndiswrapper.ko

Now, usb sticks mount from the first time.
see also: https://help.ubuntu.com/community/WifiDocs/Driver/Ndiswrapper
update 25/11/2009: apparently, the ndiswrapper is part of the ubuntu distribution. After applying the latest security-fix, the ndiswrapper module re-appeared.

update 6/6/2010: the ndiswrapper bug still exists in Ubuntu 10.4.

Devoxx 2009: Performance for the Performance-shy

2009-11-19T18:55:00.002+01:00

19/11/2009, Holly Cummins, IBM

costs of bad performance

electricity
employee productivity
lost business (e.g. slow pages)
hard cash (e.g. trading)

tuning technique

find the bottleneck in the application
- cpu → consistently high
- I/O → cpu not consistently high
- lock → cpu not consistently high
- space ??? cpu can be low or high ...
what is the JVM doing? → tools … must have:
- visual indicators
- explanation
- suggested solution (command line)
measure performance
- risk to make it worse
- get a baseline
- system must be as similar as possible to production
- how:
  - System.currentTimeMillis()
  - warm up your system
    - JIT
    - trigger caches

tools: IBM Perfomance Tools

for IBM jvm's
free
IBM Support Assistant: centralized repository for IBM tools

space-bound applications: heap

diagnose:
- lower throughput
- crash (out-of-memory exception)
- trigger gc's → higher cpu
tools:
- verbosegc: very low overhead → use on production
- live memory monitoring → IBM Monitoring & Diagnostic tools for Java
  - GC and Memory Visualizer
    - for Websphere → also works on Solaris & HP-UX (i.e. sun jvm syntax)
    - heap details, including heap histogram
    - get dominator tree
    - search inefficiently used data structures & redundant data
    - recommendations
      - increase heap size
      - gc takes between 10% - 20% cpu → tune gc policy
  - Health Center
    - agent in JVM
    - client (IBM Support Assistant)
    - visualize gc
    - recommendations
    - check gc performance
- Memory Analyzer (not from IBM → eclipse.org): diagnose footprint issues:
  - heapdumps
    - automatically on out-of-memory
    - trigger programmatically
    - tool support is essentialµ
- OS-tools
  - track native memory: perfmon, vmstart, ps (os-specific)
  - use gcmv to visualize os-specific dump
  - some native structures have java-wrappers in the heap (e.g. Threads and NIO buffers)

cpu-bound applications

target your optimizations → measure
diagnose:
- method trace:
  - System.out.println()
  - log4j
  - ibm-jvm param -Xtrace:print=mt,methods=(HW*)
- method profiler:
  - health center: low overhead sampler

I/O bound applications

gc and memory analyser → long gc times might indicate paging
method trace for network and disk I/O
os-specific tools

synchronization issues (concurrency)

sample thread dumps
health center:
- lock analysis
- identify contended locks

Devoxx 2009: The not so Dark Art of Performance Tuning

2009-11-19T16:37:00.002+01:00

19/11/2009, Kirk Pepperdine & Dan Hardiker

loadtest <> stresstest:
- loadtest: test user experience
- stresstest: find when the application breaks
performance anti-patterns & solutions:
- lack of stress testing → introduce a stress test harness
  - repeatable test
  - control load
- shot in the dark: fix ugly code
  - ugly code runs just fine
  - measure, don't guess!
Apache JMeter → root node: “Test Plan”
understand the environment, identify dominating consumer → look at the full system
- actors: usage patterns
- application
  - locks
  - external systems
- jvm/os
  - memory
  - hardware management
  - tools e.g.:
    - vmstat on hpux
    - jps
    - jstack
    - visual vm (thread and heap dumps)
- hardware
  - cpu
  - memory
  - disk I/O
  - network
response time budgets:
- detail the time of the different components → add customizable time measures
  - database
  - application server
  - client
common problems, demo's:
- poor response times, low cpu
  - thread pools
  - locks
  - → VisualVM, thread dump + plugin thread dump analyzer, identify lock

Devoxx 2009: Using XML with Java: Sploit for Choice?

2009-11-19T16:34:00.002+01:00

19/11/2009, Michael Kay, Saxonica

Xml interfaces in java, from low level to high level

SAX/Stax
- high performance: +/- 40 MB/sec
- low-level
- SAX/Stax: comparable performance
- use pipelines:
  - SAX: push
    - best for multiple output
  - Stax: pull:
    - best for multiple input
DOM: tree based
- don't use plain DOM, prefer JDOM, DOM4J or XOM – I once experienced problems with the XOM-handling of unicode characters
- uses a lot of memory: +/- 5 times the document size
- complex, tedious usage
databinding: compile to / (un)marshall from java objects
- like JAXB
- fits for simple xml
- not for complex xml or document xmls → create too much java classes
- give 2 views on the same: xml <> java objects
language support:
- doesn't exist in Java...
- scala: XQuery-like capabilities
XSLT/XQuery
- should be your default choice
- good performance: +/- 10MB/sec
- XSLT:
  - more for documents
  - transformations
  - richer functionality
- XQuery:
  - more for data
  - xml-database
  - easier to optimize

Architectures

prefer xml pipelines: (xproc pipelines)xml database → xquery → xslt → rest → xforms
don't mix data: objects & xml

Devoxx 2009: Project Coin

2009-11-18T18:48:00.005+01:00

18/11/2009, Joe D. Darcy, Sun

slides: http://blogs.sun.com/darcy/entry/project_coin_devoxx_2009

Project Coin

small language changes in JDK7
keep broad consideration of java landscape → remain consistent with the Java Language
bigger language changes, not in scope of Project Coin
- JSR308: annotations to improve static checkers (e.g. @NotNull)
- JSR294: modularity

principles of Project Coin:

reading more important than writing
- 'beautiful' code
- don't hide what's happening
simplicity matters
no slash-and-burn: remember broad user-base
balance stability versus progress
prefer library / IDE changes over language changes (String.join)
no new keywords
no type system changes
small in specification, implementation and testing

methodology:

open: blog entries / white board on Devoxx2008
call for proposals
prototypes recommended
- lot of work
- very beneficial → learn / find issues
existing requests for enhancement: 120 changes → too much

final list of Project Coin enhancements

numbers:
- binary notation: 0b00100100001
- group numbers with '_': long big = 1_123_456_789;
switch on a String – really cool!
- specification: only one word added
- proposal contains a suggested implementation:
  - uses String.hashCode() for an initial switch
- big patch in javac
- lots of tests to validate change
  - reject pre -source 7 classes
  - null behavior
  - hash collisions
  - fall-throughs
  - → comparable to enum switch control flow
diamond <> operator : type interference → e.g. Map <String,String> map = new HashMap<>();
- a lot of prototyping work
- quantitative analysis
- some concerns on language evolution were considered
JSR292: invokedynamic for dynamic languages
auto-close of Closable types within a special try block (ARM)
- try(File f = new File) {} → f is automatically closed when leaving the block
simplified varargs method invocation, working along with generics
language support for Lists and Maps:
- List<String> list = [“1”, “2”];
- list[0]
- map{“hello”}

not retained in Project coin:

Elvis operator:
- use ?: as a shorthand for testing on null
- conclusion from building a prototype→ doesn't fit well in java
multiple exception handling
- disjunctive means of | operator
properties:
- changes the type system
- a lot of work (e.g. reflection api, corner cases …) – the 'enum' specification is demonstrated as a comparable change that involved a lot of rework
- pseudo-properties workaround: via annotation-processing → Project Lombok
reified generics
- source language compatibility issues – java generic erasure was chosen for backward compatibility
- C# created a separate generic library
- difficult for other languages to target the a platform with reified generics
extensible enums
- most enums are constant
- low utility

Devoxx 2009: Solr Power with Lucene

2009-11-18T17:59:00.001+01:00

18/11/2009, Erik Hatcher

Lucene core

full-text search library
concepts
- inverted index:
  - term + proximity
- documents
- fields
  - field-ids: e.g. category, title, name...
  - types: number, date, text...
  - unique keys: unique id per document
- terms (aka tokens):
  - processed through filters
    - synonyms
    - ignore words
    - stemming
- scoring relevancy
  - term frequency
  - inverse document frequency
  - field length normalization → control how field length / # occurrences affects scoring
  - boost factors: favor or boost some fields (e.g. titles)
core
- standalone jar
- core index

Apache Solr properties:

search server
based on Apache Lucene
- → Lucene exposed over http
- spell checking
- highlighting
extensible
scalable
- caching
- replication
- master/slave distributed search → sharding
multiple inputs
version 1.4

Using Solr

setup:
- solrconfig.xml:
  - cache settings
  - Lucene indexing parameters
API:
- RequestHandlers:
  - mini-servlets,
  - flexible responses:
    - http GET/POST
    - JSON
    - SolrJ
    - ruby, php, …
    - content streams (must be shielded)
- indexing / deleting a document
  - through api: xml document with commands
  - POST or GET with request parameters
- other actions:
  - commit / rollback: batching document indexing
  - optimize
- search request: simple GET, with optional parameters
  - debug
  - lucene explanation
  - pagination: start / raws
  - score: lucene score
DataImportHandler
- import from RDBMS, xml and e-mail
- incremental indexing
- extensible
- debug console
Solr Cell: uses Lucene Tika:
- index Word, pdf, html ...
- ExtractingRequestHandler
Query parser framework with plugable parsers:
- Lucene syntax:
  - powerful
  - but user-unfriendly syntax
  - exceptions visible to end-users
- Dismax query parser
  - simplified syntax

Advanced Solr: Search Components

standard: query, facet, mlt, highlight, stats, debug
others: elevation, clustering, term, term vector
faceting
- counts subset within results
- group 'facets' of a document (like a category field)
spell checking
pluggable distance algorithms: Levenstein or JaroWinkler
highlighting: custom prefix and suffix → response is highlighted
query elevation → elevate.xml: boost or exclude a document
clustering: grouping of documents into labeled sets
enumerate terms for a field
term vectors: term frequency, document frequency, position, offset
statistics: stats.jsp (in RAM); returns xml
scaling:
- replication:
  - master is polled
  - replicant pulls Lucene index / config files
  - replicate + load balance
- distributed search: single index is too large → sharding

Staring with Solr

agile, iterative process works best:
- basic schema
- bring in data
- check requirement gaps
- adjust solr

Devoxx 2009: Architecting Robust Applications for Amazon EC2

2009-11-18T15:29:00.002+01:00

18/11/2009, Chris Richardson, SpringSource (VMware) www.cloudtools.org

Amazon EC2 cloud computing solution:

choose your virtual machine, with machine images of standard OS's
cloud services
- SQS: Simple Queue Services
- SimpleDB: alternative to relational DB
- S3: storage
- EC2 firewall: security groups
- Elastic Block Storage:
  - replicated, mounted as device
  - snapshots → backup
- Elastic Relational DB Service:
  - MySQL 5.2
  - maintenance window: 4hour/week
  - backup window
standard software: simply install (with yum or apt) your packages
console
local storage:

plenty (160GB → 1690 GB)
ephemeral
slow down on first access
→ Elastic Block Service

easy upgrades:
- clone your production environment
- upgrade the clone
- test the clone
- if OK: put the clone in production
- remove the old clone
regions:
- US
- EU
- Asia
- sub-zones:
  - “availability zones” that are isolated from each other.
  - better availability
  - more expensive

issues:

security:
- not PCI-compliant
- discomfort of putting your core data on the cloud
- where is your data really?
- but:
  - Amazon experience
  - unlimited logging
  - standard best practices for security
  - isolated instances
  - put servers in different security groups → Amazon EC2 firewall
  - encrypt your data and backups
cost:
- expensive for large servers
- bandwidth
- but: no extra cost of cooling, power etc.
machines:
- no very small, very big machines
- maintenance windows
- load-balancing:
  - ipbased: no support for sessions
  - dns (cname)
is it reliable?
- is internal IT better?
- Amazon expertise
- easy to launch a new instance within a few minutes

future:

Platform as a Service:
- complete stack
- simpler scaling, without being involved into the technical details
- Google Application Engine
- Cloud Foundry:
  - Spring & Grails

Devoxx 2009: JDK 7 Update

2009-11-18T13:09:00.002+01:00

18/11/2009, Mark Reinhold

higlight: Closures in java are BACK!

Aims of JDK7

modular platform
- JDK7 status:
  - 13MB download with pack200 compression
  - difficult to support smaller (embedded) hardware
- → Jigsaw project
  - low-level modules, at the rt.jar level
  - language support: module-info.java file
    - explicit module / version requirements
    - create simpler OS-specific package, with correct dependencies (e.g. deb-packages)
  - OSGi interoperability
multi-lingual support
- better JRuby, Jython support on the JVM
- InvokeDynamic
- “Da Vinci Machine project”
productivity enhancement
- “Project Coin”
- language changes that improve programming experience. E.g.:
  - type inference with the <> (aka. 'diamond') operator: Map <String,String> m = new <> HashMap();
  - split numbers with '_' for better readability: int a=123_456_789;
  - support binary notation: int b=0b00100100001;
  - Lists: direct construction and access to set/get with []: a[4] = b[5];
  - Map: direct construction and access to set/get with {}: a{“b”}=”xyz”;
  - auto-close Closeable interfaces: try(File f = new File()) where f is always closed correctly.
performance
- multi-cores are there! → java support needed.
- → fork/join framework
- Dual-Pivot Quicksort
- Closures: “It's time to add them” – I thought this was dropped from Java7 plans...
  - needed:
    - function literals
    - function types
    - extension methods
  - “doesn't fit in java”:
    - non-local control transfer
    - capture non-final vars
    - library-defined control structures

When?

Planning shifted
Milestone 10: 2010/09/09: release candidate

Devoxx 2009: Keynotes 18/11/2009: Parleys.com, Oracle, Sun (JEE6) and Adobe

2009-11-18T11:41:00.004+01:00

Stephan Janssen: Devoxx organizer,Parleys.com founder

presented his new Parleys v3
- support for 'spaces' and channels in spaces → propose parleys.com as a platform
- monetize:
  - subscription for impatient viewers (i.e. for immediate access to the talks); talks with free access are provided within a year
  - give private spaces to companies for education purposes
- still presented in the over-Flashy way
- I hope they'll get the podcast feed – finally – fixed.

Oracle slot

mostly boring management talk, with lots of empty words
WebLogic DM:
- heavily based on OSGI
  - 'Profiles' group technical bundles
  - 'Bundles' are the application-specific modules
- possibility to run on a “bare-metal” hypervisor VMWare: http://en.wikipedia.org/wiki/Hypervisor– new concept to me

Sun:

JEE6
- final release scheduled for 10/12/2009
- → enterprise development made simpler
- basic contents (a.o.)
  - new stuff:
    - JAX-RS: annotation-based RESTfull API
    - Bean validation, used in the full stack (JSF and JPA)
    - Dependency Injection (DI) 1.0
  - updates:
    - EJB 3.1
    - Servlet 3.0
    - JSF 2.0
    - …
- support for technical profiles: set of basic, but quite complete JEE components.
  - e.g. WebProfile: Servlet 3.0, JSF, EJB 3.1 lite, DI
- pluggable, modular web applications
  - web.xml is now optional
  - web-fragment.xml
  - annotations:
    - @WebServlet
    - @WebFilter
  - resource jars → finally a standard way to package static resources
  - programmable registration API
- Dependency Injection 1.0:
  - @Resource
  - @Inject → Guice-like standard solution
  - Inject metamodel
- EJB3.1
  - @Singleton
  - @Startup
  - @Asynchronous
  - EJBContainer API, useable in Java SE (for testing)
  - EJB's, directly useable in webapps
- JSF 2.0
  - standard facelets
  - autodiscovery of component libraries
  - Ajax & partial view loading
  - JavaScript API
  - composite components
Glassfish demo
- Eclipse & NetBeans support
- not sure how Oracle will handle the GlassFish versus WebLogic proposal
- great programmers' experience:
  - fast startup
  - incremental deploys applied to a running application without restarting → JRebel-like feature.

Adobe stuff

skipped this one – too much flash for me :-)

Devoxx 2009: A Year of Monitoring with Java-Monitor

2009-11-17T17:31:00.011+01:00

17/11/2009, Tools in Action, Kees Jan Koster

Slides are available at: http://java-monitor.com/forum/showthread.php?t=646

Case-by-case approach from experiences reported on java-monitor.com, from an Operations' perspective.

Memory Leak:

diagnose:

Scavenger gc gives up and stop the world gc (= mark and sweep gc) takes over
monitor # scavenger gc's versus mark and sweep gc's

solution:

profile your app and fix your app
workaround: increase heap

Misbehaving GC

problem: System.gc() in code freezes the JVM!
diagnose:

many stop the world gc's (= mark and sweep gc)
low heap usage (e.g. 20%)

solution:

workaround: -XX-DisableExplicitGC
use findbugs and fix code

Native threads in 32 bit environments

problem:

OutOfMemoryException: Couldn't allocate native Thread
32bit Systems share 4GB with OS Kernel, JVM (heap + perm space) and thread stack space.

solution:

workaround: -Xss limitThreadStack
migrate to 64 bit!

In verboseGC: "promotion failed"

problem:

verboseGC logs "promotion failed" --> stop-the-world GC (mark and sweep) is triggered
CMS is non-compacting: the heap can end up to be fragmented. Fragmentation can prevent promotion of young heap (eden spaces) to old heap.

solution:

bigger heap
no real solution

File Descriptors

problem:

"too many open files"
files not properly closed on exceptions --> file description leakage
GlassFish opens file-descriptors for each static file served (efficient usage of FileChannels)

solution:

check ulimit / limits of your OS
fix your application (findbugs)

others

hyperthreading doesn't make any significant difference
use the magic "-server" switch

Devoxx 2009: Full Text Search for Hibernate

2009-11-17T16:24:00.004+01:00

17/11/2009, University sessions, Emmanuel Bernard

Search solutions:

categorize upfront
show detailed search screen
use single search box (preferred)

Plain SQL search limits:

performance: like '%...%' causes a full table scan
no support for approximation nor synonyms
no proximity concept
lacking relevance scoring
no simple multi-column search

Full-text search solutions:

word based
captures / indexes frequency and position
solutions:
- RDMS: (like Oracle Text):
  - less flexible
  - not portable (vendor-specific API and behavior)
- standalone: Lucene
  - text-only
  - no synchronization with model objects

Hibernate Search, general features:

LGPL
uses Hibernate core
uses Lucene under the hood
solves object vs text mismatch
convert object to text document (+reverse) → Hibernate application uses objects, not text
documents
convention over configuration
heavily built on annotations
Optimize Lucene access:
- update Lucene docs on commit
- object graphs are consolidated to single Lucene docs to provide relevant searches
- avoid flooding Lucene indexer:
  - batch Lucene updates on commit
  - optionally trigger the Lucene indexer asynchronously
- support clustering (JMS)

Hibernate Search Annotations:

@Indexed
@Field: tunable how to convert to text with, among others, @FieldBridge. E.g. convert number to 0-padded number.
@IndexedEmbedded
@Boost: promote a particular field in the relevance score (can be at indexing time or at query time)
@Analyzer: e.g. anagram-support

Lucene Index as used by Hibernate Search:

event based
batches updates per transaction (=at commit time)
sync or async mode (optimize Lucenes' locking mechanism)

Query:

HQL
Full Text (Lucene syntax) e.g. with the ~ opperator
JPA2 criteria
native SQL
→ always returns Objects, not Lucene documents.

Advanced stuff:

tokenizer: split text in words, remove common words
complex searches: combination of indexing and querying
fuzzy search:
- “Levenstein distance”: quantifies similarity
- “n-gram”: word is split in groups of 3 letters → matching groups determines score. (demo looked a bit hackery)
phonetic search (soundex-like): disappointing in practice
synonyms: use your application-specific list
stemming: → 'reduction'
- Porter Algorithm
- Snowball stemmer
filters: provide efficient an pluggable support for
- security, categories, temporal data, caching...
“explain” query result

clustering / Scaling Lucene
- one Lucene writer at a given time
- use a JMS queue for indexing (→ 'Master')→ small delay, but very scalable.
- Distributed in-memory index (Infinispan 4.0) – technical preview
- index optimizations:
  - sharding
  - defragmenting or re-indexing

Devoxx 2009 – Google Application Engine - A Real Live Voyage to The Cloud

2009-11-17T12:49:00.004+01:00

17/11/2009, University Sessions, Sam Brodkin and Scott Stevenson

Google Application Engine for Java (GAE/J) is a cloud computing solution that provides

a JVM
OS support
access to the Servlet API
a management console
access to Google utilities:
- Logon / Sign up → com.google...User
- image service: resize images
- mails
building blocks:
- DataStore
- MemCache

Cost model is based on actual usage, with a free quota for easy starting:

bandwidth usage (in/out)
cpu usage
datastore size
emails sent

Management console:

access to logs
shows bandwidth / cpu usage → cost
users with admin role
statistics
manage multiple versions: one default, and many test versions (beware of data-incompatibilities)

Limits of GAE/J:

requests timeout: 30sec
no background processes allowed
no server push
no threads
read-only file-system
no transaction
only access to white-listed classes (e.g. ImageIO class of java is not accessible → use Google Image service)
a maximum of 3000 static resources per application
Migrating your application off the GAE/J cloud: difficult, but not impossible → google for “GEA Bar”

DataStore for data persisency

not a relational database
BigTable → like a big HashMap
no transaction
JDO or JPA access

Development Environment:

eclipse plugin
Jetty based emulator → fair emulation, but not 100% relevant
MemCache
disk-based datastore
compile-time checks
no maven support yet
DIY ant scripts are possible and recommended.

Demo: “swagswap.org”

3 tiers:
- DAO
- service layer
- 4 frontends that use the same
  service layer:
  - jsf 2.0 (+IPhone version)
  - gwt 1.7
  - SmartGwt
  - Spring MVC
wired through Spring 3.0
get detailed cost of a request by setting a special HTTP header

Quirks:

BLOB type
Text type for long Strings
white-listed classes
- → find workarounds for blocked classes (e.g. Image services)
- → some thirdparty libraries don't work
JSF2.0:
- disable threading in web.xml
- no JNDI: hack WebConfiguration class
- server side state doesn't work → use client side state
- RichFaces, IceFaces and Commons FileUpload use non-whitelisted classes → incompatible with GAE
GWT:
- BLOB and Text are non-serializable
- GWT <> Spring integration requires a special service class
SmartGWT:
- requires 1000 static resources → closer to the max of 3000 of GAE → workaround with zip-file and cache, but increases cpu-cost

Sound in Ubuntu 9.10 - Intel 82801G

2009-11-11T22:38:00.005+01:00

The sound on my Medion MD96290 didn't work after upgrading to Ubuntu 9.10 (Karmik Koala). I experience this glitch on every Ubuntu upgrade - see my previous post.

My sound chip:

$ sudo lspci -v
00:1b.0 Audio device: Intel Corporation 82801G (ICH7 Family) High Definition Audio Controller (rev 02)
Subsystem: Wistron Corp. Device 4071
Flags: bus master, fast devsel, latency 0, IRQ 21
Memory at d8440000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [50] Power Management version 2
Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [130] Root Complex Link
Kernel driver in use: HDA Intel
Kernel modules: snd-hda-intel
...

Fixing is easy:

gksudo gedit /etc/modprobe.d/alsa-base
add one line to the end of the file:
options snd-hda-intel model=6stack-dig
save and reboot
rightclick on the volume icon and select Sound Preferences
in the hardware tab, select a profile with "Surround" in it.

update 6/6/2010: this tip also works for ubuntu 10.4.

Tweaking the Full-screen Mode of Firefox

2009-11-08T14:42:00.004+01:00

I like to use Firefox in full-screen mode (hotkey: F11) on my netbook. Unfortunately, Firefox hides the tabs and navigation bar and only shows the bar when your mouse pointer is on the top of the screen.

Luckily, it's easy to modify this default setting:

enter 'about:config' in the URL-field
confirm that you really will be careful
set the 'browser.fullscreen.autohide' setting to false (double-click on the line)

This gives me a Chrome-like browser experience with Firefox on Ubuntu, while maximally using the small screen size.

Getting a Digital Camera working under Linux

2009-08-09T21:43:00.005+02:00

My daughter bought a simple Toy Digital Camera. Unfortunately, it doesn't register itself as a universal mass storage device to the OS. The 'lsusb' command returns

Bus 002 Device 004: ID 2770:905c NHJ, Ltd Che-Ez Snap SNAP-U/Digigr8/Soundstar TDC-35

Luckily, retrieving the photos from the camera is very easy. Install the gtkam GUI:

sudo apt-get install gtkam

And start the program from Applications / Graphics / Gtkam. In the Camera / Add camera ... menu, you can press "detect" to register your camera. The application is self-explaining for downloading / deleting the photos.

http://ubuntuforums.org/showthread.php?p=2952334#post2952334

Sunny Beam under Ubuntu: SunnyBeamTool v0.6.0

2009-06-12T00:48:00.008+02:00

http://sourceforge.net/projects/sunnybeamtool/

The sbtool Perl script of Stefan Arts provides a nice basic setup for accessing the Sunny Beam under Linux. But the Perl solution has some drawbacks: it is quite complex to set up and the codebase, with low-level hex-manipulations, is rather complex to extend, at least for an occasional Perl-developer like me...

With this in mind I translated the sbtool into C language and put it on sourceforge:

http://sourceforge.net/projects/sunnybeamtool/

The basic functionalities are included, but additional feedback, testing and extensions are welcome. Of course use at your own risk ! :-).

$ sudo ./sunnybeamtool
Dev #6: 1587 - 002D -SMA Technologie AG -
Serial Number: 00024383
pac: 0.00 W
e-today: 12.37 kWh
e-total: 1551.83 kWh

$ sudo ./sunnybeamtool -l
Dev #6: 1587 - 002D -SMA Technologie AG -
Serial Number: 00024383
pac: 0.00 W
e-today: 12.37 kWh
e-total: 1551.83 kWh

today:
...
2009-06-11 13:00:00: 756 W
2009-06-11 13:10:00: 888 W
2009-06-11 13:20:00: 569 W
2009-06-11 13:30:00: 205 W
...

last month:
...
2009-06-07: 11.790 kWh
2009-06-08: 10.060 kWh
2009-06-09: 8.480 kWh
2009-06-10: 12.370 kWh

This commandline is much easier and cleaner than running the full blown complex Sunny Data Control application under windows.

If your Sunny Beam doesn't respond to the USB-communication, try this: unplug the USB cable, switch a few times between daily graph and monthly graph, go to the menu and select 'exit', plug the USB cable. This should 'unfreeze' the USB buffer of the Sunny Beam. I never had to remove the batteries from my Sunny Beam device during all my experiments.

If you're still having trouble, send the output of 'sudo sunnybeamtool -l -v' in your bug-reports on sourceforge.

Other background info: I kept the libusb approach, as this allows the application to run in user mode, outside of the kernel. The development is done wih Netbeans 6.5 and Subversion as development platform.

Using the Sunny Beam under Linux: first steps!

2009-05-28T22:02:00.012+02:00

update 13/6/2009: check out my new sunnybeamtool command line application.

In an earlier post, I discussed the missing Linux drivers for the Sunny Beam device. In the 'comments' section of that post, Stefan Arts dug deeper into the Sunny Beam protocol by 'snooping' the USB-communication with SnoopyPro. His findings were put in a very basic perl script on sourceforge. In order to run it I had to add some packages to my ubuntu 9.04 install:

sudo perl -MCPAN -e 'install +YAML'
sudo perl -MCPAN -e 'install Inline::MakeMaker'
sudo apt-get install libusb-dev
sudo perl -MCPAN -e 'install Device::USB'
sudo perl -MCPAN -e 'install Digest::CRC'

running:

sudo ./sbtool

Unfortunately, I didn't get any data back.

Update 1/6/2009:
The messages sent to the Sunny Beam have a device-dependent 'destination' field. Stefan suggested me to replace one line in the sbtool script:

my $CMD_GET_DATA="\x7e\xff\x03\x40\x41\x00\x00\xd4\xf5\x10\x00\x0b\x0f\x09\x00\x69\xda\x7e";

And, I get a successful response now:

$ sudo ./sbtool
Sunny Beam, SMA Technologie AG
Serial number: 00024383

Pac: 270 Watt
E-Today: 16.38 kWh
E-Total: 1427.98 kWh

3/6/2009:
Stefan now added CRC support and 'destination parsing' to his '0.0.4' version, so his sbtool script should work on any sunny beam device.

Cool, isn't it? Next steps:add the 'get daily details' and 'get last 31 days' calls to the script.Update

On a parallel track, I also noted that Alexander Morozov provided some USB-support to wine. This could potentially allow a Windows driver to be emulated in Wine. The fix isn't merged into the git source repositories and I couldn't evaluate it with the Sunny Beam USB driver yet.

Great to see this area moving forward :-).

background info:

Details of the SMA protocol: http://256.com/solar/scripts/smadat-11-ze2203.pdf
recap of stefans' findings:

data sent: 7e ff 03 40 41 00 00 ff 00 10 00 0b 0f 09 00 07 ac 7e
7e = start flag
ff = address byte (broadcast)
03 = a bit coded HDLC control field, 0x03 stands for unnumbered data blocks
40 41 = protocol used (SMA data telegram)
00 00 ff 00 10 00 0b 0f 09 00 = data telegram content:
00 00 = destination address
00 ff = source address
10 = control 'STRINGFILTER'
00 = packet counter
0b = command 'CMD_GET_DATA'
0f 09 00 = content:
09 0f = channel type
00 = channel Idx
ac 0f = frame check sequence, 16 bit check sum
7e = stop flag
data received: 7e ff 03 40 41 ff 00 00 00 50 00 0b 0f 09 00 01 00 f1 cb 09 4a 01 00 00 00 00 00 70 42 29 5c d7 40 ae 47 65 41 be d9 7e
7e = start flag
ff = address byte (broadcast)
03 = a bit coded HDLC control field, 0x03 stands for unnumbered data blocks
40 41 = protocol used (SMA data telegram)
ff 00 00 00 50 00 0b 0f 09 00 01 00 f1 cb 09 4a 01 00 00 00 00 00 70 42 29 5c d7 40 ae 47 65 41 = data telegram content
00 ff = destination address
00 00 = source address
50 = control 'ANSWER'
00 = packet counter
0b = command 'CMD_GET_DATA'
0f 09 00 01 00 f1 cb 09 4a 01 00 00 00 00 00 70 42 29 5c d7 40 ae 47 65 41 = content:
09 0f = channel type
00 = channel Idx
00 01 = number of data records: 1
f1 cb 09 4a = time/date in unix format
01 00 00 00 = time basis (1 sec)
00 00 70 42 = Pac (floating point, 60 WATT)
29 5c d7 40 = E-today (floating point, 6.7 kWh)
ae 47 65 41 = E-Total (floating point 14.3 Kwh)
be d9 = frame check sequence, 16 bit check sum
7e = stop flag
it sends the data string:
7e ff 03 40 41 00 00 ff 00 10 00 0b 0f 09 00 07 ac 7e 01 00
a result was:
01 60 7e ff 03 40 41 ff 00 00 00 50 00 0b 0f 09 00 01
01 60 00 64 33 0b 4a 01 00 00 00 00 00 00 00 ec 51 00
01 60 41 33 33 b2 41 68 5e 7e
where

64 33 0b 4a = 4a0b3364 HEX = 1242248036 DEC = Wed, 13 May 2009 20:53:56 GMT
00 00 00 00 = 0
ec 51 00 41 = 8.02
33 33 b2 41 = 22.4

Fixing a Readonly SD Card Reader in Ubuntu

2009-02-08T13:46:00.007+01:00

On one of my laptops running Ubuntu I noticed that my SD card was always mounted in readonly mode. My other Linux-powered laptop mounted the same SD card just fine in read/write mode. As a workaround, I could remove photo's directly from the camera or from the other laptop, but I was intriged and decided to delve deeper into this issue.

A few relevant commands stumbled upon thanks to Google:

$ dmesg | grep -i sdhci [ 15.122623] sdhci: Secure Digital Host Controller Interface driver [ 15.122627] sdhci: Copyright(c) Pierre Ossman [ 15.194747] sdhci-pci 0000:0a:09.1: SDHCI controller found [1180:0822] (rev 19) [ 15.194768] sdhci-pci 0000:0a:09.1: PCI INT B -> GSI 18 (level, low) -> IRQ 18 [ 15.197935] mmc0: SDHCI controller on PCI [0000:0a:09.1] using PIO $ lsmod | grep -i sdhci sdhci_pci 15360 0 sdhci 23940 1 sdhci_pci mmc_core 58268 2 mmc_block,sdhci $ lspci | grep -i ricoh ... 0a:09.0 FireWire (IEEE 1394): Ricoh Co Ltd R5C832 IEEE 1394 Controller 0a:09.1 SD Host controller: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter (rev 19) 0a:09.2 System peripheral: Ricoh Co Ltd R5C843 MMC Host Controller (rev 0a) 0a:09.3 System peripheral: Ricoh Co Ltd R5C592 Memory Stick Bus Host Adapter (rev 05) 0a:09.4 System peripheral: Ricoh Co Ltd xD-Picture Card Controller (rev ff) $ tail -f /var/log/syslog ... Feb 8 13:48:51 michael-laptop2 kernel: [ 3910.801967] mmc0: card lacks mandatory switch function, performance might suffer. Feb 8 13:48:51 michael-laptop2 kernel: [ 3910.802543] mmc0: new SD card at address b368 Feb 8 13:48:51 michael-laptop2 kernel: [ 3910.805647] mmcblk0: mmc0:b368 E0421 501248KiB (ro) Feb 8 13:48:51 michael-laptop2 kernel: [ 3910.806012] mmcblk0: p1 Feb 8 13:48:51 michael-laptop2 hald: mounted /dev/mmcblk0p1 on behalf of uid 1000 $ cat /etc/mtab /dev/mmcblk0p1 /media/disk vfat ro,nosuid,nodev,uhelper=hal,shortname=mixed,uid=1000,utf8,umask=077,flush 0 0

After a long search on google and in my logs, I couldn't find any clue. The SD read-only-switch was correctly turned off ; my tests on the other laptop confirmed this. Finally I rebooted my laptop under Vista and ... the SD was also mounted read-only. Conclusion: it was not an OS / driver problem and the problem was related to that PC. I then simply blow some air with a "dust remover" into the SD slot and the magic happened: the SD card mounted in read write mode... A trivial solution to a seemingly complex problem.

Devoxx: Developing Beyond localhost by Brian Leonard

2009-01-02T15:07:00.009+01:00

Brian demoed how combining VirtualBox and ZFS can provide a very flexible and dynamic development and testing platform for web-developers. VirtualBox is a simple, free yet powerful virtualization tool. It allows you to setup very easily virtual machines having multiple OS's and configurations. You don't even have boot those virtual machines over and over again: a snapshot can be saved, and restored almost instantly. After setting up several virtual machines, you quickly get a relevant set of client-machines on which you can test your webapp. All virtual machines get a unique ip address, so connecting to the web server from you virtual client is as simple as finding out your local IP serving the web app. Since all vm's are simple files, you can backup the 'clean' vm's and start playing around with them. Restoring a vm is as simple as restoring the original snapshot file. Virtual Systems can run windows outside of the 'vm-box', or on contrary, you can run the Virtual machine full-screen; that way it's easy to forget that you were actually playing in a virtual environment.

I really do like the Virtual machine principle and Virtual Box is a pretty neat solution -- for free! The issues however start with closed OS's: Windows e.g. requires new license keys and Mac OS doesn't run on Virtual Box.

The ZFS (file system) is a crown-jewel of Sun. Its feature-set (security, backup ...) really stands out compared to legacy NFS, ext2/3 etc. In ZFS creating a volume is almost as cheap as creating a directory. Brian used this on his OpenSolaris laptop to create a dedicate ZFS volume under his home directory for Virtual Box snapshots. ZFS-volumes can be configured to take regular snapshots, or to take the snapshot on request. After messing around with his Virtual Box snapshots, he simply undoed everything with a single ZFS "restore snapshot" command in only a few seconds. Truly amazing.

I really think ZFS should be supported in Linux, although the CDDL license is not compatible with GPL v2. There is a zfs-on-Fuse project (=file system in userspace), but it is still in beta fase and performance should be lower than native kernel support. Although the excellent ntfs-support (ntfs-3g) also uses fuse with good performance.

references:

Devoxx: JRuby by Charles Nutter and Thomas Enebo

2009-01-02T14:52:00.005+01:00

Charles and Thomas did a good job in getting the java-audience in touch with ruby's beauty. The demos looked really slick and development seems very efficient and intuitive. I still have the impression though that ruby works best if you completely adhere and embrace ruby's principles. The trouble starts when you want to take slightly different approaches than the 'sensible defaults'.

Some (J)Ruby's features:

closures
operator overloading
Ruby Gems is the repository
'module' = mixins of scala

Ruby on Rails web development platform:

convention over configuration
DRY don't repeat yourself
'agile': hot deploy, generate basic structure
persistence: ActiveRecord (=driven by DB structure)
rhtml + partial rhtml = (sort of similar to) jsp and jspf
multi-format support: e.g. xml and html based on http accept header
warble: create standard WAR based on bundle

examples of sites using JRuby:

references:

Devoxx: Introduction to Asynchronous I/O (NIO.2) using the Grizzly Framework by Jean-Francois Arcand

2009-01-02T14:38:00.004+01:00

I was hoping to get a general 'New NIO' overview, but Jean-François concentrated most of his talk discussing low-level Asynchronous IO usage patterns in Grizzly using New-NIO:

waiting for Java 7
'Selector' and Notify principle
'CompletionHandler' that uses a 'Future' to listen to the socket without wasting threads
thread pools: fixed, cached thread pools built-in to the kernel (but: risk of blocking the kernel!)

Other New NIO features in java.nio.file:

FileSystem
FileRef for having native symbolic links in Java
Path
FileStore

references:

Devoxx: Towards a dynamic VM by Brian Goetz and Alex Buckley

2009-01-02T13:27:00.006+01:00

The JVM and the Java Language should be treated as two decoupled things. The JVM provides the basic infrastructure on which the Java Language builds: Garbage collection, reflection, security, Hotspot ... All those things make the JVM appealing to other languages like JRuby, Jython, Groovy and Scala. Choosing for the JVM as a platform has also tradeoffs like for example the fact that the JVM is stack based (vs. the registry based VM in for example the Davlik VM of Android).

Dynamic language on the JVM problems and solutions (mostly focusing on JRuby). All this is quite low level for me, I just wrote down some keywords I might investigate further some day:

'Open classes' --> JSR 292 invokedynamic =consider argument types at runtime = invokevirtual optimized
overhead of reflection and BigInteger
'inline caching' trick to cache 'Method Objects'
'invoke interface'
tail calls
continuations
tuples
value objects

references:

http://www.devoxx.com/display/JV08/Towards+a+dynamic+VM
concerning register-based vs. stack-based vm's:

Devoxx Keynote: Java SE 7 Update by Mark Reinhold

2009-01-02T11:08:00.013+01:00

In the online schedule this item is titled differently: 'The Modular Java Platform'. Anyway, Mark talks about both the Modular stuff and Java SE 7 features.

Concerning Modularity in Java 7:

Java 6 Update 10 couldn't add any features, but managed to speed up download and startup times thanks to, amongst others, the Java Kernel and pack200 compression. Still startup time can't compare to other more targeted environments like python for example.
(*) JSR 277 is on hold (frozen?). I suppose there was too much resistance from the OSGI camp as being competing technologies. I do think that explicit language-support for modules is needed in Java, but this is JSR-294's sweet spot.
(*) JSR 294 will contain all language modularization efforts for Java 7
(*) The scope of the 'Jigsaw project', which seems to run outside the JPC, wants to differentiate from OSGI by having different more low-level objectives in applying modularization to the JRE:

allow the java kernel itself to be modular: e.g. distribute targeted Java 'Profiles': Java FX, applets, full SE...
modularizing the kernel might be a solution to all the deprecated APIs in Java that still have to be provided for backwards compatibility. A possibility could be to provide a basic Java profile without the 'old' classes, and allow a simple module to extend the Java Runtime in order to add the deprecated stuff when needed.
integrate with OS package systems: cfr. JPackage project creates rpm's from jars. --> how to package / deploy reusable modules / JVMs?
what about version collisions?
integrate with the VM, language and native packaging
allow multi-module packages
support "friend" packages (for tighter integration)

Java SE 7 update:

JSR 292: better JVM support of dynamic language (JRuby, Jython ...)
JSR 203: New NIO: FileSystem, access to symbolic links ...
small language changes, too be discussed / decided --> Joe Darcy

safe rethrow
null dereference: shorthand to do chained tests on null in one line:
a?.b()?.c()?.d()
type inference
multicatch

others:

JSR 296: Swing application framework
upgrade Java 6 Update 10 to Java 7
update supported protocols
enhance classloaders
Unicode 5.0 support
improve HotSpot performance
new G1 garbage collector
compressed 64 bit pointers (use less memory)
support "Multiple VM's" (MVM) --> run light, isolated apps like Java FX applets
JSR 308: standard annotations eg. @NonNull used for static analysis
concurrency update: collections, fork/join ...

What will not be included in Java 7:

Closures: prototypes exist, but too early to dump in java
properties
beans binding: need more time to see if this dependency injection framework fits well in core java
operator overloading

references:

Parleys.com now published Mark's talk:

(*) modified thanks to comments / further reading...

Devoxx: Caching web contents in the browser by Ignacio Coloma

2009-01-02T10:59:00.003+01:00

Always nice to refresh basic web skills like caching:

last-modified http header
e-tag header
cache-control header of http1.1
http 304 'not modified' indicator
refresh vs shift-refresh --> the latter ignores the cache

Reference:

http://www.devoxx.com/display/JV08/Caching+web+contents+in+the+browser

Devoxx: From Concurrent to Parallel by Brian Goetz

2009-01-02T10:04:00.006+01:00

The trend of getting faster processors every year just stopped a few years ago. Modern CPUs now get more cores, but each single core is actually slower than its single-core ancestor. Buying new hardware won't automatically make your software run faster anymore, unless your software is built for parallel processing. The more cores are added to the cpu, the more important it becomes to find fine-grained parallelism in your apps, in other words, to split coarse-grained request into smaller chunks.

Java 5 brought the basic concurrency support beyond Threads into Java: Futures + Executors. Those are very powerful concepts, but the resulting code is complex to maintain. Java 7 builds on the existing java.util.concurrent foundation by providing the 'fork-join' framework to Java (JSR166).

The concept behind the fork-join framework is "divide and conquer": a job is recursively split in halves, until a manageable size for the sub-jobs is found. The correct size for the chunks is typically determined by the number of cores (=Runtime.availableProcessors()). Another key concept is "work stealing": if one thread finishes its job early, it 'steals' work from other 'double-ended queues' (Deque) of other threads. That way all cores can remain busy while limitting congestion for the queue.

Java 7 java.util.concurrent.forkjoin:

basic fork join classes: RecursiveAction, AsyncAction, CyclicAction ...
higher abstraction: ParallelArray offers parallel functions out of the box: filter, map, replace, aggregate ... I like the declarative style use for this class: it chains declarations before firing the action: students.withFilter(isSenior).withMapping(selectGpa).summary();

Definitively an interesting step for Java, although once you enter a multi-threaded environment, you loose many safety guarantees from Java and you as a developer has to ensure that concurrency doesn't break your app. The growing number of cores puts parallel programming research in the spotlight, with the renewed interest in functional languages like haskell, erlang and Scala for example.

References:

(very small) intro http://www.devoxx.com/display/JV08/From+Concurrent+to+Parallel
jsr-166: http://www.infoq.com/news/2008/03/fork_join
research paper: http://gee.cs.oswego.edu/dl/papers/fj.pdf
IBM: http://www.ibm.com/developerworks/java/library/j-jtp11137.html
IBM: http://www.ibm.com/developerworks/java/library/j-jtp03048.html

Devoxx Keynote: RFID and IBM

2009-01-02T09:31:00.005+01:00

Devoxx experimented for the first time with RFID tags to monitor talk attendance. In fact, every Devoxx access-badge was an RFID tag and each room had an RFID reader on its entrance that monitored each individual RFID passing through.

IBM presented their backend processing infrastructure to capture, process and present the RFID information at Devoxx. Looked really very complex -- not sure if such a complex setup was needed or if it was just a way to demo their Tivoli monitoring infrastructure.

I was more interested by the RFID-technology itself:

the used RFID readers had a 5 meter detection range, but they stilled missed about 20% of all tags.
depending on the frequency used, some RFID tags can be 'shielded' from the readers by water. This can explain the 20% error range because a human body mostly consists of water.
the RFID readers can't tell if you're entering or leaving a room. This is deduced by the backend system by correlating the different individual tag-readings. E.g. if a tag is detected in room A and then in room B after 30 minutes, it supposes that you were out for those 30 minutes
RFID tags exist in several types, each with its own usage patterns and pros and cons:

'active': the tag has its own battery and radio
'active backscatter': the tag has a battery embedded
'passive': no battery nor radio

RFID tags come also in different form factors:

single use: as 'printed' labels. This was used for the Devoxx badge
rugged reusable tags.

the RFID readers are actually quite advanced systems that contain a built-in JVM. This provides opportunity to do some basic processing when retrieving the scanned data.
the classical questions on privacy / security / health remain.

It was an interesting experience to get 'live' in touch with the RFID technology, its pros&cons and its technical and ethical issues.

Devoxx Keynote: JavaFX - The Platform for Rich Internet Applications by Danny Coward

2009-01-02T08:45:00.010+01:00

Java FX version 1 is out now and looks promising. Although it hits version 1 I don't think it is really ready for prime time:

no Linux (or OpenSolaris ;-) support yet. Really a missed opportunity to get 'Java' in a more positive spotlight in the Linux users' mindset.
native Mac and Windows video formats are supported, but to get real cross-platform media, FXM must be used (http://java.sun.com/javafx/faqs.jsp#19). Supporting native video formats but claiming full cross-platform support seems contradictive to me.
I could still see some small glitches like small GC-pauses and startup-waits. Not sure if those are relevant or just demo-quirks.
the promise of 'multiple devices, one language' is still very vague to me. I have no clue how full screen apps can run great on small phones without any change.
licensing is still blurred. Although Sun is committed to opensource the Java FX core (http://openjfx.org). Real open source of Java FX would be a big plus compared to Flash, especially for the Linux distro's, but I doubt Sun will be able to remove all encumbered stuff like the multimedia codec's.

I'm not really a "shiny flashy apps" developer, but once Linux support is there I might delve deeper into Java FX.

References:

Devoxx: Profiler, the better debugger? (Heiko W. Rupp)

2008-12-31T15:21:00.006+01:00

Understanding existing 'legacy' java applications is not always simple. Especially if the original developers left the company and you have to fix a bug or apply some enhancements. Dependency injection frameworks (Spring) and other very dynamic factors like Hibernate make it very difficult to quickly grasp the structure of an application and understand where the 'core business' resides. Traditional tools like a debugger do help, but their view is often too fine-grained or they cause timeouts, breaking the normal program flow.

A better alternative to understand an opaque java application might be to use a profiler. JProfiler is my favorite, other choices are JProbe or the embedded profiler of NetBeans. When running your application against a profiler, you don't hit timeouts and you quickly get a very useful and in-depth view of the application: cpu an heap usage statistics with detailed call tree graphs immediately point you to the relevant 'hot' parts of the application.

Conclusion: profilers can be a welcome addition to debuggers and other analysis tools when inheriting a complex codebase. This is the first time that I look at profilers this way, and I must admit, having done many JProfiling sessions that you quickly get an indepth view in the most obscure (generated) code with these tools.

References:

intro: http://www.devoxx.com/pages/viewpage.action?pageId=1377607

Devoxx: Creating performance test data with the benerator (Volker Bergmann)

2008-12-31T15:11:00.006+01:00

The Benerator tool focuses on a single problem: generating testdata. Although a simple problem to start, getting good testdata is tough:

testing extremes
using anonimized production-data
applying complex validations on inputdata (e.g. realistic names, gender, birth date, addresses,...)

Benerator has following advantages:

multiple input/output formats supported: xml, csv, database
highly customizable
free (GPL)

I've never used such testdata generation tools, but this is definitely tool I might look at in future testing work...

Resources:

Devoxx: GlassFish in Action: forget the app server for a minute!

2008-12-31T14:55:00.003+01:00

Speakers: Linda Schneider, Jean-Francois Arcand, Paul Sandoz, Alexis Moussine-Pouchkine and Bogdan Stefanescu

This presentation showed an overview of the Glassfish V3 'prelude' features:

core: Apache Felix (OSGi)
hot deploy
embedded API starts <1>
WebSynergy: Portal/portlet technology
Grizzly: Async IO (with 'NIO') as an efficient http frontend to glassfish (+AJAX and Comet support)
Metro 1.4 (WebServices)
Jersey: JAX-RS REST support
OpenMQ: JMS provider
Open ESB

Resources:

Devoxx: Java Performance (Kirk Pepperdine and Holly Cummins)

2008-12-31T14:00:00.007+01:00

In this University session of Devoxx, Kirk Pepperdine and Holly Cummins detailed java performance issues and optimizations. Kirk concentrated on the Sun JVM while Holly detailed the IBM JVM.

Tuning Java applications for performance covers the following areas:

technology: specific JVM details, garbage collector ...
tooling: visualvm, vmstart, verbose gc ...
methodology: identify goals, measure production data, benchmark ...

Java applications run on a dynamic environment. This is a huge advantage as the HotSpot compiler can adapt and optimize the runtime according to production-time measurements, usage scenarios and available hardware. Static languages like C and C++ don’t have that possibility and can only be optimized at compile time. With this dynamicity in mind, one investigating Java Performance must investigate the complete ‘box’, which consists of these layers:

Actors: batch jobs, users, peak usage …
Application: locks (locks are not always bad: they can preserve CPU), static code, libraries …
JVM / OS: Sun JVM vs IBM, thread implementation …
Hardware: memory, cpu, optimized instruction sets …

--> look for the dominant element of performance issues (=break down).

Some tools to investigate performance:

OS: vmstat, mpstat, corestat …
JVM: -verbosegc (even on production), gchisto, hpjmeter
specific IBM “monitoring and diagnostic” tools (beta)
java –xtrace: … for simple ‘entering/leaving method’ style logging, built in the jvm
profilers (JProfiler is my favorite)
visualvm
loadtesting tools: JMeter and Grinder (read on)

Although standard Garbage Collectors run well out of the box, they often require optimizations and tuning once ‘real life’ applications run. When using “Generational collectors”, tuning is typically done with visualvm by modifying the eden, generational and old heap spaces at the JVM command line.

Load-testing Java applications:

<> stress test: load tests an application using production-like load, while a stress test investigates the breaking point of an application.
have a production-like setup
gather relevant expected load figures
avoid caches
avoid ‘noise’ like irrelevant network overhead
use mocks for external systems
tools: e.g. JMeter

resources:

Getting the Realtek RTL8187b WiFi stable under Ubuntu

2008-12-19T20:15:00.005+01:00

The latest Ubuntu 8.10 with its Linux 2.6.27 kernel supports the Realtek Rtl8187b wifi chipset (USB) natively. In prior versions of Ubuntu, I had to use ndiswrapper to get the integrated WiFi going on my Medion MD96290 laptop. This is not needed anymore, so I removed the ndiswrapper call from /etc/modules.

Unfortunately, the native rtl8187 module proves very unstable with the Rtl8187B: although the connection appears fine in the menu bar, the wifi interrupts its connection or is suddenly very slow.

After some investigation I found a workaround: http://ubuntuforums.org/showthread.php?t=792092

gksudo gedit /etc/rc.local
add at the end, before the exit 0 command:
iwconfig wlan0 rate 5.5M

Nice to have my wifi up again!

Monitoring the Energy-production of my Solar Panels

2008-12-14T20:39:00.005+01:00

update 13/6/2009: check out my new sunnybeamtool command line application.

I just retrieved and processed the energy-production details of my photovoltaic panels using the SunnyBeam under Windows:

Google docs offers a great way to share charts and detailed data; even the atom feed is not missing.

Using the Sunny Beam under Linux: Not OK - yet?

2008-12-05T17:01:00.013+01:00

update 13/6/2009: check out my new sunnybeamtool command line application.

As mentioned in my previous post, I have a "Sunny Beam" appliance that monitors the electricity production of my solar panels. The Beam maintains a wireless connection with the inverter to capture the power data. The clean LCD display offers a nice graph and the module recharges its batteries with solar cells. In addition, the Sunny Beam has a USB connection to send the data to a PC. Unfortunately, the Sunny Beam uses no standard 'USB mass storage' interface, but requires a Windows-only USB-driver. SMA.de responded to my mail stating that "The Sunny Beam can only work on windows operating systems. Windows 2000, XP (32 and 64) and Vista (32)". It is sad to see that succesful companies ignore the fact that other operating systems like MacOS and Linux are becoming increasingly popular. I'm not a Mac-fanboy, but MacOS has around 9% market share according to this site.

Once the USB driver is setup under Windows, you must use the complex Sunny Data Control application to gather the data. It is not a simple program to use, especially for small scaled solar systems. You really need to dive into the manual to get started. SMA informed me that a new, simpler application will be released in the beginning of 2009. According to the sma-america.com site, this new program will probably be called 'Sunny Beam WebConnect'.

I tried to install the Sunny Data Control under the latest version of Wine HQ. By default Ubuntu provides an outdated version of Wine, but upgrading is well documented on the Wine site. The application installs seamingly and starts fine; I even managed to register the USB driver with the following command:

wine rundll32 setupapi.dll,InstallHinfSection DefaultInstall 128 Desktop/sbeamdriver/SBeamUSB.inf

Unfortunatly, I found out that USB kernel drivers are not supported in Wine. Dead end street...

Next, I found out that SMA provides a C library, YASDI -- 'Yet Another SMA Device Interface' -- for generic low-level access to their inverter interfaces. The Yasdi-API has everything to convince me: it's LGPL, linux is supported out-of-the box, Java JNI wrappers exist and it has a nice layered architecture where the technical drivers are decoupled from higher layers like the protocol-handling. Unfortunatly, Yasdi has no USB driver. I contacted the author of Yasdi, to check if the Sunny Beam fits in the standard Yasdi-API framework concept, so it would be easy to add the USB driver to it. Bad luck, Heiko informed me that the protocol used in SunnyBeam is a little bit different from other -- I suppose older -- devices. In order to add the SunnyBeam module to Yasdi, the protocol-layers will have to be adapted. I suppose the YASDI support and development is frozen at SMA.

The suggestion I got from Heiko is to use VMware or VirtualBox to run a Windows-system. Since my system is dual-boot (Windows is the last -- painful -- option) this doesn't add much value to the setup. Other suggestions are welkom.

The device presents itself to Ubuntu as follows:


$ lsusb -v
Bus 001 Device 002: ID 1587:002d SMA Technologie AG
Device Descriptor:
bLength                18
bDescriptorType         1
bcdUSB               1.10
bDeviceClass            0 (Defined at Interface level)
bDeviceSubClass         0
bDeviceProtocol         0
bMaxPacketSize0         8
idVendor           0x1587 SMA Technologie AG
idProduct          0x002d
bcdDevice            2.00
iManufacturer           1 SMA Technologie AG
iProduct                2 Sunny Beam
iSerial                 3 00024383
bNumConfigurations      1
Configuration Descriptor:
bLength                 9
bDescriptorType         2
wTotalLength           32
bNumInterfaces          1
bConfigurationValue     1
iConfiguration          0
bmAttributes         0xa0
(Bus Powered)
Remote Wakeup
MaxPower              200mA
Interface Descriptor:
bLength                 9
bDescriptorType         4
bInterfaceNumber        0
bAlternateSetting       0
bNumEndpoints           2
bInterfaceClass       255 Vendor Specific Class
bInterfaceSubClass    255 Vendor Specific Subclass
bInterfaceProtocol    255 Vendor Specific Protocol
iInterface              2 Sunny Beam
Endpoint Descriptor:
bLength                 7
bDescriptorType         5
bEndpointAddress     0x81  EP 1 IN
bmAttributes            2
  Transfer Type            Bulk
  Synch Type               None
  Usage Type               Data
wMaxPacketSize     0x0040  1x 64 bytes
bInterval               0
Endpoint Descriptor:
bLength                 7
bDescriptorType         5
bEndpointAddress     0x02  EP 2 OUT
bmAttributes            2
  Transfer Type            Bulk
  Synch Type               None
  Usage Type               Data
wMaxPacketSize     0x0040  1x 64 bytes
bInterval               0
Device Status:     0x0000
(Bus Powered)
$ dmesg | tail
[ 3100.544038] usb 1-1: new full speed USB device using uhci_hcd and address 2
[ 3100.781392] usb 1-1: configuration #1 chosen from 1 choice

Photovoltaic Panels on my Roof

2008-12-04T20:51:00.012+01:00

At home I have a reef aquarium that consumes a lot of electricity. In order to cut my electric bill and thanks to local subsidies and regulations, I decided to install photo-voltaic panels on the southern part of my roof. The planning of the system was executed by Planet Eco.

Technical details:

17 panels from EcoStream ("Atrium");
single panel specs:

185Wpeak
1628x840x42mm
16,2 kg
polycrystalline cells
frame: aluminium

total: 3,145kWp
minimum 2673 kWh
The inverter is a SMA "SunnyBoy" 2500, with a wireless 'piggyback' connection to a standalone 'SunnyBeam' unit.

the SunnyBoy 2500

the official 'green' current meter for getting subsidies

As an extra, I bought the wireless 'SunnyBeam' module. Once the wireless piggyback is installed on the inverter by a professional, the SunnyBeam offers a clean way to monitor the electricity production of the solar panels. The SunnyBeam itself uses solar energy to charge its batteries. The device maintains a detailed graph of the production for one day and the detailed total production for the last 31 days.

The connection from the inverter to the SunnyBeam is over wifi, although a proprietary protocol is used. There is also a USB connector, but unfortunately, only Windows drivers are available. The "Sunny Data Control" software is needed to retrieve the data over USB, but the software's interface is rather overloaded and non-intuitive, especially for small-scaled needs. You really need to delve into the manual to get started.

update 13/6/2009: check out my new sunnybeamtool command line application.

The worst RIA-site ever

2008-12-01T20:41:00.009+01:00

I've never been a big fan of the 'Rich Internet Application' movement, with it most obvious defender: Flash. There are two contenders that fight for the RIA dominance: Microsoft Silverlight and Suns' Java FX. Flash is currently the only platform that's ready for real use: Silverlight and Java FX are still in beta.

Java FX has a huge potential as being a real Open Source (GPL) RIA platform compared to its more or less closed source competitors. I know, Flash has some parts of it made 'open', but Flash basically remains a commercial platform, requiring commercial Adobe developer tools for the 'real' work (including the now classic video à la youtube). It remains to see if Sun will be able to maintain the full GPL license for Java FX, since video and sound are usually licensed from third party providers using patented technologies (e.g. On2).

Although RIA apps have a tremendous 'whaw' factor (e.g. parleys.com) , I'm quickly bored by all the transitions, fading in/out and other visual magic. It's also very hard developing good RIA applications that are both functionally relevant and appealing to the eye. Developers must combine design skills, good taste and programming talents. Otherwise you end up with disastrous websites like this amusement park. It takes about 15-20s of looking at a shiny progress bar, hearing silly music and looking at stupid animations to finally find the opening hours. Definitely in the top-10 of 'worst-RIA-sites-ever'.

RIA apps fail to embrace the strengths of the classic web:

bookmarkability & addressability: In typical RIAs, only the main page has an address; forget to send a to-the-point URL to a friend... Bookmarking individual parts of a RIA app requires the oh-so-stupid 'link this' button. This is opposite to the current "REST" trend where everything is a URL, a resource.
user driven action: RIA apps tend to limit the users' behavior, forcing them to follow a fixed pattern, where classic webshops let the user decide where to click. Have a look at the on-line promotion folder of a classic brick and mortar shop. It takes about 2-3 seconds to change to the next page; at least one or 2 minutes of clicking if you're searching a specific item.
speed: it looks like every RIA app needs its shiny progressbar, tricking the user to wait much longer than he'd be willing to wait for a classic web-page
search engine friendliness: by default RIA apps can not be indexed by google, unless tricking google by offering a 'ghost' site to the search robots. This is how parleys.com solves their search ranking problem. Someone said 'hacking'?
OS-agnostic: only Java FX promises to target Windows, MacOS and Linux. Of course, Flash support under Linux has made huge progress, with e.g. the Flash 64bit version released under Linux first. But all RIA platforms try to address the 'fat client' offline problem: running apps while disconnected from internet, even outside the browser (e.g. Adobe AIR or Java Update 10). Unfortunatly, those 'fat client' standalone versions of RIA apps offer a consequent look on all OSes. This might be a plus, but the current trend is to adapt your UI to the OS. Firefox 3 for example has a different look and feel under Gnome than under Vista.
hackability: I'm addicted to noscript, adblock and firebug, but those tools are useless in a RIA context. Even a simple 'save as', as Firefox provides, is not straightforward. Perhaps the development of gnash might bring some new hacker-tools to the scene.
open standards: html4, css, ... why are they missing from the RIA picture? Yes, ActionScript is like EcmaScript, which is like JavaScript. Anyway, in real apps, ActionScript only runs in Flash.

So do I hate RIA? I generally tend to dislike websites that are mostly made in Flash, unless Flash really adds value. For parleys.com with its highly integrated sound/video and slides Flash adds value, although I preferred the initial version that used standard html and 'Ajax'.

I do think that Flash and Java FX have their value when used to enrich standard web pages. You'd be amazed to see what is possible in plain html/javascript/css. Look at the spectacular Apple ITunes store for example. In general everything that's not supported in html4 requires flash: audio/video or drawing. By putting those pieces in a classical webapp you get a tremendous combination. For example, Google Analytics or Google Finance are html pages, but the nice graphs are Flash. The integration is seamless, an end-user can see no difference between the html and the flash parts.

We'll see what the RIA-future brings: all Flash (flex/air), Java FX, or a "back to the web" reaction?

Measuring Bootup Time of Ubuntu with Bootchart

2008-09-30T20:51:00.008+02:00

I just discovered on slashdot the "bootchart" tool. Installation on Ubuntu is as simple as looking up the bootchart package in Synaptic package manager and installing it.

After booting, a png file is created in /var/log/bootchart/. Looks like my bootup-time could be improved, but it's not clear to me (yet) how I can optimize. This is my bootchart of the Medion MD96500 laptop (Intel Pentium M 1.73GHZ - linux 2.6.24-21 - Ubuntu 8.04.1:

I did some first experiments with the showdelta script that parses the kprint bootup timings, but bootchart shows parallel timings of processes fired during boot. Great tool! Suggestions for optimization are welcome.

Effective Java - 2nd Edition

2008-09-25T21:40:00.004+02:00

The second edition of Effective Java (by Joshua Bloch) is now updated for java version 6. This book is a must-read for any serious java-developer. Contains very practical and up-to-date tips for writing great java programs.

Checkout the effective java site.

The first edition -- still valuable but a bit outdated -- is available on google

Joshua Bloch presents the new version of his book on parleys.com. Highly recommended:

eGovernment: Using the Belgian eID under Ubuntu

2008-09-07T13:42:00.039+02:00

In Belgium, online eGovernment services require the usage of an electronic Identity Card (eID) as a security token. This eID, coupled with a PIN, allows easier paperless interaction with the official services. Last spring 2008 the eID infrastructure was still Windows-only, but now Linux is supported. This post details how to enable the Belgian eID card reader on Ubuntu (8.04). The official info-website is http://eid.belgium.be/nl/Hoe_installeer_je_de_eID_/Linux/index.jsp.

I borrowed an ACR38 card reader. From the command line it displays:

$ lsusb
Bus 001 Device 004: ID 072f:9000 Advanced Card Systems, Ltd ACR38 AC1038-based Smart Card Reader

Simply install the ACR38U driver from the Synaptic Package Manager:

Synaptic recommends the pcscd as a dependent package.

Finally, select the 'beidgui' package and accept all dependencies that are proposed:

Now the standalone application can be run. Select Applications/Accessories/Reading and Administration in the Menu or trigger the command line:

$ b eigui

The final step is making firefox accept the external card reader. For this, install the Belgium eID add on for firefox at https://addons.mozilla.org/firefox/addon/belgium-eid/.

If you get a "ssl renegotiation error", you should temporarily allow the specific e-government site to accept ssl renegotiations by going to the magic "about:config" url and updating the security.ssl.renego_unrestricted_hosts key with ccff02.minfin.fgov.be.

Nice to see Linux-friendly eGov applications!

More info:

update 2011/05/09 for ubuntu 11.4:

- in about:config update the security.ssl.renego_unrestricted_hosts with ccff02.minfin.fgov.be

- use the 'belgium eID' firefox addon that registers the correct library instead doing the manual configuration.

update 2011/06/14: unfortunately, the minfin.fgov.be site added a "firefox 3" test instead of fixing the ssl renegotiation error. Luckily, it's easy to circumvent this by installing the User Agent Switcher plugin and activating a new "FireFox 3" header. The following user-agent header worked for me: "Mozilla/5.0 (X11; Linux i686; rv:2.0.1) Gecko/20100101 Firefox/3.5.0"

Recommended Linux Podcasts

2008-09-05T21:41:00.005+02:00

When commuting to work, I like to listen to podcasts. Most of the podcasts I listen to are tech-related. Let's review some of those.

My favorite Linux podcast is "The Linux Action Show" (http://www.jupiterbroadcasting.com/?cat=4). At first, the exaggerated intonations of the intro are surprising, but this is just part of the 'format'. The show itself is very entertaining and also great to keep up with the latest Linux news and new devices that run Linux. I like the more reasonable view of the presenters on running non-GPL software on Linux. Always great to hear about new cool Linux apps and products. If they'd talk a bit more about java and openjdk instead of mono (and banshee), the show would be perfect :-).

The second (and last) item in my "Linux Podcast top-10" (well, top 2 actually) is the "Lottalinuxlinks podcast" (http://lottalinuxlinks.com). This is the "anti-Linux action show". No great radio-voices and high-end recording studio here: every lottalinuxlinks show is recorded from within a driving car when returning home. The 'cowboy' southern USA accent just adds to the geek-level of this podcast. Cool linux info.

A ToDo list for Gmail and Google Calendars: Remember the Milk

2008-07-24T07:45:00.006+02:00

I'm a big fan of gmail and google calendar as my centralized PIM. One major feature is missing however: there is no support for ToDo lists. You can add a simple ToDo google gadget on your iGoogle page, but that todo list is only accessible from within iGoogle; there is no integration with gmail or calendar whatsoever.

Some greasemonkey scripts also add todo lists to google calendar. Unfortunately, they are very sensitive to site fixes and updates: any update can break those nice scripts.

As an alternative, I just signed up to Remember the Milk. 'RtM' provides a rich ToDo list that integrates well in gmail and calendar:

nice ajax input
extensive features
firefox extension to integrate the todo list into gmail: your inbox-view is enhanced with a RtM list on the left
integration with google calendar

Not a perfect solution yet, as you have to logon separately to RtM and the integration with gmail and calendar is a hack -- a very nice one, but still a hack. But it's ok for now.

Podcasts in Rhythmbox: removing special characters from the downloaded files

2008-07-17T20:06:00.008+02:00

When Rhythmbox downloads podcasts, it uses the URL as a filename. I like to copy/paste the podcasts to my mp3 player with Nautilus. Unfortunately, my mp3 player has trouble with special characters like colons and question marks (':' and '?'). This is probably due to the FAT32 file system.

I use the following script (triggered from a launcher) to remove those chars from my podcasts:

#!/bin/bash
find $HOME/Desktop/downloads/ | while read f
do
  REN="$(echo "$f" | tr ":" "_" | cut -d '?' -f 1)"
  echo "mv $f $REN"
  mv "$f" "$REN"
done

After that, copy/paste of my podcasts goes fine.

Remote Power off the CH3WNAS

2008-06-28T21:27:00.014+02:00

In my previous post, I found out that the CH3WNAS is shutdown by touching a /tmp/shutdown file. It is possible to use the existing web admin pages to trigger a remote shutdown from a script by using 2 wget-commands. One major drawback of this solution is that your admin-password is stored in cleartext in the script.

In this post I show how you can do a remote shutdown via the LAN, without hard coding your admin password. Since this tweak could be misused for a remote DOS attack, make sure you put the CH3WNAS behind a NAT router.

Busybox (activated with the fun_plug script) provides a powerful nc (netcat) command that also has a -l listen mode. This listen-mode is ideal for simple lightweight client/server communication.

Steps for enabling remote shutdown from a script:

make sure telnet (through busybox) is enabled: http://www.aroundmyroom.com/2008/01/03/ch3wnas-enabling-telnet/
create a shutdown_listener.sh script (e.g. with ftp) on the root of HD_a2 with following contents:
while true
do
  /mnt/HD_a2/busybox nc -l -p 1234 | while read line
  do
     if [ "$line" -eq "0" ]
     then
       touch /tmp/shutdown
     fi
  done
done
This scripts on the NAS listens in a loop on port 1234. When it reads a "0" it puts a /tmp/shutdown file that triggers the immediate poweroff.
add the following line to the fun_plug script and restart the CH3WNAS:
/mnt/HD_a2/shutdown_listener.sh &
from the client-side it's enough to create a nc-script that sends a "0" to the NAS.
on Ubuntu I created a Launcher on my desktop with following command :
bash -c "echo 0 | nc 192.168.1.30 1234 -w 1"

If you're stuck with Windows, look out for a Windows version of netcat (nc) and put similar commands in a shortcut (cmd /c "..."). Or better, upgrade to Ubuntu :-)

I finally have a script that I can put on all my laptops to trigger a remote shutdown of the NAS. Unfortunately, for powering the device back ON again, you still have to push the powerbutton.

Getting Compiz to run on Ubuntu 8.04 with an ATI X700 chipset

2008-06-20T21:07:00.009+02:00

One of my laptops, a Medion md96500, I couldn't start Compiz after I upgraded it from Ubuntu Gutsy to Hardy (8.04). I remember that when I installed 7.04, I had to do some tweaking to get the video working. Luckily, since Ubuntu 7.10, the display configuration goes really well 'out of the box', but maybe some of my tweaks of last year did survive in my current Hardy install (8.04). This is the video chipset in my MD96500:

$ lspci | grep VGA
01:00.0 VGA compatible controller: ATI Technologies Inc M26 [Radeon Mobility X700 XL (PCIE)]

I tried to play with the "Hardware Drivers" dialog-box, but this didn't help. Luckily, fixing was rather easy:

I removed the proprietary fglrx-driver of ATI and switched back to the open-source driver. This is well documented in: https://help.ubuntu.com/community/RadeonDriver. The switch went rather smoothly, but compiz still failed when trying to activate it.
Next, I ran the excellent compiz-check script http://forlong.blogage.de/article/pages/Compiz-Check. As you can see in the output, compiz-check not only detects that my x700 is blacklisted, but it also fixes it by letting Compiz ignore the blacklist:

$ ./compiz-check

Gathering information about your system...

Distribution: Ubuntu 8.04
Desktop environment: GNOME
Graphics chip: ATI Technologies Inc M26 [Radeon Mobility X700 XL (PCIE)]
Driver in use: radeon
Rendering method: AIGLX

Checking if it's possible to run Compiz on your system...

Checking for texture_from_pixmap... [ OK ]
Checking for non power of two support... [ OK ]
Checking for composite extension... [ OK ]
Checking for FBConfig... [ OK ]
Checking for hardware/setup problems... [FAIL]

There has been (at least) one error detected with your setup:
Error: Laptop using radeon driver.

Would you like to know more? (Y/n) Y

It has been detected, that you are running a laptop with an ATI chip.
The radeon driver supports Compiz out-of-the-box but because of a nasty bug
in the driver that causes X to freeze on some cards, this particular
combination had to be blacklisted in Ubuntu "Hardy Heron".

In case you already used Compiz successfully on Ubuntu 7.10 (Gutsy), it is
safe to skip the blacklist.

Do you want to skip blacklist checks by Compiz? (y/N) y

$ ./compiz-check

Gathering information about your system...

Distribution: Ubuntu 8.04
Desktop environment: GNOME
Graphics chip: ATI Technologies Inc M26 [Radeon Mobility X700 XL (PCIE)]
Driver in use: radeon
Rendering method: AIGLX

Checking if it's possible to run Compiz on your system...

Checking for texture_from_pixmap... [ OK ]
Checking for non power of two support... [ OK ]
Checking for composite extension... [ OK ]
Checking for FBConfig... [ OK ]
Checking for hardware/setup problems... [ OK ]

Compiz is now fully functional on my laptop, and till now, it seems stable. gr8 :-)

Banshee 1 on Ubuntu

2008-06-17T21:28:00.014+02:00

I listened to the recommendations of the Linux Action Show and installed Banshee v1 on my Ubuntu laptop. I simply followed the steps in http://www.simplehelp.net/linux/how-to-install-banshee-10-in-ubuntu/. If you hate the command-line, use Synaptic Package Manager for installing the banshee-1 package and it's dependencies.

I'm not a big fan of the mono environment and its Microsoft origin, but I must admit: starting banshee V1 gives the 'wow'-effect. It looks like a rhythmbox on steroids:

As you can see, my muvo v100 is recognized and it shows a nice "memory available" graph. Generic USB devices like the v100 must define a .is_audio_player file in the root to be automatically recognized by banshee or rhythmbox. Banshee also has a user-friendly "eject media device" option in the context-menu.

And there is built-in podcast support. You cannot import a list of podcasts, but the "add podcast" automatically pastes the url that you copied from another window. You cannot remove older podcast from the list, but you have to "mark them as old". When copy/pasting to the media device, it uses the (usually) bogus "artist" data of the podcast. I couldn't find a way to "autosync" podcasts to my mp3 player yet.

I didn't try to connect to my firefly daap or mediatomb upnp server, but mediatomb should work through djmount at least. But maybe I'll use samba for read & write access.

I noticed a few minor problems:

the banshee icon is enlarged when you put the menu-bar on the left (or right) side of the screen (see screenshot above)
when the usb media player is already mounted, banshee hangs at startup:

I suppose those problems will be fixed soon. And perhaps Ubuntu should update the banshee package in their distro and why not, consider replacing Rhythmbox.

update 27/6/2008: Unfortunatly banshee hangs / crashes quite often once you use it more. I'm still using Rhythmbox for my 'normal' media player stuff, surviving with Rhythmbox's limitations: single threaded (and buggy) podcast retrieval, picky when media drive is unmounted, too simple media device support, almost no configuration options... But at least Rhythmbox doesn't crash that often.

update 22/1/2009: Banshee is much more stable now and really usable as a music player, more 'finished' than rhythmbox. Unfortunately, the podcast retriever still has a bug: when the download speed is too low, the download stops; retrying will download the full podcast again and fails again when the bandwidth is still too low (I experience this with e.g. 'Buzz out Loud'). It seems like a core Mono bug since no stacktrace is logged. I reverted to gPodder as my podcast client.

Restaging the CH3WNAS

2008-06-17T21:04:00.004+02:00

Suddenly, after a reboot, my CH3WNAS couldn't start the wireless. It connected to a different ip address than the fixed I assigned it, and the admin-pages didn't react correctly to my actions: e.g. you couldn't see the existing users, but when adding a user I got a "User already exists" error message. Even the ftp server wasn't reacting. Through telnet, I could disable the fun_plug script, but that didn't solve the problem either.

Finally, I reset the firmware in the admin-pages, and the setup my wireless/ftp again. I re-enabled the fun_plug and everything is OK now: telnet is available and I can connect to firefly and mediatomb to access my music...

As a first prevention measure, I edited the firefly.conf file to log onto the hard drive instead of filling up /var/log:

logfile = /mnt/HD_a2/firefly/mt-daapd.log

We'll see if this helps...

Tweaking the CH3WNAS Power Off

2008-06-16T17:55:00.014+02:00

On the CH3WNAS, you have to press the power button for an endless 10s before it finally shuts down. Apparently this 10s delay is determined by the /usr/sbin/chkbutton daemon that polls the power button. There exists a replacement for this process for the dsm-g600 in http://forum.dsmg600.info/t955-chkbutton-replacement.html.
Unfortunately, I couldn't download the attachment to try it out on the CH3WNAS. I asked the author to resubmit his binaries. Maybe for a next post...

There is also shutdown-option in the web-admin pages. After logon, the shutdown is done by firing a POST /goform/sysShootDown request onto the CH3WNAS. After some searching in the sourcecode (download it from the conceptronics' site), I found out that CH3WNAS_GPL/goahead/LINUX/web_api.c handles the shutdown:

void sysShootDown(webs_t wp, char_t *path, char_t *query)
{
if( checkIdleTime(wp) == WEB_IDLE_TIMEOUT )
{
websRedirect(wp, T("web/login.asp"));
return;
}
system("touch /tmp/shutdown");
websRedirect(wp, T("web/tools/shutdown.asp"));
}

--> A simple (but unconventional) touch /tmp/shutdown triggers the shutdown. I suppose it's the same chkbutton daemon process that monitors and picks up the /tmp/shutdown file.

If you need to do a fast remote shutdown, you can exercise the existing web-interface from command line (the wget is standard under linux; under windows, have a look at wget for windows):

wget -o /tmp/test "http://192.168.1.30/goform/formLogin?f_LOGIN_NAME=admin&f_LOGIN_PASSWD=xxx&f_login_type=0"
wget -o /tmp/test2 "http://192.168.1.30/goform/sysShootDown"

Replace the ip & password according to your local setup. Put those commands in a script and you get a simple way to shutdown the unit remotely with a simple double-click. The problem with this code is that you have to put your admin-password in cleartext on your pc. I haven't figured out yet how to circumvent the logon page to remove this security-issue.

SpringOne 2008 - Inside SpringSource Application Platform by Rob Harrop

2008-06-13T13:49:00.005+02:00

The S2AP builds on OSGI and enriches it with a lot of functionality: e.g. combining relating bundles in libraries.

See SpringSource for details.

SpringOne 2008 - What's new in Spring MVC 2.5 and beyond by Keith Donald

2008-06-13T13:26:00.010+02:00

@MVC (=Spring MVC 2.5)
The @Controller is now the recommended way to build a controller in Spring-MVC. It allows any pojo to act as a MultiActionController, but without the need to extend any class. In a @Controller class you can group all logical actions together. The classes with the @Controller are found automatically through classpath scanning.

The @Controller maximally relies on naming-conventions for your classes and methods to intercept URL-patterns (e.g. "Hotels.index()" is accessible through "/hotels/index"). Of course, this is customizable. All action methods have plain java types, but can also, if needed, ask for special parameters like Request, Response,... etc. @MVC detects those parameter types and injects them automatically.

If the action-methods return a String, then this is assumed to be the view-identifier. If it's a non-string, the returned object is then made available for the view. The modelobject is named by using generics: e.g. List<hotel> will be called 'hotelList' in the view.

Spring JS

enhance or decorate existing elements
render subfragments --> uses Apache Tiles for composition
use dojo query() to decorate multiple elements on a page.

WebFlow 2 (+ 'Deep Dive' session)

lighter, more consise syntax: e.g. first state is start-state by default.
Ajax enabled through: ViewScope
extensive Expression Language use
abstract flows that can be extended
better JSF support
(new) best practice: put flow with associated template (e.g. jsp), messages and resources and in same directory --> relies on simpler defaults
Spring Security

WebFlow 3
Will add a site-definition language to mix non-webflow pages with webflows.

SpringOne 2008 - Using Spring Web Services 1.5 by Arjen Poutsma

2008-06-13T13:08:00.005+02:00

Contract-first is considered best practice for designing a WebService:

create a sample xml message
generate the xsd based on the xml: use trang or XmlSpy
tweak the xsd

Define your Spring-WS endpoints as a POJO with @EndPoint:

input message
start a business service
result message

Let Spring generate the WSDL-boilerplate around your existing xsd and @Endpoints. For the xml parsing/generation, you can use SAX, DOM or Trax or use a more advanced (un)marshaller with the Spring OXM module: e.g. Jaxb2 or XmlBeans.

For testing your WS, use soapUI.

Other Spring WS features are:

pluggable interceptors: logging, optional validating,...
client WS: WebServiceTemplate
WS over JMS or SMTP
OSGI Bundles
WS Security (see ws.apache.org/wss4j/)
WS-Addressing

SpringOne 2008 - Keynote 2 by Adrian Colyer and Rob Harrop

2008-06-13T12:59:00.005+02:00

Spring Dynamic Modules (formerly called Spring OSGI) extensively builds on OSGI.

Bundles can be loaded/replaced dynamically at runtime
multiple versions of bundles can coexist
with the import-library extension, you can import a collection of bundles
most common OSGI bundles are available at http://springsource.com/repository
it provides IDE integration through Spring IDE

The SpringSource Application Platform builds further on the Dynamic Modules concept.
Through 'personalities', the platform only loads what you really need:

batch: no web / servlet api needed; only jar
online: servlet api , but perhaps no JMS etc...

--> startup time / footprint is minimized.

SpringOne 2008 - Decorating Web Pages with AJAX using Spring JavaScript by Jeremy Grelle

2008-06-13T12:54:00.003+02:00

Spring JS provides a javascript layer above Dojo. The Dojo javascript files are provided by Spring JS in a gzipped format. The dojo-integration is pluggable and will be enhanced with other javascript tool sets (e.g. jQuery).

The basic idea is to start from a fully-functional 'classical' webapp and progressively enhance the application with AJAX. But: even with javascript disabled, the page must remain functional, degrading back to it's basic behavior - really interesting, since I'm a big noscript fan :-). The Ajax page-elements are defined with Apache Tiles and served as a whole or through fragments depending on the request-type. WebFlow support serving separate 'fragments' in one state to allow for example pagination without page-refresh.

Spring JS also provides a basic CSS from Mike Stenhouse that provides a good basic look and feel that can be easily overridden.

The Spring JS provides a JavaScript syntax to enhance existing page elements with e.g. client-side validation, tool tips or partial page-refreshes. Again, the idea is that even without Javascript, the page is still OK.

SpringOne 2008 - Spring Transaction choices for Performance by Jürgen Höller

2008-06-13T12:51:00.001+02:00

Tune with the @Transactional annotation:

propagation: "requires", "requires-new" or, the spring-specific, "nested" (=uses save points)
isolation level: default: read committed
set the readonly flag --> hint for runtime optimization.

Prefer native JDBC transactions over XA transactions:

XA is difficult to setup

run a transaction coordinator
XA JDBC driver
XA Message Broker
put the XA recovery log to a reliable log storage

XA has a significant run time overhead

two phase commit
state tracking
recovery
on restart: complete pending commits/rollbacks --> read the "reliable recovery log"

Check if you really need XA! Prefer the use of a native transaction or combine two native transaction, with some small extra safety provisions. E. g. Usually the "exactly once" guarantee of an XA JMS can be relaxed --> use a native JMS transaction and detect duplicate messages. --> much higher throughput.

SpringOne 2008 - Using Spring Security 2 by Mike Wiesner

2008-06-13T12:49:00.003+02:00

What?

formerly Acegi security -- Acegi is an abbreviation for AbCdEfGhI :-)
Authentication: URL (coarsegrained) or method (finegrained) authentication
ACL
LDAP support
SSO

How?

pluggable filters (through 1 delegating servletfilter); config in security.xml.
Eg: LDAP, JAAS, OpenID plugins
repository for storing (e.g.) additional user details
--> implemented in e.g. JDBC or LDAP
--> combine multiple data sources
SecurityInterceptor (AOP) for method-based security
@RolesAllowed annotation (JSR-250)

gotcha: The RBAC pattern decouples users from permissions through roles: users >-<roles>-< permissions! Test on permissions and let the repository associate your user with the correct permissions through the assigned roles. Never test on roles in your code.

Spring security also provides a authz-taglib for querying security-permissions while building a web page. This is helpful, but you still have to secure the basic URLs and methods.

SpringOne 2008 - Persistence Tuning for the Spring Environments by Tomas Risberg

2008-06-13T12:47:00.002+02:00

a few best practices:

get your DBA involved early
take time to design your db: tables, indexes, data types, normalize, partition,... all have impact on the performance
consider batch vs online. (consider native db-tools for loading)
limit the number of rows / columns per table -> normalize
avoid nulls in where clause (nulls are not indexed!)
avoid starting with a wildcard in "like '%xyz'" tests (not indexed)
select the proper transaction manager: JTA and XA is the slowest solution --> prefer plain JMS and native JDBC transactions
avoid 'SERIALIZABLE' transaction isolation
use the readonly-flag (JDBC) whenever possible.
use a proper connection pool: Apache DBCP or Oracle Datasource Connection Pool (never use the 'standard' DriverManagerDatasource!)
early test your connections at startup: JDBC 4: isValid() or "select 1 from dual;"
use prepared statements --> optimized by db
setFetchSize()
use a RowMapper of BatchJdbcTemplate
caching
sql tuning: run EXPLAIN and ANALYZE commands
tune Oracle memory: SGA (=global + sessions) and disk (blocks/files); put logs on separate volume

SpringOne 2008 - Spring 2.5 on the way to 3.0 by Jürgen Höller

2008-06-13T12:45:00.004+02:00

Part of this talk overlapped with Rod's talk, but Jürgen could go in more detail.

Spring is aligning itself with modern standards / APIs:

JDK1.6 support: e.g. JDBC 1.4, with better LOB handling
support for JMX and MXBeans
AspectJ loadtime-weaving --> avoids proxy pitfalls; support "new" operator
JEE5: Servlet API 2.5 & JAX-WS

"Self-describing classes" with annotations, removing config. As a general rule, annotations should be descriptive; even if a compiler ignores them (e.g. in tests), they should add documentation to the class:

JSR 250 @PostConstruct & @PreDestroy: quick wins: explicit initializers
JSR250 @Resource on steroids: standard is JNDI-only, but spring also resolves spring-bean names.
@Autowired (& @Qualifier for more specific rules): elegant way to minimize xml-config, while keeping fine grained control over the DI. It's not an all-or-nothing: mix&match between xml and @autowired makes sense.
autodetectable components with: @Component. Without declaring a bean, the component automatically becomes available in the spring context.
@Configurable (AspectJ) --> use regular new operator and benefit from Spring config! (load-time weaving)
"@MVC": No need to extend a MultiActionController anymore: with @Controller and @RequestMapping, the multi-action controller is now the recommended MVC-way, using simple POJOs. It relies extensively on standard naming of classes and methods to interpret URL paths. But, of course, this can be tweaked to the extreme. Even method-parameter names can be used for auto-passing request params if you built your classes with debug-info enabled. Gr8!
decide what you prefer: 'externalized' xml config complemented with annotations or fully annotated components with embedded config. As usual Spring doesn't impose one view. The xml-config can, for example, be interesting for a service layer, while the MVC-layer could prefer the annotations.

Spring 3.0

Milestone 1: 8/2008 ???
Java5+: generics & varargs;
extensive use of Expression Language (cfr WebFlow)
remove deprecate stuff like commons attributes
WebSpere 6.1
Rest support

SpringOne 2008 - Keynote by Rod Johnson

2008-06-13T12:43:00.003+02:00

Spring 2.5

With the new @autowired annotation (+ @qualifier) , auto wiring is now finally usable in real systems. This can help reducing the configuration-overhead to a minimum while keeping maximal control over your code. Looks really great - if you can use Java 5.
JSR-250 support: Rod is less enthusiastic about the @Resource, but the @PostConstruct and @Predestroy annotations are highly recommended --> no need to explicitly set the init-method in your spring configs anymore.

Testing

favor JUnit 4.4 or TestNG --> simple annotations; no need for the abstract testing hierarchies.

Spring WebMVC & WebFlow

less configuration, more sensible defaults.
annotations instead of extending MultiActionController --> controllers are less tied to Spring MVC
better integration with JSF (Spring Faces)

Ajax support

support for 'fragments' in a view state: with Apache Tiles all fragments are rendered as a whole (=classic refresh) or individually, triggered by Ajax events
introduction of a new ViewScope in WebFlow: objects in view-scope survive as long as you remain in the same state, surviving multiple Ajax requests.
Spring JavaScript is a new project that builds on Dojo to provide Ajax support. The accent is on 'clean degradation', where your webapp is fully functional, even with JavaScript turned off.

SpringSource products

the SpringSource platform builds on OSGI to provide a highly modular framework. Very interesting ideas, but maybe a bit too revolutionary?
SpringIDE is now bundled in the Spring Tool Suite. This is an Eclipse distribution that comes with all plugins needed for Spring development embedded. One of the plugins is Mylin for 'task-based' development. The idea is to deliver a development environment that just works out of the box.

SpringOne 2008 - Welcome by Stephan Janssen

2008-06-13T12:41:00.005+02:00

I attended the SpringOne 2008 conference in Antwerp, Belgium. As a professional Java EE designer and developer, the Spring framework brings a refreshing look to enterprise java development. My interest started with these two, now classical, Rod Johnson books:

In this blog I'll summarize a few highlights of the first day of the conference.

As usual, Stephan Janssen promoted his new Parleys.com, completely written in Flash Air & Flex (GWT and JavaFX versions also exist). Very impressive, especially the tools for creating online web-presentations with synced video, slides an outlines. The business model of parleys will be the 'rent-a-space'-model: any company will be able to buy some 'space' to publish their presentations, benefiting from the extensive and appealing tool set and from using the reliable systems and the huge bandwidth that parleys has. Hopefully, they'll continue to serve the podcast feeds as well.

The JavaPolis conference is now rebranded into Javoxx. This is apparently to avoid trademark issues with Sun... Sad.

UPnP client under Ubuntu: djmount

2008-06-08T10:27:00.019+02:00

While there are quite a few UPnP servers for Linux out there (MediaTomb, Rhythmbox Coherence Plugin,...), I couldn't find any serious UPnP client (or UPnP-renderer in UPnP-speak) for Linux. I did find some promising libraries like the C++ Platinum UPnP library (used in XBMC) or the Java UPnpLib., but as far as I know no Linux MediaPlayer (like Rhythmbox or Amarok) uses those. There is also a UPnP 'controller', Cidero written in Java, but it has no playback capabilities. Cidero is very useful for debugging your UPnP setup, however.

The only relevant linux UPnP-client is djmount, also used in GeexBox. It is quite primitive: the UPnP devices are mounted as a filesystem. There are no debian packages available for djmount; you have to build it from the sources. The best copy/paste scenario for installing djmount on ubuntu is http://linuxowns.wordpress.com/2008/06/05/accessing-upnp-server-from-ubuntu/.

Unfortunately the last step '6', ie. autorunning djmount at logon doesn't work -- at least under Ubuntu 8.04. If you have a wlan, djmount fails when no network is up. As a comment by 'anonymous' points out, the Ubuntu NetworkManager fires the /etc/NetworkManager/dispatcher.d/01ifupdown script that runs (with run-parts) all scripts in /etc/network/if-up.d under root.

The following steps do work for autostarting djmount under ubuntu:

cd /etc/network/if-up.d
gksudo gedit ./djmount
contents of script:
#!/bin/sh

# Not for loopback!
[ "$IFACE" != "lo" ] || exit 0

modprobe fuse
fusermount -u /media/upnp
djmount -o allow_other /media/upnp

sudo chmod 755 ./djmount
sudo chown root ./djmount
sudo chgrp root ./djmount

Djmount is now (re)mounted whenever the network is up. As stated previously, djmount is a very rudimentary solution because Rhythmbox for example will parse and analyse all mp3s again. Maybe the linux mediaplayers will integrate some of the existing UPnP Libraries in the future. This would be a welcome addition. I'm still hesitating between using djmount or using regular samba-mounts on my laptop.

update 7/7/2008 : This fixed version runs djmount after the network is connected. No need for setuid or "sleep 30" anymore. (*)

(*) to remove the old version of the scripts this post in its first form suggested, you must:

remove the upnp action from the System/Preferences/Sessions
sudo rm -f /usr/local/bin/djmount.sh
sudo chmod u-s /usr/local/bin/djmount
sudo chmod g-s /usr/local/bin/djmount

Running Mediatomb on CH3WNAS natively!

2008-06-05T16:50:00.032+02:00

I managed to get MediaTomb running on the Grab'n'Go CH3WNAS without gentoo! There is no need to follow the gentoo steps in http://forum.dsmg600.info/t916-Installing-mediatomb.html since the MediaTomb project itself provides binaries for the PPC cpu with no dependencies. MediaTomb provides you a better uPnP Server than the original upnp in the firmware.

Since uPnP has no authentication, make sure you run the CH3WNAS behind a NAT router!

Those are the steps to setup MediaTomb on the CH3WNAS:

Download the mediatomb build for the ppc-platform:
http://downloads.sourceforge.net/mediatomb/mediatomb-static-0.11.0-r1-linux-uclibc-ppce300c2.tar.gz
(or, to get the latest version, surf to http://mediatomb.cc/pages/download#static and then click on the latest mediatomb-static*-ppc*tar.gz file)
unzip the tar.gz and ftp (e.g. with gftp) the full mediatomb-directory to the CH3WNAS under HD_a2 (= /mnt/HD_a2/ on the device)
telnet to the device (scroll down in my previous post for fun_plug'in and enabling telnet)
make executable:
chmod 775 /mnt/HD_a2/mediatomb/usr/bin/mediatomb
chmod mediatomb.sh for easier ftp(*):
chmod 777 /mnt/HD_a2/mediatomb/mediatomb.sh
edit (e.g. with ftp mediatomb.sh to start mediatomb as daemon +fix LAUNCHDIR bug):
#!/bin/sh

# if you want to launch this script from a different directory (for example
# from an another init script or similar, set the LAUNCHDIR variable below
# to the absolute path of the mediatomb.sh script

#LAUNCHDIR=`pwd`
LAUNCHDIR=/mnt/HD_a2/mediatomb

rm -f $LAUNCHDIR/mediatomb.log

export MEDIATOMB_DATADIR="$LAUNCHDIR/usr/share/mediatomb"
export MEDIATOMB_MAGIC_FILE="$LAUNCHDIR/usr/share/file/magic"
$LAUNCHDIR/usr/bin/mediatomb -m $LAUNCHDIR -f config -d -l $LAUNCHDIR/mediatomb.log

!! Remark: Make sure the script is Unix-encoded (single \n for line-breaks), otherwise you'll get a "-ash /mnt/hd_a2/mediatomb/mediatomb.sh" error. try this command to translate the linebreaks:
/mnt/HD_a2/busybox dos2unix /mnt/HD_a2/mediatomb/mediatomb.sh > /mnt/HD_a2/mediatomb/mediatomb.sh.new
mv /mnt/HD_a2/mediatomb/mediatomb.sh.new /mnt/HD_a2/mediatomb/mediatomb.sh
a first run will create the config.xml:
/mnt/HD_a2/mediatomb/mediatomb.sh
then run kill `pidof mediatomb` to stop the daemon
for easier editing of the config.xml (*):
chmod 666 /mnt/HD_a2/mediatomb/config/config.xml
now modify (ftp) the newly created /mnt/HD_a2/mediatomb/config/config.xml in order to add a fixed ipadres or interface and optional port.

<server>
...
<name>Michael's MediaTomb</name>
<udn>...</udn>
<ip>192.168.1.30</ip>
<port>50505</port>

<home>/mnt/HD_a2/mediatomb/config</home>
...
<protocolInfo extend="yes"/>
...

<map from="JPG" to="image/jpeg"/>
<map from="jpg" to="image/jpeg"/>
<map from="mpg" to="video/mpeg"/>
<map from="m4a" to="audio/mpeg"/>
...
The config.xml is well-documented at http://mediatomb.cc/pages/documentation#id2536421. Check out the docs in order to support for example PS3 connections.

for auto-startup, you can add the following lines your fun_plug:
...
#kill legacy upnp
kill -9 `pidof upnp`

#start mediatomb
/mnt/HD_a2/mediatomb/mediatomb.sh

As you can see, I still run firefly, because I couldn't get a uPnP client working in Ubuntu (Rhythmbox).
after startup, connect to MediaTomb for configuring the directories to watch: (e.g.) http://192.168.1.30:50505/

I'm not 100% sure that my config is OK, because I couldn't get Rhythmbox (+python-coherence) connecting to MediaTomb. I can see the uPnP broadcast messages from MediaTomb in WireShark and the Cidero java client can connect to MediaTomb and show a tree of media files. Cidero has no playback capabilities, so it's still possible I'll have to tweak the config a bit.

update 9/7/2008: fixed the config for serving jpeg (see comments below) / using correct 'magic' files. If you used a previous version of this post, delete the bogus config.xml and mediatomb.db* files and follow the steps starting with fixing mediatomb.sh.

(*) Close this small security hole: revert to the original protection with a chmod 440 or 550 afterwards. Of course, with telnet and upnp enabled, you'd better keep the CH3WNAS behind a NAT router.

Sound in Ubuntu

2008-06-02T20:57:00.006+02:00

On one of my laptops (Medion MD96290) I noticed that the sound wasn't working any more under Ubuntu 8.04. A while ago, I managed to get sound going in Ubuntu7.10 (http://forum.ubuntu-fr.org/viewtopic.ph p?pid=1290804)

After some research (http://ubuntuforums.org/archive/index.php/t-765671.html) I added the following line to /etc/modprobe.d/alsa_base:
options snd-hda-intel model=6stack-dig

This didn't help. In the same forum, they suggest to enable surround. In Edit/Preferences you can display the Surround slider.

And the magic happened: the sound workz :-). Setting up sound can be quite complex in Ubuntu...

update 15/11/2008: I had to repeat the same actions after upgrading to Ubuntu 8.10: add line to alsa-base file and enable the surround slider.

Rhythmbox meets MP3 players

2008-06-01T21:09:00.006+02:00

As stated in my previous post, FireFly is now serving my personal music collection from the CH3WNAS. This is nice for live streaming songs to Rhythmbox, but I also want to put some mp3s on my mp3 player.

I have 2 USB mass storage mp3 player: the Creative Muvo 100 and the Samsung YP-S5 with Asiatic v1.50 firmware. Rhythmbox has very simple support for those USB devices: simply add a ".is_audio_player" file in the root of the mp3 player (see: http://live.gnome.org/Rhythmbox/FAQ). The mp3 player is now automatically detected by Rhythmbox, and filling the player with songs from the FireFly server is as simple as drag'n'drop. The transfer is terribly slow, but that's probably because I'm connecting through wifi.

For both the Muvo 100 and the YP-S5 I put the following file in the root: .is_audio_player

audio_folders=Music/
folder_depth=2
output_formats=audio/mpeg

As the FAQ above mentions, this overrides the HAL. I didn't check (yet) if the default HAL settings are OK for the 2 devices. Anyway, this works fine for me.

Tip: with ctrl+H you can display the hidden files (=files that start with a dot '.') in Nautilus.

CH3WNAS and FireFly

2008-06-01T00:43:00.013+02:00

I managed to get FireFly running om my CH3WNAS. It's fairly easy if you're not afraid of the command line :-). I was wrong in my post yesterday: you do NOT need Gentoo in order to run firefly! Gentoo is apparently only needed if you want to build FireFly from the sources.

The info in http://dsmg600.info/howto:firefly contains too much steps for the CH3WNAS.

This is the 'real' way to setup firefly on CH3WNAS:

enable telnet in fun_plug: http://www.aroundmyroom.com/2008/01/03/ch3wnas-enabling-telnet/
download the latest build of FireFly for the CH3WNAS: http://willie-wortel.nl/ch3wnas/ch3wnas-ff-1696.tgz. Unzip the build onto the NAS. (/mnt/HD_a2)
I renamed the main directory from /mnt/HD_a2/firefly-1696 to /mnt/HD_a2/firefly. This is consequent with the config.
edit the following line in /mnt/HD_a2/firefly/mtdaapd.conf file (e.g. with ftp):
mp3_dir = /mnt/HD_a2/media
--> replace media with the root-directory to your mp3s.
--> check out the other options like default admin password if you wish.
add the following lines to your fun_plug:
#kill legacy upnp (if you don't use the upnp ;-)
kill -9 `pidof upnp`

#start firefly
rm -f /mnt/HD_a2/firefly/var/cache/mt-daapd/*
/mnt/HD_a2/firefly/sbin/mt-daapd -c /mnt/HD_a2/firefly/mt-daapd.conf
As you can see, I completely remove the DB before starting FireFly up. This is because while I was testing, the DB went corrupt after a reboot. The indexing at startup is very fast (4000 mp3s in 64 seconds) so this isn't really a problem, I think. Maybe there are more elegant ways to prevent corruption of the DB?

I tested it with Rhythmbox under Ubuntu 8.04 & it works like a charm :-). Make sure the "Multicast DNS service discovery" service (= mDNS) is running in Ubuntu (System/Administration/services) and that the "daap plugin" is checked in rhythmbox (edit/plugins). I'm seriously considering buying the Roku (Pinnacle) Soundbridge now...

Other tips:

check out the FireFly admin pages in your browser: go to port 3689 of your CH3WNAS:
e.g. http://192.168.1.30:3689/ user: admin; password: mt-daapd (default)
you can follow the logs in /var/log/mt-daapd.log (maybe not the best location :-/):
tail -f /var/log/mt-daapd.log

thanks to http://forums.fireflymediaserver.org/viewtopic.php?p=20282&sid=5b52a2902da3f5b3462bed5e1bc826fd and http://willie-wortel.nl/ch3wnas/!

(updated: 12/6/2008: corrected willie-wortel links)

YP-S5 and scratches

2008-05-31T22:36:00.006+02:00

The Samsung YP-S5 is particularly prone to scratches. I bought it a month ago, and I have only used it for about 3-4 hours, but there are already a few scratches on the shiny plastic. I think I put the YP-S5 in my pocket next to my keys...

I found there is a commercial shield to protect the YP-S5: http://mp3playersandheadphones.co.uk/the-invisible-shield-samsung-yp-k5yp-s5-full-body-shield/ and http://www.advancedmp3players.co.uk/shop/product_info.php?products_id=2339&aff=window. It's not clear to me if these products really help, without altering the the look & feel of the player. Time will tell...

Concerning the V1.50 firmware, I don't see much improvement compared to the previous V1.07 firmware. It still takes about 10 seconds to startup the device.

I like to listen to podcasts, but simply playing a few podcasts that are in different folders without manual intervention is almost impossible. I have also a simple Creative Muvo; the Muvo, with its smaller screen, is much better than the YP-S5 for playing podcasts. I like to play the podcasts in a sequential order, respecting the order of the podcasts that are under a directory-structure. On the YP-S5, I first tried to select all songs of style "podcast". But since not all podcasts set their style to 'podcast', it's not possible to use this attribute. I also tried the file-browser, but when the last song in a folder is reached it stops playing. It would have been better that the YP-S5 jumps to the next folder, and plays the first song in that folder. The only workable solution I found is to go to 'Albums', 'all' and than search for the podcasts. Unfortunatly, music and podcast are not separated.

CH3WNAS and Gentoo under chroot

2008-05-30T22:01:00.011+02:00

Update 28/6/2008: Although running gentoo is a nice exercise, there's no need for installing gentoo in order to run firefly or mediatomb. A this moment, I disabled gentoo on the ch3wnas. See http://michael-peeters.blogspot.com/2008/06/running-mediatomb-on-ch3wnas-natively.html and http://michael-peeters.blogspot.com/search/label/ch3WNAS for other, more useful, hacks.

I added the follwing line to my fun_plug script:

   kill -9 `pidof upnp`

This kills the standard shaky upnp server that is run on the CH3WNAS and constantly spawns errors in dmesg.

Next, I managed to run Gentoo under chroot on CH3WNAS. For this, I had to follow the steps on an excellent wiki: http://dsmg600.info/howto:chroot_gentoo. After ftping (i use gFtp) the files to the device and telneting to it, it's simply a matter of copy pasting the commands one-by-one. The 2 bzcat commands take some time to complete.

As I understand, the main purpose of running Gentoo under chroot is to have a recent Linux that can start modern binaries like dropbear (ssh), FireFly(iTunes specific daap), MediaTomb (uPnP), MLDonkey (torrent client), lighthttp (http server),...

For now, I simply have tried the bash:

/mnt/HD_a2/chroot /mnt/HD_a2/gentoo /bin/bash

This is currently my fun_plug:

#!/bin/sh

dmesg > /mnt/HD_a2/dmesg.out

if [ ! -e /mnt/HD_a2/ash ]
then
ln -s /mnt/HD_a2/busybox /mnt/HD_a2/ash
fi
if [ ! `grep root /etc/shadow` ]
then
echo kontroll.`grep admin /etc/shadow` >> /etc/shadow
/mnt/HD_a2/sed -i -e 's/kontroll.admin/root/' /etc/shadow
fi
/mnt/HD_a2/sed -i -e 's/root:.*/root:x:0:0:Linux User,,,:\/:\/mnt\/HD_a2\/ash/' /etc/passwd
cd /dev && /mnt/HD_a2/busybox makedevs ptyp c 2 0 0 9
cd /dev && /mnt/HD_a2/busybox makedevs ttyp c 3 0 0 9
/mnt/HD_a2/busybox telnetd &

#kill legacy upnp
kill -9 `pidof upnp`

#prepare gentoo stuff
mount -o bind /dev /mnt/HD_a2/gentoo/dev
mount -t proc none /mnt/HD_a2/gentoo/proc
cp /etc/resolv.conf /mnt/HD_a2/gentoo/etc/

#dropbear (ssh) stuff (not operational yet)
#ln -s /dev/urandom /dev/random
#/mnt/HD_a2/chroot /mnt/HD_a2/gentoo /usr/sbin/dropbear

dmesg > /mnt/HD_a2/dmesg.out2

I tried out dropbear (ssh), but I couldn't figure out what user/password to use when ssh-ing to the CH3WNAS. I did some unsuccessful experiments with copy/pasting the /etc/shadow and /etc/passwd under /mnt/HD_a2/gentoo. --> I'll look at it later, but now I have gentoo as a basis to try out other stuff.

Those hacks make the CH3WNAS very appealing for the more advanced user.

An idea for my future experiments: tweak the power button
http://forum.dsmg600.info/t955-chkbutton-replacement.html

CH3WNAS: Grab'n'Go Wireless Media Store

2008-05-29T20:15:00.010+02:00

One month ago, I bought a Conceptronic WiFi 500 GB Grab'n'Go Wireless Media Store (CH3WNAS) at about 130 EUR. This is cheap if you look at the features and compare it with other NAS-devices.

Check out the Conceptronic's website for more info.

I found a few minor glitches in the device. Those are not blocking in my case:

There is no way to turn the device automatically on after a power-cut. You have to press 'power on' button manually for starting the device up. The FAQ states this is hardware-related and can not be fixed. This is unfortunate if you want to reach your NAS while you're on holiday, or, if you want to routinely cut off the power to remove latent power consumption.

For a 'clean' shutdown, you have to keep the power-button pressed during 5s. At first, this gives the impression the device hangs and you have to force the shutdown -- at least that's how I shutdown a laptop when it hangs. This is also documented in the FAQ. I couldn't find any 'chkfs' recommendations in the manual to make sure the data is save.

There is also a 'shutdown' button in the web-interface that performs a clean remote shutdown. This looks more like a regular 'shutdown' command.
I was planning to use the CH3WNAS to replace my wifi router. Unfortunately, it has no support for NAT (and similar standard router-functionalities). Particularly with the built-in UPnP server, it's wise to keep the device behind your NAT. --> I'll keep my router in place to protect my home network; since the router itself has wifi (like almost any router nowadays), the Wifi server feature of the CH3WNAS is less appealing. Of course, the CH3WNAS could still act as a range extender if I install a long LAN-cable.
I connected a USB drive into the CH3WNAS to perform automatic backups. The web-interface has nice provisions for backing up. This is called 'scheduled downloads'. Not very intuitive, but after selecting 'folder' and clicking on 'local', the device does what it's expected to do: backup.

Unfortunately, the support for NTFS is very limited: the NTFS drive in the USB port is mounted as readonly and all folders/filenames are in uppercase. I reformatted the drive to FAT32 and now the device is supported correctly. A bit disappointing when you're used to Ubuntu and its out-of-the box ntfs-3g support, but I suppose the Linux-version on the device is a bit too old for NTFS (or the resources are too limited?).
There are reports that the built-in UPnP Server is primitive, but I couldn't verify this. I couldn't get the UPnP to work correctly from within Ubuntu.

The good news is that the CH3WNAS is a linux-device! There are a lot of tweaks out there:

enable telnet (make sure you're behind a NAT router: http://www.aroundmyroom.com/2008/01/03/ch3wnas-enabling-telnet/

The DSM-G600 appears to be a very similar device to the CH3WNAS. That's a good start for interesting hacks:

http://forum.dsmg600.info/viewforum.php?id=2

http://dsmg600.info/

I'm considering installing gentoo (under chroot) and Firefly, or better(?) MediaTomb. Those look like a killer-solution. --> next blog :-)

Switch the YP-S5 from MTP to UMS

2008-05-28T22:14:00.007+02:00

I managed to switch the YP-S5 from MTP to UMS by installing the Korean firmware v1.50. I don't understand why North America and Europe are stuck with the MTP version while Asia and south America get the UMS version. Perhaps Samsung wants to experiment with DRM on a limited scale?

MTP (Media Transport Protocol specific for Windows Mediaplayer) is really not user-friendly:

Doesn't work under macos or linux (support through libmtp is still limited). Ubuntu is my main OS .
I couldn't get it work under XP ("error mtp device")
Does work under Vista, but only after installing the latest patches.

Use at your own risk -- not sure if this voids your warrranty. But the YP-S5 did survive my unsuccesful attempts with the previous firmware v1.07.

Download the asiatic V1.50 version from Samsung Singapore (it's an English site):

http://www.samsung.com/sg/support/download...disp_nm=YP-S5QW

Unzip the file and under Vista, put the S5.rom and Config.dat in the root of the S5. I also put he Config.dat file in the subdirectory SYSTEM/ -- not sure if this is needed, but it doesn't harm... I had some trouble with copy/pasting the files. Drag and drop did however work under Vista.

After unconnecting and starting up the YP-S5, the new firmware is installed. The display shows up in Korean. This is quickly fixed in the Settings/Language menu. The first submenu item allows you to switch the language.

The S5 is now automatically supported under Ubuntu, but with a small issue: two S5 devices are mounted. No big deal...

See also http://www.anythingbutipod.com/forum/showthread.php?t=30155

UPDATE: Maybe it's not necessary to start from the Asiatic version; the 'original' V1.50 MTP version can also be switched by altering the Config.dat - this info seems unverified, however: http://mp3.generationmp3.com/2008/05...le-samsung-s5/

UPDATE 29/6/2008: apparently, you need a USB 2.0 connection to your YP-S5. See the comment below