"Don't panic," thus the first advice from T'so in his presentation on hardware crashes. Never spare time with his second advice: "make backups!" T'so himself puts a lot of weight in scripts to secure data, but he also points to open source solutions like Amanda and Sync, or commercial variants like BRU and Arkeia. After presenting a few possible hardware catastrophe scenarios ("Linux hardware can do some pretty creative crashing"), T'so suggests looking for causes at the lowest level. "I apologize" he says, "but I have to admit that Linux error messages aren't among the most sysadmin friendly." His suggestion is to integrate kernel log messages into the search.

Kernel developer Ted Ts'o at LISA USENIX conference

His first solution is to use the image backup as a way to rescue at least what's left. T'so suggests having a good handle on the material to help in resolving problems. One area is understanding hard disk partitions and Logical Volume Manager (LVM), where T'so spent some time in an overview on filesystem monitoring. His example of an effective file recovery system is e2fsck, part of the Linux ext filesystem. His encouraging conclusion was, if you know where the problem emanated and if you take the proper care, most data recovery will be successful.

During his afternoon session, Ted T'so, also representing the Linux Foundation, launched into presenting SystemTap, a new developer tool. The software package targets technical support folks who can use it to tap into Linux kernel data noninvasively. T'so points to its effectiveness even in its current early phases. He cautions against using any of the other tools currently shipped with products, especially the enterprise versions where updates are few and far between.

Detailed information on SystemTap is available in the Sourceware wiki, which includes an FAQ and advice pages. Upcoming versions of SystemTap should include a Script Toolchest, and the wiki contains informtion on it already.