Debugging with strace
So far, I've used strace with very few options: just -o to send the output to a file. But a variety of options are available that make it easier to understand and debug programs.
If you want to use strace with any complicated programs, you'll probably want to be able to trace their forks as well. The -f switch traces child processes (forks) as they are created. -F will do the same thing, but with vforks. (vfork is a special case of fork that doesn't take certain sorts of information from the parent process.)
Another useful set of options have to do with time. These can come in handy if you're debugging a program that hangs or that seems to be running particularly slowly. In this way, you can find out where the problem actually lies. The -t, -tt, and -ttt switches all put the time of day – to the second, microsecond, or second+microsecond, written as seconds since epoch – at the start of each line. The -T switch shows the time spent in system calls – that is, the time difference between the beginning and end of each system call. Similarly, -r prints a relative time stamp at the start of each system call.
The -c switch counts the time, calls, and errors for each system call and summarizes this information when the program exits. This information is helpful if you're debugging and you're not sure where to start, because it will help you work out where the problem might lie.
Additionally, you can use -e trace=set to trace only the system calls you specify – for example, -e trace=open,read,write (note the lack of spaces) to trace only the open, read, and write system calls. This can minimize the amount of output you have to wade through if you've used -c to identify what you think are the problem calls.
Finally, if you want more gory details for your traces, try -v, which gives full values of stat, environment, and other common calls. (Use the -v flag and then have another look at the fstat call as discussed above to get all the information that's returned about that file.)
Shell Script Wrappers
Although it's not always possible to run your command under "real" conditions when something goes wrong, a possible workaround might be to write a shell script wrapper for the command so that you can run it via strace.
Move your program command out of the way by renaming it to, for example, command.old. Then save the script in Listing 4 to command.
Wrapping Another Command with strace
01 #!/bin/sh 02 # 03 strace -o/tmp/command_out.$$ /usr/bin/command.old $*
This not only calls the real command program with any arguments you used but also runs strace on it and dumps the output to a new file each time the script is called. Then you can check out what's happening under these real circumstances. Don't forget to move the real program back where it belongs once you're done!
Another very useful option is to use the -p PID option: This attaches strace to the process with the specified process ID. So if you suddenly see a process hanging or eating up lots of CPU, you can find out what it's doing in real time.
Knowing what system calls should look like and how they work makes it much easier to notice when something's not quite right. Often it's clear very quickly what's going wrong (e.g., a failed open call to a particular file). Looking at file permissions can also be helpful.
Strace is incredibly useful when debugging, but it can also be educational to run it on programs or commands that are running perfectly well, just to see what's going on under the hood of your working system.
One of the major advantages of running Linux is that the innards of the system are both accessible and (largely) well documented. Strace is one of the tools that can let you make the most of this access.
- strace article part 1: http://www.linuxpromagazine.com/issues/2009/103/bug_bumper
- strace tool: http://sourceforge.net/projects/strace/
Read full article as PDF:
Symantec says Linux-Darlioz burrows in through PHP.
Dell renews its quest for the ultimate developer machine.
Innovative back door looks like normal SSH traffic.
One of CeBITs most successful forums opens the new year with a new name. The popular Open Source Forum continues in 2014 under the name Special Conference: Open Source. This year, the forum will be bigger and offer a wider range of possibilities for sponsors.
New release offers better graphics drivers and expands filesystem support.
New mail protocol will shut out the NSA and prevent snooping on metadata.
A new web application helps users visualize distributed denial-of-service attacks.
Ubuntu 13.10 takes a step toward convergence, with lots of mobility, but Mir only partly here.
Galileo board is targeted to embedded developers and educational institutions.