Debugging with strace
So far, I've used strace with very few options: just -o to send the output to a file. But a variety of options are available that make it easier to understand and debug programs.
If you want to use strace with any complicated programs, you'll probably want to be able to trace their forks as well. The -f switch traces child processes (forks) as they are created. -F will do the same thing, but with vforks. (vfork is a special case of fork that doesn't take certain sorts of information from the parent process.)
Another useful set of options have to do with time. These can come in handy if you're debugging a program that hangs or that seems to be running particularly slowly. In this way, you can find out where the problem actually lies. The -t, -tt, and -ttt switches all put the time of day – to the second, microsecond, or second+microsecond, written as seconds since epoch – at the start of each line. The -T switch shows the time spent in system calls – that is, the time difference between the beginning and end of each system call. Similarly, -r prints a relative time stamp at the start of each system call.
The -c switch counts the time, calls, and errors for each system call and summarizes this information when the program exits. This information is helpful if you're debugging and you're not sure where to start, because it will help you work out where the problem might lie.
Additionally, you can use -e trace=set to trace only the system calls you specify – for example, -e trace=open,read,write (note the lack of spaces) to trace only the open, read, and write system calls. This can minimize the amount of output you have to wade through if you've used -c to identify what you think are the problem calls.
Finally, if you want more gory details for your traces, try -v, which gives full values of stat, environment, and other common calls. (Use the -v flag and then have another look at the fstat call as discussed above to get all the information that's returned about that file.)
Shell Script Wrappers
Although it's not always possible to run your command under "real" conditions when something goes wrong, a possible workaround might be to write a shell script wrapper for the command so that you can run it via strace.
Move your program command out of the way by renaming it to, for example, command.old. Then save the script in Listing 4 to command.
Wrapping Another Command with strace
01 #!/bin/sh 02 # 03 strace -o/tmp/command_out.$$ /usr/bin/command.old $*
This not only calls the real command program with any arguments you used but also runs strace on it and dumps the output to a new file each time the script is called. Then you can check out what's happening under these real circumstances. Don't forget to move the real program back where it belongs once you're done!
Another very useful option is to use the -p PID option: This attaches strace to the process with the specified process ID. So if you suddenly see a process hanging or eating up lots of CPU, you can find out what it's doing in real time.
Knowing what system calls should look like and how they work makes it much easier to notice when something's not quite right. Often it's clear very quickly what's going wrong (e.g., a failed open call to a particular file). Looking at file permissions can also be helpful.
Strace is incredibly useful when debugging, but it can also be educational to run it on programs or commands that are running perfectly well, just to see what's going on under the hood of your working system.
One of the major advantages of running Linux is that the innards of the system are both accessible and (largely) well documented. Strace is one of the tools that can let you make the most of this access.
- strace article part 1: http://www.linuxpromagazine.com/issues/2009/103/bug_bumper
- strace tool: http://sourceforge.net/projects/strace/
Linux Foundation's big event celebrates the 25th anniversary of Linux
Competitors get in the game with RHEL without Red Hat
Security researchers have already notified Microsoft; some fixes are available
The company is collaborating with Google and Intel to use Kubernetes as an engine for Fuel
Customers can take a free test drive of SLES for HPC on the Azure Cloud
San Francisco-based chip company announces their first fully open source chip platform.
The whole distro gets rebuilt on glibc 2.3
Ubuntu Vendor tries to solve app packaging and distribution problem across distributions.