Debugging with strace
So far, I've used strace with very few options: just -o to send the output to a file. But a variety of options are available that make it easier to understand and debug programs.
If you want to use strace with any complicated programs, you'll probably want to be able to trace their forks as well. The -f switch traces child processes (forks) as they are created. -F will do the same thing, but with vforks. (vfork is a special case of fork that doesn't take certain sorts of information from the parent process.)
Another useful set of options have to do with time. These can come in handy if you're debugging a program that hangs or that seems to be running particularly slowly. In this way, you can find out where the problem actually lies. The -t, -tt, and -ttt switches all put the time of day – to the second, microsecond, or second+microsecond, written as seconds since epoch – at the start of each line. The -T switch shows the time spent in system calls – that is, the time difference between the beginning and end of each system call. Similarly, -r prints a relative time stamp at the start of each system call.
The -c switch counts the time, calls, and errors for each system call and summarizes this information when the program exits. This information is helpful if you're debugging and you're not sure where to start, because it will help you work out where the problem might lie.
Additionally, you can use -e trace=set to trace only the system calls you specify – for example, -e trace=open,read,write (note the lack of spaces) to trace only the open, read, and write system calls. This can minimize the amount of output you have to wade through if you've used -c to identify what you think are the problem calls.
Finally, if you want more gory details for your traces, try -v, which gives full values of stat, environment, and other common calls. (Use the -v flag and then have another look at the fstat call as discussed above to get all the information that's returned about that file.)
Shell Script Wrappers
Although it's not always possible to run your command under "real" conditions when something goes wrong, a possible workaround might be to write a shell script wrapper for the command so that you can run it via strace.
Move your program command out of the way by renaming it to, for example, command.old. Then save the script in Listing 4 to command.
Wrapping Another Command with strace
01 #!/bin/sh 02 # 03 strace -o/tmp/command_out.$$ /usr/bin/command.old $*
This not only calls the real command program with any arguments you used but also runs strace on it and dumps the output to a new file each time the script is called. Then you can check out what's happening under these real circumstances. Don't forget to move the real program back where it belongs once you're done!
Another very useful option is to use the -p PID option: This attaches strace to the process with the specified process ID. So if you suddenly see a process hanging or eating up lots of CPU, you can find out what it's doing in real time.
Knowing what system calls should look like and how they work makes it much easier to notice when something's not quite right. Often it's clear very quickly what's going wrong (e.g., a failed open call to a particular file). Looking at file permissions can also be helpful.
Strace is incredibly useful when debugging, but it can also be educational to run it on programs or commands that are running perfectly well, just to see what's going on under the hood of your working system.
One of the major advantages of running Linux is that the innards of the system are both accessible and (largely) well documented. Strace is one of the tools that can let you make the most of this access.
- strace article part 1: http://www.linuxpromagazine.com/issues/2009/103/bug_bumper
- strace tool: http://sourceforge.net/projects/strace/
A major setback for the Linux desktop.
Improved support for GPU in virtualization.
News site for the openSUSE community falls victim to a Wordpress exploit.
The source code is available online.
One out of three virtual machines on Microsoft Azure Cloud run Linux.
The form factor of the board makes it a drop-in replacement for Raspberry Pi.
Makes it easier for customers to move workloads into container-centric applications.
SUSE’s answer to container-centric operating systems.
Linux 4.9 is the biggest release in terms of number of commits.
The latest version of the official RHEL clone is here.