Debugging with strace

Options

So far, I've used strace with very few options: just -o to send the output to a file. But a variety of options are available that make it easier to understand and debug programs.

If you want to use strace with any complicated programs, you'll probably want to be able to trace their forks as well. The -f switch traces child processes (forks) as they are created. -F will do the same thing, but with vforks. (vfork is a special case of fork that doesn't take certain sorts of information from the parent process.)

Another useful set of options have to do with time. These can come in handy if you're debugging a program that hangs or that seems to be running particularly slowly. In this way, you can find out where the problem actually lies. The -t, -tt, and -ttt switches all put the time of day – to the second, microsecond, or second+microsecond, written as seconds since epoch – at the start of each line. The -T switch shows the time spent in system calls – that is, the time difference between the beginning and end of each system call. Similarly, -r prints a relative time stamp at the start of each system call.

The -c switch counts the time, calls, and errors for each system call and summarizes this information when the program exits. This information is helpful if you're debugging and you're not sure where to start, because it will help you work out where the problem might lie.

Additionally, you can use -e trace=set to trace only the system calls you specify – for example, -e trace=open,read,write (note the lack of spaces) to trace only the open, read, and write system calls. This can minimize the amount of output you have to wade through if you've used -c to identify what you think are the problem calls.

Finally, if you want more gory details for your traces, try -v, which gives full values of stat, environment, and other common calls. (Use the -v flag and then have another look at the fstat call as discussed above to get all the information that's returned about that file.)

Shell Script Wrappers

Although it's not always possible to run your command under "real" conditions when something goes wrong, a possible workaround might be to write a shell script wrapper for the command so that you can run it via strace.

Move your program command out of the way by renaming it to, for example, command.old. Then save the script in Listing 4 to command.

Listing 4

Wrapping Another Command with strace

01 #!/bin/sh
02 #
03 strace -o/tmp/command_out.$$ /usr/bin/command.old $*

This not only calls the real command program with any arguments you used but also runs strace on it and dumps the output to a new file each time the script is called. Then you can check out what's happening under these real circumstances. Don't forget to move the real program back where it belongs once you're done!

Another very useful option is to use the -p PID option: This attaches strace to the process with the specified process ID. So if you suddenly see a process hanging or eating up lots of CPU, you can find out what it's doing in real time.

Conclusion

Knowing what system calls should look like and how they work makes it much easier to notice when something's not quite right. Often it's clear very quickly what's going wrong (e.g., a failed open call to a particular file). Looking at file permissions can also be helpful.

Strace is incredibly useful when debugging, but it can also be educational to run it on programs or commands that are running perfectly well, just to see what's going on under the hood of your working system.

One of the major advantages of running Linux is that the innards of the system are both accessible and (largely) well documented. Strace is one of the tools that can let you make the most of this access.

The Author

Juliet Kemp has been playing around with Linux ever since she found out that it was more fun than Finals revision, and has been a sys admin for about five years. She finds strace output deeply fascinating, and had great fun delving into system call man pages while researching this article.

Read full article as PDF:

Related content

  • strace

    Get started with strace by examining a pair of "Hello World" programs. Next month, in the second part of this two-part series, I'll take a deeper look at strace output.

  • Tracing Tools

    Programs rarely reveal what they are doing in the background, but a few clever tools, of interest to both programmers and administrators, monitor this activity and log system functions.

  • Perl: Ptrace

    Linux lets users watch the kernel at work with a little help from Ptrace, a tool that both debuggers and malicious process kidnappers use. A CPAN module introduces this technology to Perl and, if this is not enough, C extensions add functionality.

  • How to Write a Rootkit

    Today’s rootkits infiltrate a target system at kernel level, thus escaping unwanted attention from administrators. Read on for a practical look at how a kernel rootkit really works.

comments powered by Disqus

Direct Download

Read full article as PDF:

News