Stat-like command-line tools for admins
Summary
Sometimes you only have simple shell or crash cart access to a wayward node, so you can't run X. That means you have to rely on simple ASCII tools to help debug the problems.
I have found vmstat invaluable for diagnosing misbehaving nodes or checking or profiling user applications. For example, often when a user's application starts running slower, it's simply a matter of the user's application swapping on the compute nodes. A quick vmstat **1 **10
lets me see the problem and address it quickly. However, sometimes other issues require that you diagnose a node, and vmstat can't help. Fortunately, other people have run into the same problem and dstat was created. I tend to use dstat to get more information than I can get with vmstat.
Another tool I use, but not really to debug nodes, is mpstat, which helps me understand what my code is doing on a node. For example, if I'm writing OpenMP code, I can use mpstat to see if the cores are being used by user applications and, if so, how much. Moreover, you can use mpstat to diagnose performance problems that answer the question, "Why isn't my code running faster?"
Infos
- "Top-like tools for admins" by Jeff Layton, ADMIN, issue 23, pg. 86, http://www.admin-magazine.com/Archive/2014/23/Top-like-tools-for-admins
- vmstat: http://en.wikipedia.org/wiki/Vmstat
- vmstat man page: http://linux.die.net/man/8/vmstat
- Kibibyte: http://en.wikipedia.org/wiki/Kibibyte
- VMSTAT Analyzer: http://www.codeproject.com/Articles/41848/VMSTAT-Analyzer
- VMLoad: https://gist.github.com/mmichaelis/1187126
- vmstat plotter: http://clusterbuffer.wordpress.com/admin-tools/vmstat-plotter/
- Web-vmstat: http://linoxide.com/tools/vmstat-graphical-mode/
- dstat: http://dag.wiee.rs/home-made/dstat/
- dstat man page: http://dag.wiee.rs/home-made/dstat/dstat.1.html
- Systat: http://sebastien.godard.pagesperso-orange.fr/
- mpstat: http://sebastien.godard.pagesperso-orange.fr/man_mpstat.html
- OpenMP: http://en.wikipedia.org/wiki/OpenMP
« Previous 1 2
Buy this article as PDF
(incl. VAT)