Tools and techniques for performance tuning in Linux
Too Many Cache Misses?
The performance of the system is highly dependent on the effectiveness of the cache. Any cache miss will degrade performance and lead to a CPU stall.
Sometimes a cache miss is caused by frequently used fields located in data structures that span across the cache line. Oprofile can diagnose this kind of problem.
Again, using the Intel Core 2 processor as an example, choose the event LLC_MISSES to profile all the L2 cache requests that miss the L2 cache. For the exact event to use, you should invoke opcontrol --list-events to read about the details of each event type available for your CPU.
Listing 5 shows how to call up a cache miss profile.
Oprofile is a very versatile tool. By carefully choosing which events to monitor, you can zero in on the CPU operation that is causing the problem.
Listing 5
Cache Miss Profile
01 #opreport -l 02 CPU: Core 2, speed 1801 MHz (estimated) 03 Counted L2_RQSTS events (number of L2 cache requests) with a unit mask of 0x41 04 (multiple flags) count 90050 05 samples % app name symbol name 06 2803 63.4163 cc1 (no symbols) 07 190 4.2986 vmlinux-2.6.25-rc9-ltop get_page_from_freelist 08 102 2.3077 as (no symbols) 09 60 1.3575 vmlinux-2.6.25-rc9-ltop __lock_acquire 10 53 1.1991 libc-2.7.so strcmp 11 39 0.8824 vmlinux-2.6.25-rc9-ltop unmap_vmas 12 38 0.8597 vmlinux-2.6.25-rc9-ltop list_del
Locking Problems
A high context switching rate, relative to the number of running processes, is undesirable and could indicate a lock contention problem. To determine the most contended locks, enable the lock statistics in the kernel, which will give you insight into what is causing the lock contention. To do so, use the lock_stat feature in 2.6.23 or later kernels. First, you'll need to recompile the kernel with the CONFIG_LOCK_STAT=y option. Then, before running the workloads, clear the statistics with:
#echo 0 > /proc/lock_stat
After running the workload, review the lock statistics with the following command:
#cat /proc/lock_stat
The output of the preceding command is a list of locks in the kernel sorted by the number of contentions. For each lock, you will see the number of contentions, as well as the shortest, maximum, and cumulative wait time for a contention. In addition, you will see the number of acquisitions, as well as the minimum, maximum, and cumulative hold times for a lock. The top call sites of the lock are also given to let you locate quickly where in the kernel the lock occurs.
It is worth noting that the lock statistics infrastructure incurs overhead. Once you have finished hunting for locks, you should disable this feature to maximize performance.
Excessive Latency
Program throughput that is inconsistent and sputters, applications that seem to go to sleep before coming alive, and a lot of processes under the blocked column in vmstat are often signs of latency in the system. LatencyTOP is a new tool that helps diagnose latency issues.
Starting with the 2.6.25 kernel, you can compile LatencyTOP support into the kernel by enabling the CONFIG_HAVE_LATENCYTOP_SUPPORT=y and CONFIG_LATENCYTOP=y options in the kernel configuration. After booting up the kernel with LatencyTOP capability, you can trace latency in the workload with a userspace latency tracing tool from the LatencyTOP website [2]. To start, compile the tool, do a make install of the LatencyTOP program, and run the following as root:
#./latencytop
The LatencyTOP program's top screen (Figure 2) provides a periodic dump of the top causes that lead to processes being blocked, sorted by the maximum blocked time for each cause. Also, you'll find information on the percentage of time a particular cause contributed to the total blocked time. The bottom screen provides similar information on a per-process basis.
« Previous 1 2 3 Next »
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
News
-
Red Hat Enterprise Linux 7.5 Released
The latest release is focused on hybrid cloud.
-
Microsoft Releases a Linux-Based OS
The company is building a new IoT environment powered by Linux.
-
Solomon Hykes Leaves Docker
In a surprise move, Solomon Hykes, the creator of Docker has left the company.
-
Red Hat Celebrates 25th Anniversary with a New Code Portal
The company announces a GitHub page with links to source code for all its projects
-
Gnome 3.28 Released
The latest GNOME rolls out with better contact management and new features for handling virtual machines.
-
Install Firefox in a Snap on Linux
Mozilla has picked the Snap package system to deliver its application to Linux users.
-
OpenStack Queens Released
The new release comes with new features for mission critical workloads.
-
Kali Linux Comes to Windows
The Kali Linux developers even managed to run full blown XFCE desktop via WSL.
-
Ubuntu to Start Collecting Some Data with Ubuntu 18.04
It will be an ‘opt-out’ feature.
-
CNCF Illuminates Serverless Vision
The Cloud Native Computing Foundation announces a paper describing their model for a serverless ecosystem.