Tools and techniques for performance tuning in Linux
Too Many Cache Misses?
The performance of the system is highly dependent on the effectiveness of the cache. Any cache miss will degrade performance and lead to a CPU stall.
Sometimes a cache miss is caused by frequently used fields located in data structures that span across the cache line. Oprofile can diagnose this kind of problem.
Again, using the Intel Core 2 processor as an example, choose the event LLC_MISSES to profile all the L2 cache requests that miss the L2 cache. For the exact event to use, you should invoke opcontrol --list-events to read about the details of each event type available for your CPU.
Listing 5 shows how to call up a cache miss profile.
Oprofile is a very versatile tool. By carefully choosing which events to monitor, you can zero in on the CPU operation that is causing the problem.
Cache Miss Profile
01 #opreport -l 02 CPU: Core 2, speed 1801 MHz (estimated) 03 Counted L2_RQSTS events (number of L2 cache requests) with a unit mask of 0x41 04 (multiple flags) count 90050 05 samples % app name symbol name 06 2803 63.4163 cc1 (no symbols) 07 190 4.2986 vmlinux-2.6.25-rc9-ltop get_page_from_freelist 08 102 2.3077 as (no symbols) 09 60 1.3575 vmlinux-2.6.25-rc9-ltop __lock_acquire 10 53 1.1991 libc-2.7.so strcmp 11 39 0.8824 vmlinux-2.6.25-rc9-ltop unmap_vmas 12 38 0.8597 vmlinux-2.6.25-rc9-ltop list_del
A high context switching rate, relative to the number of running processes, is undesirable and could indicate a lock contention problem. To determine the most contended locks, enable the lock statistics in the kernel, which will give you insight into what is causing the lock contention. To do so, use the lock_stat feature in 2.6.23 or later kernels. First, you'll need to recompile the kernel with the CONFIG_LOCK_STAT=y option. Then, before running the workloads, clear the statistics with:
#echo 0 > /proc/lock_stat
After running the workload, review the lock statistics with the following command:
The output of the preceding command is a list of locks in the kernel sorted by the number of contentions. For each lock, you will see the number of contentions, as well as the shortest, maximum, and cumulative wait time for a contention. In addition, you will see the number of acquisitions, as well as the minimum, maximum, and cumulative hold times for a lock. The top call sites of the lock are also given to let you locate quickly where in the kernel the lock occurs.
It is worth noting that the lock statistics infrastructure incurs overhead. Once you have finished hunting for locks, you should disable this feature to maximize performance.
Program throughput that is inconsistent and sputters, applications that seem to go to sleep before coming alive, and a lot of processes under the blocked column in vmstat are often signs of latency in the system. LatencyTOP is a new tool that helps diagnose latency issues.
Starting with the 2.6.25 kernel, you can compile LatencyTOP support into the kernel by enabling the CONFIG_HAVE_LATENCYTOP_SUPPORT=y and CONFIG_LATENCYTOP=y options in the kernel configuration. After booting up the kernel with LatencyTOP capability, you can trace latency in the workload with a userspace latency tracing tool from the LatencyTOP website . To start, compile the tool, do a make install of the LatencyTOP program, and run the following as root:
The LatencyTOP program's top screen (Figure 2) provides a periodic dump of the top causes that lead to processes being blocked, sorted by the maximum blocked time for each cause. Also, you'll find information on the percentage of time a particular cause contributed to the total blocked time. The bottom screen provides similar information on a per-process basis.
Richard Stallman calls for the W3C to remain independent of vendor interests.
The new release supports nine architectures, 73 human languages, and zero non-Free components.
Fedora developers release the first alpha version of Fedora 19, known as Schrödinger’s Cat, for general testing. The final release is expected in July 2013.
ack is a grep-like, command-line tool that has been optimized for programmers to search large trees of source code.
New features in SUSE Studio 1.3 include enhanced cloud integration, VM platform support, and lifecycle management.
The Linux Foundation recently announced that the Xen Project is becoming a Linux Foundation Collaborative Project.
Open source version of LiveCode is now available for developing apps, games, and utilities for all major platforms.
OpenDaylight is an open source software-defined networking project committed to furthering adoption of SDN and accelerating innovation in a vendor-neutral and open environment.
The new Gnome release includes privacy and sharing settings, allowing more user control over access to personal information.
Mozilla is collaborating with Samsung on a new web browser engine called Servo.