Using LTTng

From OMAPpedia

Revision as of 21:56, 7 June 2011 by Greenscientist (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Jump to: navigation, search

The following wiki how to use LTTng (Linux Trace Toolkit - Next Generation). It describes its different functionnality and components, and to make it work on OMAP.

Note: The procedure was tested under linux-omap-2.6 kernel at v2.6.33 .

[edit] What is LTTng?

LTTng, the Linux Trace Toolkit Next Generation, is a project that aims at producing a highly efficient full system tracing solution.

It is composed of several components to allow tracing of the kernel, of userspace, trace viewing and analysis and trace streaming.

LTTng is open source software. It is being actively developed with the community.

LTTng itself is a kernel patch. It is used with

ltt-control, a toolchain to control tracing, and
LTTV, the LTT Viewer, a trace viewing and analysis program.

LTTV is designed to handle huge traces (10 GB and more). It has a plugin-based architecture and provides both graphical and text-mode user interfaces. LTTng includes a set of kernel instrumentation points useful for debugging a wide range of bugs, that are otherwise extremely challenging. These include, for example, performance problems on parallel systems and on real-time systems. Custom instrumentation is easy to add. LTTng is designed for minimal performance impact and has a near zero impact when not tracing. LTTng has at least basic support for all Linux-supported architectures.

[edit] Installation from sources

[edit] Prerequisites

You have to install the following libraries and programs to compile LTT-ng and his components on the host PC.

$ sudo apt-get install libstdc++6 libgtk2.0-dev libglib2.0-dev libpopt-dev libpango1.0-dev libc6-dev

Check the compatibilities between LTTng, ltt-control, LTTV and the Linux Kernel at Compatibility List

Note: LTTng was tested under v2.6.33 with the following sources LTTng, lttctl and LTTV.

[edit] Getting and apply LTTng patches

Note: if you use the patches above, you can jump the wget instruction

In the kernel sources directory, execute the following instructions.

$ wget http://lttng.org/files/lttng/patch-2.6.X-lttng-0.x.xx.tar.gz
(see http://lttng.org/files/lttng for package listing)
$ tar jxvf patch-2.6.X-lttng-0.x.xx.tar.gz
$ mv patch-2.6.X-lttng-0.x.xx patches
$ quilt push -a

[edit] Building a LTTng kernel

First of all, you have to configure your kernel

$ make CROSS_COMPILE=arm-none-linux-gnueabi- ARCH=arm distclean
$ make CROSS_COMPILE=arm-none-linux-gnueabi- ARCH=arm omap_zoom2_defconfig 
$ make CROSS_COMPILE=arm-none-linux-gnueabi- ARCH=arm xconfig

Set the following in xconfig

General setup -> CONFIG_LTT
General setup -> Linux Trace Toolkit Next Generation -> CONFIG_LTT_TRACEPROBES
General setup -> Linux Trace Toolkit Next Generation -> CONFIG_LTT_DEBUG_EVENT_SIZE
General setup -> Linux Trace Toolkit Next Generation -> CONFIG_LTT_USERSPACE_EVENT
General setup -> Linux Trace Toolkit Next Generation -> CONFIG_LTT_STATEDUMP

Note: Some of the previous options will be already activated with a dot in xconfig. It means that they will be compile as module, but for more simplicity check them a second time to compile them directly in the kernel.

Compile the kernel

$ make CROSS_COMPILE=arm-none-linux-gnueabi- ARCH=arm uImage

(For further information about the compilation of the kernel see Linux OMAP Kernel Project)

[edit] Editing the system wide configuration

You must activate debugfs and specify a mount point. This is typically done in fstab such that it happens at boot time. If you have never used DebugFS before, these operation would do this for you :

# mkdir /mnt/debugfs
# cp /etc/fstab /etc/fstab.lttng.bkp
# echo "debugfs         /mnt/debugfs    debugfs rw              0       0"  >> /etc/fstab
# mount /mnt/debugfs

NOTE: LTTng has to be mount in the /mnt/debugfs directory, otherwise ltt-control won't work.

[edit] Getting and installing the ltt-control package

$ wget http://lttng.org/files/lttng/ltt-control-0.x-xxxx20xx.tar.gz
$ tar zxvf ltt-control-0.x-xxxx20xx.tar.gz
$ cd ltt-control-0.x-xxxx20xx
$ ./configure --host=arm-none-linux-gnueabi --prefix=/tmp/mmc2
$ make
$ sudo -s
# make install

These commands will install ltt-control into the mmc. To finish the installation, type the following command on the board.

# export LTT_DAEMON=/bin/lttd

[edit] Getting and installing the LTTV package (on the visualisation machine)

$ wget http://lttng.org/files/packages/lttv-0.x.xx-xxxx20xx.tar.gz
$ tar zxvf lttv-0.x.xx-xxxx20xx.tar.gz
$ cd lttv-0.x.xx-xxxx20xx
$ ./configure
$ make
$ sudo -s
# make install
(run ldconfig to ensure new shared objects are taken into account)
# ldconfig

[edit] Using LTTng

IMPORTANT : Arm Linux Kernel Markers after each boot before tracing

# ltt-armall

[edit] Use text mode LTTng to control tracing

The tracing can be controlled from a terminal by using the lttctl command.

Start tracing :

# lttctl -C -w /tmp/trace1 trace1

Stop tracing and destroy trace channels :

# lttctl -D trace1

NOTE: to see if the buffers has been filled, look at the dmesg output after lttctl -D, it will show an event lost count. If it is the case, try using larger buffers. See lttctl --help to learn how. lttv now also shows event lost messages in the console when loading a trace with missing events or lost subbuffers.)

[edit] Tracing in "Hybrid" mode

Another mode can be use to trace data : hybrid. It can be especially useful when studying big workloads on a long period of time.

When using this mode, the most important, low rate control information will be recorded during all the trace by lttd (i.e. process creation/exit). The high rate information (i.e. interrupt/traps/syscall entry/exit) will be kept in a flight recorder buffer (now named flight-channelname_X).

The following lttctl commands take an hybrid trace :

Create trace channel, start lttd on normal channels, start tracing:
# lttctl -C -w /tmp/trace2 -o channel.kernel.overwrite=1 trace2

Stop tracing, start lttd on flight recorder channels, destroy trace channels :
#lttctl -D -w /tmp/trace2 trace2

Each "overwrite" channel is flight recorder channel.

[edit] Tracing in flight recorder mode

The flight recorder mode writes data into overwritten buffers for all channels, including control channels, except for the facilities tracefiles. It consists of setting all channels to "overwrite". it will dump only the last buffers to disk.

The following lttctl commands take a flight recorder trace :

lttctl -C -w /tmp/trace3 -o channel.all.overwrite=1 trace3
...
lttctl -D -w /tmp/trace3 trace3

[edit] Analyse Trace Data with LTTV

The trace are composed of several file which contain a lot of metadata not understanding by a human beeging. That's why we have to use LTTV to analyse all this among of data. LTTV can be use by two different ways:

Text-mode
GUI Mode

[edit] LTTV Text mode

The text mode will formatted the data to understandable text file which contains every informations. These files can be very heavy, but still usefull if we need to looking for something into the trace with a grep command.

A simple trace dump in text format is available with :

$ lttv -m textDump -t /tmp/trace

If you use the --help option on the textDump module, you will see all the detail about the switches that can be used to show per cpu statistics and per process statistics.

[edit] Use LTTV GUI mode

LTTV

This is the main part of LTTV, it have a module architecture which allow to enable or disable some library options Type the following command to launch LTTV

$ lttv-gui

A user guide of LTTV is available here

Note The white dots in the LTTng viewer indicate that some event cannot be resolved at the current display resolution (this is why the dots disappear when zooming).

[edit] Adding new instrumentations with the markers

[edit] Adding kernel instrumentation

See Documentation/markers.txt and Documentation/trace/tracepoints.txt in your kernel tree.

Also see ltt/probes/ for LTTng probe examples.

[edit] Adding userspace instrumentation

The easy quick-and-dirty way to perform userspace tracing is currently to write an string to /mnt/debugfs/ltt/write_event.

echo "some text to record" > /mnt/debugfs/ltt/write_event

It will appear in the trace under event :

channel : userspace
event name : event

LTTng has a specific tools for tracing userspace activity (LTTng-UST) , but this one works only on x86-32 and x86-64 architecture.

[edit] LTTng on Android

Note: For more informations about Android see here.

A series of patch was developed to enable LTTng on Android. After applying the patches, LTTng works the same way as the versions above. Unfortunately, some options of LTTng (PM support for example) are not enabled, but the main option of tracing works with good performance.

LTTng for Android can work only with old versions of lttcontrol and LTTV (According to the Compatibility List). The following tools was tested, and works with this version of LTTng.

[edit] Benchmark

As every measuring tool, LTTng brings an overhead for every function call by the kernel. To get a good idea of the overhead, three benchmark was tested

Hackbench, this test is a benchmark for measuring the performance, overhead, and scalability of the Linux scheduler. Created by Rusty Russell, it uses client and server processes grouped to send and receive data in order to simulate the connections established for a chat room. Each client sends a message to each server in the group.
Interbench, It is designed to measure the effect of changes in Linux kernel design or system configuration changes such as cpu, I/O scheduler and filesystem changes and options.
Duration for uncompress a large .tar.gz file.

[edit] Hackbench

Hackbench of LTTng

According to this graph, we can see that LTTng brings an overhead when the trace is active, its brings an average of 90% of overhead for this task which implies a lot of scheduler switch. The most important point is that LTTng practically doesnt influence the system when the trace is unactive (only 3% of overhead for this example). Comparing to FTrace event tracer, which trace similar things, the overhead of LTTng is much smaller because Ftrace event tracer can bring until 230% percent of overhead, and also Ftrace has always a non negligible overhead even when the tracer is off.

[edit] Uncompressing a large .tar.gz file

Decompression of a large tar.gz file

This test consists to decompressing a large tar.gz file (an ubuntu minimal filesystem ~90MB) and measuring the time elapse to do it. During this test, we can see that the overhead of LTTng is low in any case. The Hybrid Mode and mostly the Flight record mode have a better overhead. The explication come from the fact that these mode don't record every trace into file but some of them are written only in a stamp

[edit] For more Information

Comparison with SystemTap and Dtrace
The LTTng tracer : A Low Impact Performance and Behavior Monitor for GNU/Linux: paper/slide
LTTng Architecture
Deploying LTTng on Exotic Embedded Architectures with ARM OMAP3 Example: paper/slides/video
The state of linux tracing