Systemtap

From OMAPpedia

Contents 1 Targeted Audience 2 Systemtap Documentation 3 Systemtap Source Code 4 OMAP Systemtap Scripts/Tapsets 5 Systemtap and Cross Compilation 6 Systemtap Install 6.1 Pre-requisite 6.2 Kernel Code 6.3 Systemtap tool 6.3.1 Systemtap 1.0 code update for OMAP ARM platforms 6.3.2 Systemtap 1.3 code update for OMAP ARM platforms 6.3.3 Systemtap 1.5/1.6/1.7/1.8 code update for OMAP ARM platforms 6.4 Systemtap installation on the target 7 Systemtap Scripts compilation 8 Systemtap Scripts Execution 9 Hints on systemtap for embedded platforms 9.1 Mounting RAM file system 9.2 Time measurements

[edit] Targeted Audience

---

This page provides all the necessary details to enable systemtap on OMAP platforms (Android or Poky).

The information given below is the result of the integration on OMAP platform (ARM) of systemtap rel 1.0 with linux kernel rel 2.6.27 (and later releases). However, it is kept up to date with latest kernel and systemtap releases that have been tested

[edit] Systemtap Documentation

---

[1] SystemTap: Instrumenting the Linux Kernel for Analyzing Performance and Functional Problems

[2] Systemtap language reference

[edit] Systemtap Source Code

---

[3] links to systemtap GIT or HTTP or FTP

[edit] OMAP Systemtap Scripts/Tapsets

---

[4] OMAP tapsets and scripts releases (available soon)

[edit] Systemtap and Cross Compilation

---

As shown in the figure

Systemtap & cross compilation

systemtap execution consists in several phases:

1. Script ".stp" parsing, script analysis to find probes and translation in C code (Pass 1, 2 and 3)

2. Compilation of C code to kernel module * .ko (Pass 4)

3. The load and execution of the *.ko module

Originally, systemtap was designed so that all 3 steps take place on the same system (for eg Ubuntu PC). Indeed, the system translating/building the scripts was also the one whose performance had to be analyzed.

For embedded systems, the 4 steps are shared between a host platform (Ubuntu or other linux based environment) and the target platform (OMAP ARM).

To use systemtap for embedded systems, the HOST system shall now support:

1. Script parsing, script analysis to find probes and translation in C code (Pass 1, 2 and 3) => stap binary

2. * .ko module build (Pass 4) => staprun/stapio binaries

These 3 steps are performed by the stap command execution on the host platform.

On the other hand, the target platform shall support:

3. The load and execution of the *.ko module

This 3^rd step is performed by the staprun command execution on the target platform under analysis.

Naturally, the *.ko module(s) shall be copied from the HOST to the TARGET platform between step 2 and step 3.

It may be worth noting that once step 2 is completed and *.ko files are transferred, step 3 execution and completion solely relies on the TARGET platform; the HOST platform is not needed anymore

[edit] Systemtap Install

---

Following steps are needed:

• get elfutils package
• patch kernel code
• enable some kernel features and rebuild kernel
• get systemtap code and build Host/Target tools
• Build systemtap script into a kernel module
• Copy target tool and kernel module on target. Execute it

[edit] Pre-requisite

In order to compile the tool and scripts, elfutils package shall be available on the host.

elfutils packages can be taken from git clone git://git.fedorahosted.org/git/elfutils.git (you need autoconf to set-up library) or older revisions downloaded from following location [5], which is ready to be used

No need to install elfutils on the host; only the untar'ed file is required.

For the compilation examples given below, elfutils was untar'ed on the home directory

Therefore, elfutils path on the system taken as an example in this wiki page is /home/xxx/elfutils-0.143

[edit] Kernel Code

To run, systemtap requires the activation of the following kernel compilation flags:

• CONFIG_DEBUG_FS: Kernel Hacking -> Debug Filesystem
• CONFIG_KPROBES: General Setup -> Kprobes
• CONFIG_DEBUG_INFO: Kernel hacking -> Compile the kernel with debug info
• CONFIG_RELAY: General Setup -> user space relay support
• CONFIG_PROFILING: General Setup -> Profiling support (EXPERIMENTAL) + General Setup -> Oprofile system profiling (EXPERIMENTAL)
• CONFIG_TRACEPOINTS (via CONFIG_SLAB), generally already enabled

To do so, it is suggested to modify the kernel settings directly accessing menuconfig (make ARCH=arm menuconfig)

Furthermore, before launching the kernel compilation, one kernel file shall be updated so that systemtap release works for ARM architecture. Patch is no longer needed in recent kernels and can even crash it (Gingerbread)

The file is ptrace.h located in kernel/arch/arm/include/asm/ directory

#define interrupts_enabled(regs) \

    (!((regs)->ARM_cpsr & PSR_I_BIT))


#define fast_interrupts_enabled(regs) \

    (!((regs)->ARM_cpsr & PSR_F_BIT))


++  #define condition_codes(regs) \

++          ((regs)->ARM_cpsr & (PSR_V_BIT|PSR_C_BIT|PSR_Z_BIT|PSR_N_BIT))


/* Are the current registers suitable for user mode?

 * (used to maintain security in signal handlers)

 */

Once the extra lines are added, the kernel shall be recompiled as per the OMAP release build instructions.

[edit] Systemtap tool

Systemtap can allow you to trace and probe any part of the kernel. Different version of systemtap should be used according to you kernel version. the list under give the basic reference that have been done on TI kernel

* systemtap 1.0 : 2.6.32
* systemtap 1.3 : 2.6.35
* systemtap 1.5 : 2.6.39/3.0
* systemtap 1.6/1.7/1.8 : 2.6.39 -> 3.3
* systemtap 2.0: -> 3.6

[edit] Systemtap 1.0 code update for OMAP ARM platforms

Following patches are required to make the tool operate properly:

/systemtap-1.0/Translate.cxx [6]

/systemtap-1.0/runtime/Sym.h [7]

/systemtap-1.0/runtime/Sym.c [8]

• Systemtap compilation for the host

To compile systemtap executable used on the host side, proceed as follows:

./configure --exec-prefix=/home/xxx/systemtap-1.0-ubuntu --prefix=/home/xxx/systemtap-1.0-ubuntu --with-elfutils=/home/xxx/elfutils-0.143

make install

/home/xxx/systemtap-1.0-ubuntu will then be the default directory where executables will be generated.

• Systemtap compilation for the target

In order to compile the executable run on the target side (staprun), proceed with the following commands:

ac_cv_file__usr_include_nspr=no ac_cv_file__usr_include_nspr4=no ac_cv_file__usr_include_nss=no ac_cv_file__usr_include_nss3=no
 ac_cv_func_malloc_0_nonnull=yes ./configure --exec-prefix=/home/xxx/systemtap-1.0-macharm --prefix=/home/xxx/systemtap-1.0-macharm
 --host=arm-none-linux-gnueabi CXXFLAGS=-static CFLAGS=-static --disable-translator --disable-docs --disable-refdocs
 --disable-grapher --without-rpm

make install

In the example above, the directory containing the target executable will then be /home/xxx/systemtap-1.0-macharm

[edit] Systemtap 1.3 code update for OMAP ARM platforms

Since Kernel 2.6.35 Systemtap version needs to be updated to 1.3 in order to be able to run some of the scripts. New compilation procedure is described under. You need first to download EFL utils package like for Systemtap 1.0.

• Systemtap compilation for the host

To compile systemtap executable used on the host side, proceed as follows:

(You need to have codesourcery inside your PATH)

git clone git://sources.redhat.com/git/systemtap.git
cd systemtap
git checkout release-1.3
./configure --exec-prefix=/home/xxx/systemtap-1.3-ubuntu --prefix=/home/xxx/systemtap-1.3-ubuntu --with-elfutils=/home/xxx/elfutils-0.143

make install

/home/xxx/systemtap-1.3-ubuntu will then be the default directory where executables will be generated.

• Systemtap compilation for the target

In order to compile the executable run on the target side (staprun), proceed with the following commands:

make clean

ac_cv_file__usr_include_avahi_common=no ac_cv_file__usr_include_avahi_client=no ac_cv_file__usr_include_nspr=no ac_cv_file__usr_include_nspr4=no ac_cv_file__usr_include_nss=no ac_cv_file__usr_include_nss3=no
 ac_cv_func_malloc_0_nonnull=yes ./configure --exec-prefix=/home/xxx/systemtap-1.3-macharm --prefix=/home/xxx/systemtap-1.3-macharm
 --host=arm-none-linux-gnueabi CXXFLAGS=-static CFLAGS=-static --disable-translator --disable-docs --disable-refdocs
 --disable-grapher --without-rpm

make install

In the example above, the directory containing the target executable will then be /home/xxx/systemtap-1.3-macharm

• Post compilation patch

Change line 59 of /home/xxx/systemtap-1.3-macharm/share/systemtap/runtime/stack-arm.c (you will get an error later otherwise pointing to this file)

 --                _stp_print_addr((unsigned long)pc, verbose, task);
 ++             _stp_print_addr((unsigned long)pc, verbose, tsk);

[edit] Systemtap 1.5/1.6/1.7/1.8 code update for OMAP ARM platforms

Since Kernel 2.6.39 and 3.0 Systemtap version needs to be updated to 1.5 in order to be able to run some of the scripts. v1.7 is preferred and described below (use older versions by replacing 1.7 by right version number)

v1.8 is required for kernel 3.4. v1.8 requires elfutils v 0.148

New compilation procedure is described under. You need first to download ELF utils package like for Systemtap 1.0.

• Systemtap compilation for the host

To compile systemtap executable used on the host side, proceed as follows:

(You need to have codesourcery inside your PATH)

git clone git://sources.redhat.com/git/systemtap.git
cd systemtap
git clean -dfx
git checkout release-1.8
git checkout release-1.7
or git checkout 2cf32f06fb5d880ce4f5a9abbd4f9d3874a0b852
or git checkout release-1.5
./configure --exec-prefix=/home/xxx/systemtap-1.7-ubuntu --prefix=/home/xxx/systemtap-1.7-ubuntu --with-elfutils=/home/xxx/elfutils-0.143 (or 0.148)

make install

/home/xxx/systemtap-1.7-ubuntu will then be the default directory where executables will be generated.

• Systemtap compilation for the target

In order to compile the executable run on the target side (staprun), proceed with the following commands:

make clean

ac_cv_file__usr_include_avahi_common=no ac_cv_file__usr_include_avahi_client=no ac_cv_file__usr_include_nspr=no ac_cv_file__usr_include_nspr4=no ac_cv_file__usr_include_nss=no ac_cv_file__usr_include_nss3=no
 ac_cv_func_malloc_0_nonnull=yes ./configure --exec-prefix=/home/xxx/systemtap-1.7-macharm --prefix=/home/xxx/systemtap-1.7-macharm
 --host=arm-none-linux-gnueabi CXXFLAGS=-static CFLAGS=-static --disable-translator --disable-docs --disable-refdocs
 --disable-grapher --without-rpm

make install

In the example above, the directory containing the target executable will then be /home/xxx/systemtap-1.7-macharm

[edit] Systemtap installation on the target

The two previous steps consisted in generating systemtap executables for the host and the target. The tool directory generated for the target (/systemtap-1.x-macharm) shall by definition be transferred to the target.

One shall pay attention that the paths indicated for the tool compilation remain hard coded. In the example above, the tool running on the target is generated in a directory called /home/xxx/systemtap-1.0-macharm. It is important to keep the directory location and structure on the target or the tool will not solve all its code dependencies.

For example on Android:

• copy the folder in /data
• at each boot execute: ln -s /data/systemtap-1.x-macharm /home/xxx/systemtap-1.x-macharm (you may need to make "/" writable through mount -o remount,rw rootfs / then create /home/xxx)

Next, you need to copy kernel modules on the target. There are 2 solutions:

• copy modules anywhere (/.../systemtap-1.x-macharm/bin recommended) and launch them through staprun /.../zzz.ko
• by default, staprun program looks for *.ko modules in the /lib/modules/xxx/systemtap directory, where xxx is related to your kernel. Copy .ko there and launch them through staprun zzz without .ko suffix

(To get xxx, simply execute "staprun xyz" on the target, and an error will be displayed indicating the right path as: "ERROR: Unable to canonicalize path "/lib/modules/2.6.29-omap1-00034-g8b1c197-dirty/systemtap/xyz.ko": No such file or directory". Create this directory or a link to another directory with this name before copying your modules there)

Note that you may need to update file permissions to make staprun work

[edit] Systemtap Scripts compilation

---

Scripts compilation is performed through the command below:

export SYSTEMTAP_RUNTIME=SYSTEMTAPCFILEPATH
export SYSTEMTAP_TAPSET=FULLTAPSETPATH
./stap -gv -a arm -B CROSS_COMPILE=arm-none-linux-gnueabi- -r FULLKERNELPATH -m SYSTEMTAPMODULENAME SCRIPTNAME

Where:

• FULLKERNELPATH is the Kernel full path

• FULLTAPSETPATH is the reference tapset path. It shall not be necessary if you did "make install" and run from configured prefix. It is necessary if tool is at another location

• SYSTEMTAPCFILEPATH is the path to systemtap runtime directory. Same comment as above

• SYSTEMTAPMODULENAME is the name of the output module to be generated

• SCRIPTNAME is the script file (stp file)

On recent SW, typically a command example could be (from dir /systemtap-1.7-ubuntu/bin):

export SYSTEMTAP_RUNTIME=/home/xxx/systemtap-1.7-macharm/share/systemtap/runtime
export SYSTEMTAP_TAPSET=/home/xxx/systemtap-1.7-macharm/share/systemtap/tapset
./stap -gv -a arm -B CROSS_COMPILE=arm-none-linux-gnueabi- -r custom_path/mydroid/kernel/android-2.6.29 -m iotop .../iotop.stp

When completed, the command generates the *.ko file in the current directory (iotop.stp comes from http://sourceware.org/systemtap/examples/io/iotop.stp)

Additional compilation debug information can be accessed by adding several “v” in the compilation command (-gvvv for eg)

As explained earlier, the script file is translated into a C file prior to compilation. All files generated during these steps can be accessed in a temporary directory (location given during the compilation phase), providing the –k option is applied (-gvk considering the example above). If not working, use "-p 3" option

[edit] Systemtap Scripts Execution

---

See above section Systemtap installation on the target

[edit] Hints on systemtap for embedded platforms

---

[edit] Mounting RAM file system

Although the tool by default generates the traces on the open terminal, these traces can be redirected to the filesystem (-o option when invoking staprun).

However, experience has shown that these accesses to the filesystem could interfere with the assessment of the performance indicators gravitating around the filesystem (MMC card access for eg if filesystem is mounted onto an SD card).

Therefore, to alleviate the interference caused by the tool accesses to the filesystem, it is suggested to mount a filesystem onto the RAM as explained below. When redirected to the filesystem mounted onto the RAM, systemtap traces output will not interfere with the typical MMC activity running on the platform.

Quick instructions:

• Create a mount point, e.g. ‘mkdir /mnt/tmp’
• Mount the RAM disk: ‘mount -t tmpfs nodev /mnt/tmp’
• Optionally, you can add ‘-o size=32m’ to specify the maximum size reserved for this ‘disk’

Run systemtap scripts typing: staprun –o /mnt/tmp/output-trace.txt …

[edit] Time measurements

By default, today on ARM architecture, the only references to time measurement are the gettimeofday_s(), gettimeofday_ms() functions.

Indeed, the default implementation of get_cycle() function can not report any cycle counter value.

In order to read the 32k clock value on OMAP3430/3630/4430 platforms, a function has been implemented and is defined in the reference script file omap_timestamp.stp: get_32k() [9]

This function can be called in any other functions/probes to provide a time resolution of ~ 30µs (1/32768 s)

For higher resolution time measurements a general purpose timer can be used. The Systemtap OMAP Profiler tapset provides an example on how this can be done.