Tracing shared library calls with ltrace and latrace

Tracing shared library calls with ltrace

Similar to strace, which can be used to trace system calls, ltrace can be used to trace shared library calls. This can be very useful to get insight into the program flow of applications while analyzing an issue. Consider the following simple sample: hello.c:

include <stdio.h>

void helloFunc(const char* s) {


void helloFunc(const char* s);


#include "hello.h"

int main() {
   helloFunc("Hello World, how are you?");
   return 0;

We create a shared library from hello.c:

$ gcc -fPIC --shared -o libhello.so hello.c

And we create the executable from helloMain.c which links against the library created before:

$ gcc -o hello helloMain.c -lhello -L.

We can now use ltrace to see which library calls are executed:

$ export LD_LIBRARY_PATH=.
$ ltrace ./hello 
__libc_start_main(0x40069d, 1, 0x7fffe2f3b778, 0x4006c0 <unfinished ...>
helloFunc(0x400744, 0x7fffe2f3b778, 0x7fffe2f3b788, 0)              = 25
Hello World, how are you?+++ exited (status 0) +++

One drawback with ltrace is that it only traces calls from the executable to the libraries the executable is linked against – it does not trace calls between libraries! Hence, the call to the printf() function (which itself resides in the libc shared library) is not shown in the output. Also, there is no option to include the library name in the output for each of the called functions.

Tracing shared library calls with latrace

Especially for larger applications, a better alternative to ltrace is latrace which uses the LD_AUDIT feature from the libc library (available from libc 2.4 onward). On Ubuntu, it can be installed with

$ sudo apt-get install latrace

When using latrace with the sample program from above, there are two important differences:

  • First, latrace also traces library calls between shared libraries, so the output includes the printf call executed from our own shared library.
  • Second, the shared library name which contains the symbol is printed after each function call:

$ latrace ./hello
10180     _dl_find_dso_for_object [/lib64/ld-linux-x86-64.so.2]  
10180     __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]  
10180       helloFunc [./libhello.so]  
10180         printf [/lib/x86_64-linux-gnu/libc.so.6]  
10180       __tls_get_addr [/lib64/ld-linux-x86-64.so.2]  
10180       __cxa_finalize [/lib/x86_64-linux-gnu/libc.so.6]  
Hello World, how are you?
./hello finished - exited, status=0

By default, function parameters are not shown, but this can be enabled with the -A option:

$ latrace -A ./hello
10190     _dl_find_dso_for_object [/lib64/ld-linux-x86-64.so.2]  
10190     __libc_start_main(main = 0x40069d, argc = 1, ubp_av = 0x7fffccd35248, auxvec = 0x4006c0, init = 0x400730, fini = 0x7f3d16b21560, rtld_fini = 0x7fffccd35238) [/lib/x86_64-linux-gnu/libc.so.6] {
10190       helloFunc [./libhello.so]  
10190         printf(format = "Hello World, how are you?") [/lib/x86_64-linux-gnu/libc.so.6] {
Hello World, how are you?10190         } printf = 25
10190       __tls_get_addr [/lib64/ld-linux-x86-64.so.2]  
10190       __cxa_finalize(ptr = 0x7f3d1650b030) [/lib/x86_64-linux-gnu/libc.so.6] {
10190       } __cxa_finalize = void

With -A, ltrace uses some configuration files at /etc/latrace.d to define the parameter format for a set of well-known functions such as printf. We can see in the output that, even though helloFunc() takes a parameter, this parameter is not shown in the output (since it is not defined in the configuration files). We can use the -a option to specify our own argument definition file. An argument definition file is essentially a header file which defines the prototypes for all functions for which the arguments should be displayed in the output. On Ubuntu, the default argument definition files are stored at /etc/latrace.d/headers/, and there is a master file /etc/latrace.d/headers/latrace.h which includes the other header files. We can use the same approach in our own definition file, by first including the master file from /etc/latrace.d/headers/latrace.h and then add the prototypes for each function we want to trace. For our ssample above, the file could look like

#include "/etc/latrace.d/headers/latrace.h"
void helloFunc(const char* s);

Assumed this file is called mylatrace.h, wen can now use the -a option to pass the file to latrace:

$ latrace -a mylatrace.h -A ./hello
10801     _dl_find_dso_for_object [/lib64/ld-linux-x86-64.so.2]  
10801     __libc_start_main(main = 0x40069d, argc = 1, ubp_av = 0x7fff4b89fb58, auxvec = 0x4006c0, init = 0x400730, fini = 0x7f1927ea1560, rtld_fini = 0x7fff4b89fb48) [/lib/x86_64-linux-gnu/libc.so.6] {
10801       helloFunc(s = "Hello World, how are you?") [./libhello.so] {
10801         printf(format = "Hello World, how are you?") [/lib/x86_64-linux-gnu/libc.so.6] {
10801         } printf = 25
10801       } helloFunc = void
10801       __tls_get_addr [/lib64/ld-linux-x86-64.so.2]  
Hello World, how are you?10801       __cxa_finalize(ptr = 0x7f192788b030) [/lib/x86_64-linux-gnu/libc.so.6] {
10801       } __cxa_finalize = void
./hello finished - exited, status=0

As you can see in the output, we now also see that helloFunc() is called with one parameter s which is set to “Hello World, how are you?”.

Defining a custom core file handler

I recently was wondering how apport can intercept core files written by the Linux kernel. Essentially, there is a kernel interface which allows to execute arbitrary commands whenever the kernel generates a core file. Earlier, this was used to fine tune the filename of the core file, like adding a time stamp or the user id of the process which generated the core file, instead of just plain core. The file name pattern can be defined through a special file located at /proc/sys/kernel/core_pattern. Since kernel 2.6.19, /proc/sys/kernel/core_pattern also supports a pipe mechanism. This allows to send the whole core file to stdin of an arbitrary program which can then further handle the core file generation. Additional parameters like the process id can be passed to the command line arguments of the program by using percent specifiers. On Ubuntu, by default /proc/sys/kernel/core_pattern contains the following string:

|/usr/share/apport/apport %p %s %c %P

This means to send the core file to stdin of /usr/share/apport/apport, and pass additional parameters like the process id to the command line parameters. See https://man7.org/linux/man-pages/man5/core.5.html for more information about the supported % specifiers.

Example: automatically launching a debugger

It is also possible to execute a shell script, which makes it very easy to execute specific actions whenever a core file is generated. Lets assume we want to launch the gdb debugger each time a core file is created, load the crashed program together with the core file and automatically show the call stack where the program crashed. This can be achieved by the following script:


# Get parameters passed from the kernel
EXE=`echo $1 | sed -e "s,!,/,g"`
EXEWD=`dirname ${EXE}`

# Read core file from stdin
cat > ${COREFILE}

# Launch xterm with debugger session
xterm -display :1 -e "gdb ${EXE} -c ${COREFILE} -ex \"where\"" &

Now, all we need to do is to register the script in /proc/sys/kernel/core_pattern (we need to do this as root, of course). Assumed that the script is stored as /tmp/handler.sh, we can use the following command to have the kernel execute it whenever a core file is to be written:

# echo '|/tmp/handler.sh %E %p %s %c %P %u %g %t %h %e' > /proc/sys/kernel/core_pattern

Fsor the script above, we would only need the %E and %t specifiers, but by passing all available parameters we can adjust the script without the need to modify /proc/sys/kernel/core_pattern when additional parameters are required. From now on, whenever a core dump is generated, an xterm window will open, gdb will be launched, the crashed file together with the core dump will be loaded into the debugger and the where command will be executed to show the call stack up to the location where the program crashed. The following screenshot shows the execution of the stack smashing sample I wrote about earlier.

Note: the xterm and all programs within it will be run as root user, so be careful with what you do inside the xterm!

The GS segment and stack smashing protection

When disassembling (32 bit i386 / x86) code on Linux, we sometimes come across instructions like

call   *%gs:0x10
movl    %gs:0x14, %eax

Here, gs refers to the Thread Control Block (TCB) header which stores per-CPU and thread local data (Thread Local Storage, TLS). The Thread Control Block header is a structure which is defined in the C library, for example in eglibc-2.19/nptl/sysdeps/i386/tls.h (slightly simplified and added the gs segment offsets):

typedef struct {
  void *tcb;              /* gs:0x00 Pointer to the TCB. */
  dtv_t *dtv;             /* gs:0x04 */
  void *self;             /* gs:0x08 Pointer to the thread descriptor.  */
  int multiple_threads;   /* gs:0x0c */
  uintptr_t sysinfo;      /* gs:0x10 Syscall interface */
  uintptr_t stack_guard;  /* gs:0x14 Random value used for stack protection */
  uintptr_t pointer_guard;/* gs:0x18 Random value used for pointer protection */
  int gscope_flag;        /* gs:0x1c */
  int private_futex;      /* gs:0x20 */
  void *__private_tm[4];  /* gs:0x24 Reservation of some values for the TM ABI.  */
  void *__private_ss;     /* gs:0x34 GCC split stack support.  */
} tcbhead_t;

Lets take a closer look at gs:0x10 which I mentioned above: The stack_guard member contains a random value which is used to protect against stack smashing – consider the following sample which contains an obvious buffer overflow:

#include <string.h>

int main() {
   char buffer[4];
   strcpy(buffer, "Hello World");
   return 0;

When we compile and run this application, we will get a runtime error:

$ gcc -m32 -o smash smash.c
$ ./smash 
*** stack smashing detected ***: ./smash terminated
Aborted (core dumped)

Lets look at the code which is generated by the compiler:

$ gcc -m32 -S -o smash.S smash.c

The following is the (simplified and commented) smash.S file:

; Set up the stack
        pushl   %ebp
        movl    %esp, %ebp
        andl    $-16, %esp
        subl    $16, %esp

; Set up stack guard
        movl    %gs:20, %eax       ; load random value
        movl    %eax, 12(%esp)     ; store value as a guard variable 
        xorl    %eax, %eax         ; make sure that noone can read the random value afterwards
; strcpy ommitted (this overwrites 12(%esp) since the string is too large for the buffer)

; Check stack guard against value from TCB
        movl    12(%esp), %edx     ; load previously stored value from stack
        xorl    %gs:20, %edx       ; check if still the same
        je      .L3                ; yes, then fine
        call    __stack_chk_fail   ; print error message and abort()

As we see, the compiler inserts instructions at the beginning of the function to set up the stack guard variable, and it inserts instructions at the end of the function to check if the value has changed meanwhile (means, if any code within the function has written beyond the area allocated for local variables and hence has potentially overwritten the return address for the current function). The generation of the stack protection code can be disabled by passing the -fno-stack-protector option to gcc – this is something which should, of course, normally not be done but it can sometimes be useful in order to analyze certain security issues.

Passing an array to a function which expects constant array elements

In C, I recently wanted to pass a pointer to an array of chars to a function, and since the function is not allowed to modify the array contents, I declared it as const. The rationale behind this is

char (*param)[20];

defines a pointer param to an array of non-constant chars, and

const char (*param)[20];

defines a pointer param to an array of constant chars. So, the straight forward definition of the function looked like this, and I also verified that the compiler properly catches attempts to write to the array elements:

void f(const char (*param)[20]) {
   /* (*param)[2] = 'A'; */ /* correctly catched by the compiler: assignment of read-only location '(*param)[2]' */

Then, I wanted to call the function as follows:

int main() {
   char (*data)[20] = calloc(20, 1);


   return 0;

But, I got the following warning when compiling this code:

$  gcc -Wall -pedantic -o so so.c
so.c: In function 'main':
so.c:15:4: warning: passing argument 1 of 'f' from incompatible pointer type [enabled by default]
so.c:3:6: note: expected 'const char (*)[20]' but argument is of type 'char (*)[20]'
 void f(const char (*param)[20]) {

Why does this happen? It should be possible to pass a pointer to any non-const data to a function which expects a pointer to const data – this is for example the way how the standard string functions are declared, like strcpy(). The source parameter is a const char*, but we can pass both a pointer to const or to non-const data. As long as the called function does not modify the contents the pointer points to, this should work well. While trying to understand why the above does not work, it turned out that “this is how C works”, and the GCC FAQ explains it: T (*)[n] is not assignment compatible to const T (*)[n] (note though that T* would be assignment compatible to const T* as described above). The C FAQ (if the site is down, use something like https://archive.today/mGvX7 instead) boils this down to the fact that it is not allowed to assign a char** pointer to a const char** pointer. It is only allowed for simple pointers to data (as in the strcpy() example above), but not for pointers to pointers (at any level, recursively). The FAQ also lists the steps which show how the unmodifiable data could be modified without using an explicit cast if this was allowed. Hence the reason that the compiler correctly prints this warning.


The simplest solution is to remove the const from the function’s parameter list. However, this would not allow the compiler to catch write access to the array within the function anymore. If we want to keep the const, we need to cast when calling the function (but explicit casts especially between non-const and const usually indicates other deeper issues, so this should be avoided). A much better solution is to wrap the array in a structure. With this approach, it is possible to keep the const so that the compiler catches modifications of the array, and the function can be called without warning. The drawback is that accessing the data inside the function requires a bit more effort to reference the array structure member:

#include <stdlib.h>

typedef struct _arr11 { char data[11]; } arr11;

void f(const arr11* param) {
   /* error: assignment of read-only location 'param->data[3]' */
   /* param->data[3] = 'A';*/

int main() {
   arr11* data = calloc(sizeof(arr11), 1);

   f(data);    /* No warning! */

   return 0;