Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
System Analysis and Tuning Guide › Kernel monitoring › Kernel probes
ContentsContents
System Analysis and Tuning Guide
  1. Preface
  2. I Basics
    1. 1 General notes on system tuning
  3. II System monitoring
    1. 2 System monitoring utilities
    2. 3 System log files
  4. III Kernel monitoring
    1. 4 SystemTap—filtering and analyzing system data
    2. 5 Kernel probes
    3. 6 Hardware-based performance monitoring with Perf
    4. 7 OProfile—system-wide profiler
    5. 8 Dynamic debug—kernel debugging messages
  5. IV Resource management
    1. 9 General system resource management
    2. 10 Kernel control groups
    3. 11 Automatic Non-Uniform Memory Access (NUMA) balancing
    4. 12 Power management
  6. V Kernel tuning
    1. 13 Tuning I/O performance
    2. 14 Tuning the task scheduler
    3. 15 Tuning the memory management subsystem
    4. 16 Tuning the network
  7. VI Handling system dumps
    1. 17 Tracing tools
    2. 18 Kexec and Kdump
    3. 19 Using systemd-coredump to debug application crashes
  8. VII Synchronized clocks with Precision Time Protocol
    1. 20 Precision Time Protocol
  9. A GNU licenses
Navigation←→
System Analysis and Tuning Guide › Kernel monitoring › Kernel probes
Top
←→
Applies to openSUSE Leap 15.6

5 Kernel probes Edit source

5.1 Supported architectures
5.2 Types of kernel probes
5.3 Kprobes API
5.4 debugfs Interface
5.5 More information

Kernel probes are a set of tools to collect Linux kernel debugging and performance information. Developers and system administrators use them either to debug the kernel, or to find system performance bottlenecks. The reported data can then be used to tune the system for better performance.

You can insert these probes into any kernel routine, and specify a handler to be invoked after a particular break-point is hit. The main advantage of kernel probes is that you no longer need to rebuild the kernel and reboot the system after you make changes in a probe.

To use kernel probes, you typically need to write or obtain a specific kernel module. Such modules include both the init and the exit function. The init function (such as register_kprobe()) registers one or more probes, while the exit function unregisters them. The registration function defines where the probe is inserted and which handler is called after the probe is hit. To register or unregister a group of probes at one time, you can use relevant register_<PROBE_TYPE>probes() or unregister_<PROBE_TYPE>probes() functions.

Debugging and status messages are typically reported with the printk kernel routine. printk is a kernel space equivalent of a user space printf routine. For more information on printk, see Logging kernel messages. Normally, you can view these messages by inspecting the output of the systemd journal (see Book “Reference”, Chapter 11 “journalctl: query the systemd journal”). For more information on log files, see Chapter 3, System log files.

5.1 Supported architectures Edit source

Kernel probes are fully implemented on the following architectures:

  • x86

  • AMD64/Intel 64

  • Arm

  • POWER

Kernel probes are partially implemented on the following architectures:

  • IA64 (does not support probes on instruction slot1)

  • sparc64 (return probes not yet implemented)

5.2 Types of kernel probes Edit source

There are three types of kernel probes: Kprobes, Jprobes, and Kretprobes. Kretprobes are sometimes called return probes. You can find source code examples of all three type of probes in the Linux kernel. See the directory /usr/src/linux/samples/kprobes/ (package kernel-source).

5.2.1 Kprobes Edit source

Kprobes can be attached to any instruction in the Linux kernel. When Kprobes is registered, it inserts a break-point at the first byte of the probed instruction. When the processor hits this break-point, the processor registers are saved, and the processing passes to Kprobes. First, a pre-handler is executed, then the probed instruction is stepped, and, finally a post-handler is executed. The control is then passed to the instruction following the probe point.

5.2.2 Jprobes Edit source

Jprobes is implemented through the Kprobes mechanism. It is inserted on a function's entry point and allows direct access to the arguments of the function which is being probed. Its handler routine must have the same argument list and return value as the probed function. To end it, call the jprobe_return() function.

When a jprobe is hit, the processor registers are saved, and the instruction pointer is directed to the jprobe handler routine. The control then passes to the handler with the same register contents as the function being probed. Finally, the handler calls the jprobe_return() function, and switches the control back to the control function.

Generally, you can insert multiple probes on one function. Jprobe is, however, limited to only one instance per function.

5.2.3 Return probe Edit source

Return probes are also implemented through Kprobes. When the register_kretprobe() function is called, a kprobe is attached to the entry of the probed function. After hitting the probe, the kernel probes mechanism saves the probed function return address and calls a user-defined return handler. The control is then passed back to the probed function.

Before you call register_kretprobe(), you need to set a maxactive argument, which specifies how many instances of the function can be probed at the same time. If set too low, a certain number of probes is missed.

5.3 Kprobes API Edit source

The programming interface of Kprobes consists of functions which are used to register and unregister all used kernel probes, and associated probe handlers. For a more detailed description of these functions and their arguments, see the information sources in Section 5.5, “More information”.

register_kprobe()

Inserts a break-point on a specified address. When the break-point is hit, the pre_handler and post_handler are called.

register_jprobe()

Inserts a break-point in the specified address. The address needs to be the address of the first instruction of the probed function. When the break-point is hit, the specified handler is run. The handler should have the same argument list and return type as the probed.

register_kretprobe()

Inserts a return probe for the specified function. When the probed function returns, a specified handler is run. This function returns 0 on success, or a negative error number on failure.

unregister_kprobe(), unregister_jprobe(), unregister_kretprobe()

Removes the specified probe. You can use it any time after the probe has been registered.

register_kprobes(), register_jprobes(), register_kretprobes()

Inserts each of the probes in the specified array.

unregister_kprobes(), unregister_jprobes(), unregister_kretprobes()

Removes each of the probes in the specified array.

disable_kprobe(), disable_jprobe(), disable_kretprobe()

Disables the specified probe temporarily.

enable_kprobe(), enable_jprobe(), enable_kretprobe()

Temporarily enables disabled probes.

5.4 debugfs Interface Edit source

In recent Linux kernels, the Kprobes instrumentation uses the kernel's debugfs interface. It can list all registered probes and globally switch all probes on or off.

5.4.1 Listing registered kernel probes Edit source

The list of all currently registered probes is in the /sys/kernel/debug/kprobes/list file. 

saturn.example.com:~ # cat /sys/kernel/debug/kprobes/list
c015d71a  k  vfs_read+0x0   [DISABLED]
c011a316  j  do_fork+0x0
c03dedc5  r  tcp_v4_rcv+0x0

The first column lists the address in the kernel where the probe is inserted. The second column prints the type of the probe: k for kprobe, j for jprobe, and r for return probe. The third column specifies the symbol, offset and optional module name of the probe. The following optional columns include the status information of the probe. If the probe is inserted on a virtual address which is not valid anymore, it is marked with [GONE]. If the probe is temporarily disabled, it is marked with [DISABLED].

5.4.2 Globally enabling/disabling kernel probes Edit source

The /sys/kernel/debug/kprobes/enabled file represents a switch with which you can globally and forcibly turn on or off all the registered kernel probes. To turn them off, simply enter 

# echo "0" > /sys/kernel/debug/kprobes/enabled

on the command line as root. To turn them on again, enter 

# echo "1" > /sys/kernel/debug/kprobes/enabled

With such operations, you do not change the status of the probes. If a probe is temporarily disabled, it is not enabled automatically but remains in the [DISABLED] state after entering the latter command.

5.5 More information Edit source

To learn more about kernel probes, look at the following sources of information:

  • Thorough but more technically oriented information about kernel probes is in /usr/src/linux/Documentation/trace/kprobes.txt (package kernel-source).

  • Examples of all three types of probes (together with related Makefile) are in the /usr/src/linux/samples/kprobes/ directory (package kernel-source).

  • In-depth information about Linux kernel modules and printk kernel routine can be found at The Linux Kernel Module Programming Guide

→Chapter 6 Hardware-based performance monitoring with Perf←Chapter 4 SystemTap—filtering and analyzing system data
Print this page

© 2024 SUSE