⚲ API

linux/err.h _inc – helper macros for error pointer handling and propagation

linux/errno.h _inc – standard error codes used throughout the kernel.

Performance

There are many factors that can affect the performance of the Linux kernel, including hardware configurations, software configurations, and workload characteristics.

In this context, performance optimization of the Linux kernel involves identifying and addressing performance bottlenecks in the system. This can involve tuning kernel parameters, optimizing system resources, and identifying and fixing bugs and other issues that may be impacting performance.

Given the complexity of the Linux kernel and the wide range of factors that can affect performance, performance optimization can be a challenging task. However, with the right tools and techniques, it is possible to significantly improve the performance and reliability of Linux-based systems.

User space debug interfaces

⚲ Interfaces

man 1 dmesg – prints or control the kernel ring buffer

man 2 syslog – system call, which is used to control the kernel printk() buffer

man 1 strace – system calls and signals tracing tool

man 2 ptrace – process trace system call

man 3 klogctl

man 5 core

/sys/kernel/debug/ – debugfs

dmesg --console-level <level>

gdb /usr/src/linux/vmlinux /proc/kcore

/proc/self/stack

dynamic _doc debug

⌨️ hands-on:

echo "module atkbd +pfl" | sudo tee /sys/kernel/debug/dynamic_debug/control

⚙️ Internals

handle_sysrq _id

📚 References

Development tools for the kernel _doc

DebugFS _doc, samples/qmi/qmi_sample_client.c _src

Kprobe-based Event Tracing _doc

Dynamic debug _doc

Linux Magic System Request Key Hacks _doc

Magic SysRq key

Tracing and logging

⚲ API:

User-space interface:

man 1 dmesg – prints or control the kernel ring buffer

man 2 syslog – system call, which is used to control the kernel printk() buffer

/proc/kmsg

https://kernelshark.org/ – front end reader of trace-cmd

https://trace-cmd.org/, man 1 trace-cmd – CLI for Ftrace _doc – Linux kernel internal tracer /sys/kernel/debug/tracing/

The most commonly used functions

linux/printk.h _inc

dump_stack _id – prints the current kernel stack trace for debugging purposes

pr_alert _id – logs an alert-level message, indicating a critical event that requires immediate attention

pr_cont _id – continues printing the current message on the same line

pr_crit _id – logs a critical-level message, indicating a severe condition that might require system halt

pr_debug _id – logs a debug-level message for developers, usually enabled in debug builds

pr_devel _id – logs a developer-specific message, typically used for fine-grained debug purposes, see dynamic _doc

pr_emerg _id – logs an emergency-level message, indicating a serious error that could cause system crash

pr_err _id – logs an error-level message, typically indicating an issue that requires attention

pr_err_ratelimited _id – logs an error-level message with rate limiting to prevent excessive logging

pr_fmt _id – defines a format string for kernel messages

pr_info _id – logs an informational-level message, providing status updates or diagnostics

pr_info_ratelimited _id – logs an informational-level message with rate limiting

pr_notice _id – logs a notice-level message, typically used for events that aren't errors but should be noted

pr_warn _id – logs a warning-level message, indicating a potential issue that doesn't immediately affect system functionality

pr_warn_once _id – logs a warning message once, preventing repeated warnings for the same event

pr_warn_ratelimited _id – logs a warning-level message with rate limiting

print_hex_dump _id – prints a hexdump of data for debugging purposes

print_hex_dump_debug _id – prints a detailed hexdump with debugging-level verbosity

printk _id – the primary function for printing kernel messages with varying severity levels

va_format _id – formats a variable argument list into a string for logging or printing purposes

⌨️ hands-on:

echo "module atkbd +pfl" | sudo tee /sys/kernel/debug/dynamic_debug/control; dmesg -w

and type on build-in keyboard

include/linux/dev_printk.h _inc – device-specific logging

dev_crit _id – prints a critical-level message for a device

dev_dbg _id – prints a debug-level message for a device if debugging is enabled

dev_dbg_ratelimited _id – prints debug messages for a device with rate limiting

dev_err _id – prints an error-level message for a device

dev_err_once _id – prints an error message for a device only once

dev_err_probe _id – prints an error related to probe failure with standard formatting

dev_err_ratelimited _id – prints error messages for a device with rate limiting

dev_fmt _id – defines a format string used by device-specific printk macros

dev_info _id – prints an informational-level message for a device

dev_notice _id – prints a notice-level message for a device

dev_printk _id – generic function to print kernel messages with specified log level for a device

dev_vdbg _id – prints verbose debug messages for a device if enabled at compile time

dev_warn _id – prints a warning-level message for a device

dev_warn_once _id – prints a warning message for a device only once

dev_warn_ratelimited _id – prints warning messages for a device with rate limiting

asm-generic/bug.h _inc

WARN_ON _id

WARN _id

⚙️ Internals

printk _id

kernel/printk/printk.c _src

arch/x86/kernel/traps.c _src

lib/dump_stack.c _src

kernel/trace _src

scripts/tracing/draw_functrace.py _src

logging _ltp, tracing _ltp

samples/ftrace _src

samples/trace_events _src

samples/trace_printk _src

linux/instrumentation.h _inc

📚 References:

Debugging by printing

Message logging with printk _doc

Dynamic debug _doc

SystemTap

man 1 stap – systemtap script translator/driver

strace

man 1 strace – trace system calls and signals

LTTng

ftrace

Linux Tracing Technologies _doc

Tracepoint Analysis _doc

Function Tracer _doc – function, latency and event tracing

Event Tracing _doc

Using ftrace to hook to functions _doc

Fprobe - Function entry/exit probe _doc

Kprobes _doc

Kprobe-based Event Tracing _doc

Uprobe-tracer: Uprobe-based Event Tracing _doc

Using the Linux Kernel Tracepoints _doc

Subsystem Trace Points: kmem _doc

Subsystem Trace Points: power _doc

NMI Trace Events _doc

In-kernel memory-mapped I/O tracing _doc

Event Histograms _doc

Histogram Design Notes _doc

Boot-time tracing _doc

Hardware Latency Detector _doc

Intel(R) Trace Hub (TH) _doc

Lockless Ring Buffer Design _doc

System Trace Module _doc

CoreSight - ARM Hardware Trace _doc

🔧 TODO. 🚀 advanced features

linux/kmemleak.h _inc – memory leak detector

pr_cont _id- continues a previous log message in the same line

print_hex_dump_bytes _id

print_hex_dump_debug _id

dump_stack _id

CONFIG_PRINTK_CALLER _id

CONFIG_DEBUG_KERNEL _id

CONFIG_DEBUG_INFO _id

https://git.kernel.org/pub/scm/libs/libtrace/

kgdb and kdb

⚲ Interfaces

linux/kgdb.h _inc

linux/kdb.h _inc

⚙️ Internals

kernel/debug _src

📚 References

Using kgdb, kdb and the kernel debugger internals _doc

kdump

kdump _doc

man 8 crash – Analyze Linux crash dump data or a live system

eBPF

⚲ API:

man 2 bpf ↪ kernel/bpf/syscall.c _src

📖 References

eBPF and BPF _doc

📚 Further reading

man 7 bpf-helpers

Linux Extended BPF (eBPF) Tracing Tools

bpftrace – High-level tracing language for Linux eBPF

BCC – Tools for BPF-based Linux IO analysis, networking, monitoring, and more

Example of trace.py

man 8 stapbpf

eBPF Programming for Linux Kernel Tracing

lockdep - Runtime locking correctness validator _doc

Watchdogs

The Linux Kernel/Softdog Driver

dev_watchdog _id – network device watchdog

The NMI watchdog lockup detectors:

⚲ API

/proc/sys/kernel/nmi_watchdog

/proc/sys/kernel/soft_watchdog

/proc/sys/kernel/watchdog

/proc/sys/kernel/watchdog_cpumask

/proc/sys/kernel/watchdog_thresh

/proc/sys/kernel/hardlockup_all_cpu_backtrace

/proc/sys/kernel/hardlockup_panic

/proc/sys/kernel/softlockup_all_cpu_backtrace

/proc/sys/kernel/softlockup_panic

linux/nmi.h _inc

👁️ Example

./lib/test_lockup.c _src – test module to generate lockups

Provoke NMI watchdog without panic:

echo 0 > /proc/sys/kernel/hardlockup_panic
insmod test_lockup.ko disable_irq=1 time_secs=13

⚙️ Internals

kernel/watchdog.c _src – detects hard and soft lockups on a system

kernel/watchdog_perf.c _src – detects hard lockups on a system using perf

kernel/watchdog_buddy.c _src

📚 References

Documentation for /proc/sys/kernel/ _doc

Softlockup detector and hardlockup detector (aka nmi_watchdog) _doc

kernel parameters:

nmi_watchdog _param

nowatchdog _param

nosoftlockup _param

softlockup_panic _param

...

⚙️ Internals

arch/x86/kernel/traps.c _src

📖 References for debugging

Ramoops oops/panic logger _doc

pstore block oops/panic logger _doc

Fault injection _doc

Bisecting a bug _doc

Development tools for the kernel _doc

Kernel Testing Guide _doc

Checkpatch _doc, scripts/checkpatch.pl _src

Selftests _doc, tools/testing/selftests _src

linux/tracepoint.h _inc

📚 Further reading

https://deepwiki.com/torvalds/linux/2.4-kernel-tracing-and-profiling

https://drgn.readthedocs.io/ – programmable debugger

https://crash-utility.github.io/

https://wiki.ubuntu.com/Kernel/Debugging

Intel VTune Profiler

Linux Applications Debugging Techniques