Performance

There are many factors that can affect the performance of the Linux kernel, including hardware configurations, software configurations, and workload characteristics.

In this context, performance optimization of the Linux kernel involves identifying and addressing performance bottlenecks in the system. This can involve tuning kernel parameters, optimizing system resources, and identifying and fixing bugs and other issues that may be impacting performance.

Given the complexity of the Linux kernel and the wide range of factors that can affect performance, performance optimization can be a challenging task. However, with the right tools and techniques, it is possible to significantly improve the performance and reliability of Linux-based systems.

User space debug interfaces

⚲ Interfaces

man 1 dmesg – prints or control the kernel ring buffer

man 2 syslog – system call, which is used to control the kernel printk() buffer

man 1 strace – system calls and signals tracing tool

man 2 ptrace – process trace system call

man 3 klogctl

man 5 core

/sys/kernel/debug/ – debugfs

dmesg --console-level <level>

gdb /usr/src/linux/vmlinux /proc/kcore

/proc/self/stack

dynamic _doc debug

⌨️ hands-on:

echo "module atkbd +pfl" | sudo tee /sys/kernel/debug/dynamic_debug/control

⚙️ Internals

handle_sysrq _id

📚 References

Development tools for the kernel _doc

DebugFS _doc, samples/qmi/qmi_sample_client.c _src

Kprobe-based Event Tracing _doc

Dynamic debug _doc

Linux Magic System Request Key Hacks _doc

Magic SysRq key

Tracing and logging

⚲ API:

User-space interface:

man 1 dmesg – prints or control the kernel ring buffer

man 2 syslog – system call, which is used to control the kernel printk() buffer

/proc/kmsg

https://kernelshark.org/ – front end reader of trace-cmd

https://trace-cmd.org/, man 1 trace-cmd – CLI for Ftrace _doc – Linux kernel internal tracer /sys/kernel/debug/tracing/

Most common functions

linux/printk.h _inc

pr_devel _id- conditional debug-level message

pr_debug _id- conditional debug-level or dynamic _doc message

⌨️ hands-on:

echo "module atkbd +pfl" | sudo tee /sys/kernel/debug/dynamic_debug/control

Log messages with other levels:

pr_info _id, pr_notice _id, pr_warn _id, pr_err _id, pr_crit _id, pr_alert _id, pr_emerg _id

asm-generic/bug.h _inc

WARN_ON _id

WARN _id

⚙️ Internals

printk _id

kernel/printk/printk.c _src

arch/x86/kernel/traps.c _src

lib/dump_stack.c _src

kernel/trace _src

scripts/tracing/draw_functrace.py _src

logging _ltp, tracing _ltp

samples/ftrace _src

samples/trace_events _src

samples/trace_printk _src

linux/instrumentation.h _inc

📚 References:

Debugging by printing

Message logging with printk _doc

SystemTap

man 1 stap – systemtap script translator/driver

strace

man 1 strace – trace system calls and signals

LTTng

ftrace

Linux Tracing Technologies _doc

Tracepoint Analysis _doc

Function Tracer _doc – function, latency and event tracing

Event Tracing _doc

Using ftrace to hook to functions _doc

Fprobe - Function entry/exit probe _doc

Kprobes _doc

Kprobe-based Event Tracing _doc

Uprobe-tracer: Uprobe-based Event Tracing _doc

Using the Linux Kernel Tracepoints _doc

Subsystem Trace Points: kmem _doc

Subsystem Trace Points: power _doc

NMI Trace Events _doc

In-kernel memory-mapped I/O tracing _doc

Event Histograms _doc

Histogram Design Notes _doc

Boot-time tracing _doc

Hardware Latency Detector _doc

Intel(R) Trace Hub (TH) _doc

Lockless Ring Buffer Design _doc

System Trace Module _doc

CoreSight - ARM Hardware Trace _doc

🔧 TODO. 🚀 advanced features

linux/kmemleak.h _inc – memory leak detector

pr_cont _id- continues a previous log message in the same line

print_hex_dump_bytes _id

print_hex_dump_debug _id

dump_stack _id

CONFIG_PRINTK_CALLER _id

CONFIG_DEBUG_KERNEL _id

CONFIG_DEBUG_INFO _id

https://git.kernel.org/pub/scm/libs/libtrace/

kgdb and kdb

⚲ Interfaces

linux/kgdb.h _inc

linux/kdb.h _inc

⚙️ Internals

kernel/debug _src

📚 References

Using kgdb, kdb and the kernel debugger internals _doc

kdump

kdump _doc

man 8 crash – Analyze Linux crash dump data or a live system

eBPF

⚲ API:

man 2 bpf ↪ kernel/bpf/syscall.c _src

📖 References

eBPF and BPF _doc

📚 Further reading

man 7 bpf-helpers

Linux Extended BPF (eBPF) Tracing Tools

bpftrace – High-level tracing language for Linux eBPF

BCC – Tools for BPF-based Linux IO analysis, networking, monitoring, and more

Example of trace.py

man 8 stapbpf

eBPF Programming for Linux Kernel Tracing

lockdep - Runtime locking correctness validator _doc

Watchdogs

The Linux Kernel/Softdog Driver

dev_watchdog _id – network device watchdog

The NMI watchdog lockup detectors:

⚲ API

/proc/sys/kernel/nmi_watchdog

/proc/sys/kernel/soft_watchdog

/proc/sys/kernel/watchdog

/proc/sys/kernel/watchdog_cpumask

/proc/sys/kernel/watchdog_thresh

/proc/sys/kernel/hardlockup_all_cpu_backtrace

/proc/sys/kernel/hardlockup_panic

/proc/sys/kernel/softlockup_all_cpu_backtrace

/proc/sys/kernel/softlockup_panic

linux/nmi.h _inc

👁️ Example

./lib/test_lockup.c _src – test module to generate lockups

Provoke NMI watchdog without panic:

echo 0 > /proc/sys/kernel/hardlockup_panic
insmod test_lockup.ko disable_irq=1 time_secs=13

⚙️ Internals

kernel/watchdog.c _src – detects hard and soft lockups on a system

kernel/watchdog_perf.c _src – detects hard lockups on a system using perf

kernel/watchdog_buddy.c _src

📚 References

Documentation for /proc/sys/kernel/ _doc

Softlockup detector and hardlockup detector (aka nmi_watchdog) _doc

kernel parameters:

nmi_watchdog _param

nowatchdog _param

nosoftlockup _param

softlockup_panic _param

...

⚙️ Internals

arch/x86/kernel/traps.c _src

📖 References for debugging

Ramoops oops/panic logger _doc

pstore block oops/panic logger _doc

Fault injection _doc

Bisecting a bug _doc

Development tools for the kernel _doc

Kernel Testing Guide _doc

Checkpatch _doc, scripts/checkpatch.pl _src

Selftests _doc, tools/testing/selftests _src

linux/tracepoint.h _inc

📚 Further reading

https://drgn.readthedocs.io/ – programmable debugger

https://crash-utility.github.io/

https://wiki.ubuntu.com/Kernel/Debugging

Intel VTune Profiler

Linux Applications Debugging Techniques