DTrace Like

awesome-dtrace

Windows/macOS

1
2
3
4
5
sudo dtrace -l | wc -l
sudo dtrace -l > dtrace.txt
sudo dtrace -ln syscall:::
sudo dtrace -ln syscall::: | grep entry | wc -l
sudo dtrace -ln syscall::: | grep entry| grep \# | wc -l

macOS kdebug

https://github.com/apple-oss-distributions/xnu/blob/main/bsd/kern/trace_codes
cat /usr/share/misc/trace.codes | grep BSC_
557个 多出来的5个
0x40e0104 BSC_msync_extended_info
0x40e0264 BSC_pread_extended_info
0x40e0268 BSC_pwrite_extended_info
0x40e0314 BSC_mmap_extended_info
0x40f0314 BSC_mmap_extended_info2

win11 21H2 22000.856 系统调用 474个 Nt开头
macOS 10.15 系统调用 552个 无名称 #开头 121个
[Darwin Kernel Version 19.6.0: Tue Jun 21 21:18:39 PDT 2022; root:xnu-6153.141.66~1/RELEASE_X86_64 x86_64]

Linux

1
2
3
4
5
6
sudo bpftrace -l > bpftrace.txt
sudo bpftrace -l 't:syscalls:'
sudo bpftrace -l 't:syscalls:sys_enter_*' | wc -l
sudo bpftrace -l 't:syscalls:sys_enter_*' > syscalls.txt
sed -i 's/tracepoint:syscalls:sys_enter_//' syscalls.txt
sudo bpftrace -l 't:syscalls:sys_enter_*' | sed  's/tracepoint:syscalls:sys_enter_//' > syscalls.txt

manjaro 系统调用 344个
[Linux wurui 5.15.60-1-MANJARO #1 SMP PREEMPT Thu Aug 11 13:14:05 UTC 2022 x86_64 GNU/Linux]
sudo ls /sys/kernel/debug/tracing/events/syscalls/ > syscalls_kernel.txt

Trace

用户自定义事件收集
https://www.cnblogs.com/zkweb/p/8126303.html
etw lttng ctf(DTrace)
Qt 6
corelib/tracing

c 语言使用lttng
http://t.zoukankan.com/rongfengliang-p-12905940.html
tracepoint/do_tracepoint/tracepoint_enabled [#include <lttng/tracepoint-event.h>]
LTTng原理
https://zhuanlan.zhihu.com/p/557393806

etw
TraceLoggingWrite/TraceLoggingProviderEnabled
https://learn.microsoft.com/zh-cn/windows/win32/tracelogging/tracelogging-wrapper-macros
有四种主要类型的提供程序：MOF (经典) 提供程序、WPP 提供程序、基于清单的提供程序和 TraceLogging 提供程序。
https://learn.microsoft.com/zh-cn/windows/win32/etw/about-event-tracing
EventPipe 是类似于 ETW 或 LTTng 的运行时组件，可用于收集跟踪数据。 EventPipe 的目标是使 .NET 开发人员能够轻松地跟踪其 .NET 应用程序，而无需依赖于平台特定的 OS 本机组件（如 ETW 或 LTTng）。
https://learn.microsoft.com/zh-cn/dotnet/core/diagnostics/eventpipe

dtrace
自定义的探测器USDT
https://blog.csdn.net/dustzhu/article/details/103686556
dtrace linux_将DTrace探针添加到您的应用程序
https://blog.csdn.net/cusi77914/article/details/107113729

一文教你如何使用 eBPF 检测分析用户态程序
https://zhuanlan.zhihu.com/p/580656180
https://www.cnxct.com/ebpf-uprobe-userspace-app/

浅谈动态跟踪技术之DTrace
https://www.jianshu.com/p/6acd36976fba

1
2
#define DTRACE_LABEL(p, n)                                                                      \
 "__dtrace_probe$" DTRACE_TOSTRING(%=__LINE__) DTRACE_STRINGIFY(_##p##___##n) ":" "\n\t"

Apple_OSS/xnu/bsd/sys/sdt.h

  • The DTRACE_PROBE*() macros are not supported on Mac OS X – instead see
  • “BUILDING CODE CONTAINING USDT PROBES” in the dtrace(1) manpage

drace_note.md

OpenDTrace
https://github.com/orgs/opendtrace/repositories

OpenDTrace Specification version 1.0
https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-924.pdf

Dtrace4Linux
https://github.com/dtrace4linux/linux

DTrace Tools
https://www.brendangregg.com/dtrace.html

https://www.brendangregg.com/books.html

Linux Tracer

ftrace

sudo pacman -S trace-cmd
GUI –> KernelShark

strace/ltrace(库函数)

babeltrace2(LTTng)

ktap(lua)

bpftrace

https://github.com/iovisor/bpftrace
https://developer.aliyun.com/article/703028
https://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html
https://www.brendangregg.com/blog/2016-10-27/dtrace-for-linux-2016.html

1
2
3
4
5
6
7
8
9
10
uname -r
pacman -Ss headers | grep linux
sudo pacman -S linux515-headers
sudo pacman -S bcc-tools python-bcc
export PATH=/usr/share/bcc/tools/:$PATH
sudo execsnoop
sudo opensnoop

bpftrace -e 'uprobe:/bin/bash:readline { printf("read a line\n"); }'
bpftrace -e 'uprobe:./a.out:sigHandler { printf("sigHandler called\n"); }'

sudo pacman -S bpftrace

sudo
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
bpftrace -l 't:syscalls:*'
bpftrace -l > bpftrace.txt
bpftrace -l 't:syscalls:*'
bpftrace -l 't:syscalls:sys_enter_*' | wc -l
bpftrace -l 't:syscalls:sys_enter_*' > syscalls.txt
sed -i 's/tracepoint:syscalls:sys_enter_//' syscalls.txt
bpftrace -l 't:syscalls:sys_enter_*' | sed  's/tracepoint:syscalls:sys_enter_//' > syscalls.txt
# bpftrace -l 'kprobe:*'
# bpftrace -l 't:syscalls:*openat*
# bpftrace -l 'kprobe:tcp*,trace
# bpftrace -l 'k:*socket*,tracepoint:syscalls:*tcp*'
bpftrace -e 'tracepoint:syscalls:sys_enter_connect { printf("%s is connect.\n", comm); }'
bpftrace -e 'tracepoint:syscalls:sys_enter_connect { printf("%s is connect %s.\n", comm, ntop(((struct sockaddr_in *)args->uservaddr)->sin_addr.s_addr)); }'
ping baidu.com -c1
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'
bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s %s\n", comm, str(args->filename)); }'
bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s called %s\n", comm, str(args->filename)); }'
bpftrace -e 'tracepoint:syscalls:sys_enter_execve { join(args->argv); }'

echo 'int main(){return 0;}' | gcc -xc -o bin -
bpftrace -e 'uprobe:./bin:main { printf("main called\n"); }'
bpftrace -e 'uprobe:./bin:main { printf("main called\n"); }' -c './bin'
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @syscalls = count(); }
    interval:s:1 { print(@syscalls); clear(@syscalls); }'

bpftrace -lv 'tracepoint:syscalls:sys_enter_connect'

bpftrace -e 'iter:task { printf("%s:%d\n", ctx->task->comm, ctx->task->pid); }'
bpftrace -e 'iter:task_file { printf("%s:%d %d:%s\n", ctx->task->comm, ctx->task->pid, ctx->fd, path(ctx->file->f_path)); }'

bpftrace -e 'kprobe:do_nanosleep { printf("%s\n", ksym(reg("ip"))); }'

echo 'int main(){return 0;}' | gcc -g -xc -o bin -
# bpftrace -e 'uprobe:./bin:main { printf("%s\n", usym(reg("ip"))); }' -c './bin'
bpftrace -e 'tracepoint:syscalls:sys_enter_sendmsg { printf("%s => ", comm);
    cat("/proc/%d/cmdline", pid); printf("\n") }'
bpftrace -e 'kprobe:vfs_read { @[comm] = count(); }'

bpftrace -l 'tracepoint:syscalls:sys_enter_*'
bpftrace -e 'tracepoint:sched:sched* { @[probe] = count(); } interval:s:5 { exit(); }'
bpftrace -e 'tracepoint:sched:sched_switch { @[kstack] = count(); } interval:s:2 { exit(); }'

perf
sudo pacman -S perf

eBPF
https://ebpf.io/
https://www.brendangregg.com/ebpf.html
https://github.com/zoidbergwill/awesome-ebpf

https://jishuin.proginn.com/p/763bfbd73692 eBPF 介绍
在很多情况下，并不直接使用 eBPF，而是通过 Cilium、bcc 或 bpftrace 等项目间接使用，它们在 eBPF 之上提供了一层抽象，无需直接编写程序而是提供了一些能力，由 eBPF 来实现。
要是没有上层抽象的话，就要直接编写程序了。Linux 内核期望 ePBF 程序以字节码的形式加载。直接编写字节码不太可能，实际开发中更常见的是使用 LLVM 等编译器套件将伪 C 代码编译成 eBPF 字节码。
Cilium -> 容器

https://developer.aliyun.com/article/947869 深入浅出eBPF｜你要了解的7个核心问题
sudo ls /sys/kernel/debug/tracing/events

https://jishuin.proginn.com/p/763bfbd6368e eBPF 基本架构及使用
总体上XDP性能全面弱于DPDK（但是差距不大）
XDP优势：可编程、内核协同工作[eBPF在内核]

https://zhuanlan.zhihu.com/p/453005342 XDP技术——Linux网络处理的高速公路

https://www.ithome.com/0/620/449.htm XDP for Windows
https://github.com/microsoft/xdp-for-windows/
https://www.zhihu.com/question/534498790

https://lwn.net/Articles/857215/ eBPF for Windows
https://github.com/Microsoft/ebpf-for-windows

https://github.com/iovisor/ubpf Userspace eBPF VM

CPU Flame Graphs
https://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html
Linux perf
eBPF profile
DTrace
Xperf [https://docs.microsoft.com/zh-cn/windows-hardware/test/wpt/xperf-command-line-reference]

Kernel Debug
Linux –> ftrace(tracepoint/kprobe/uprobe,etc)
ftrace 的主要作者 Steven Rostedt 创建了 trace-cmd，我创建了 perf-tools 集合。 我最大的抱怨是它不可编程，所以你不能，例如，保存和获取时间戳、计算延迟，然后将其存储为直方图。 您需要以一定成本将事件转储到用户级别和后处理。 它可以通过 eBPF 变得可编程。
macOS –> kdebug(ktrace)/dtrace?
Windows –> windbg(kernel-mode)/dtrace-on-windows?

kdebug 单用户 一次只能有一个运行 第二个调用sysctl会返回失败
$(OBJROOT)/syscall.codes: $(TARGET)/bsd.syscalls.master
$(OBJROOT)/trace.codes: $(SRCROOT)/bsd/kern/trace_codes $(OBJROOT)/syscall.codes
https://github.com/apple-oss-distributions/xnu/blob/main/bsd/kern/syscalls.master
https://github.com/apple-oss-distributions/xnu/blob/main/bsd/kern/trace_codes

sysctl CTL_KERN, KERN_KDEBUG, KERN_KDREADTR
http://newosxbook.com/tools/kdv.html
Apple’s own utilities - fs_usage(1), sc_usage(1), latency(1) and trace(1) - all part of the system_cmds package, use this facility. The Apple utilities impose filters on KDebug, and don’t tap the full power of this amazing facility. The fabulous Instruments.app also uses kdebug, which it also invokes on the device with the help of iprofiler(1), present both in OS X and in iOS (in the DeveloperDiskImage)
https://github.com/apple-oss-distributions/system_cmds.git
fs_usage目前使用ktrace(ktrace_event_t) see man ktrace
https://github.com/Torrekie/apple_internal_sdk

ktrace info
tailspin info

sysctl ktrace
sysctl kperf
sysctl kern.coredump
sysctl kern.corefile

coredump太大 –> corpses
task_map_corpse_info

kperf subset –> kpc

Trace & Debug
proc_info
stackshot
microstackshot
–> systemstats
/var/db/systemstats
logd
/var/db/diagnostics/

DTrace
dtrace -l | tr -s ‘ ‘ | tr -d ‘[0-9]’ | cut -d ‘ ‘ -f2 | sort -u > dtrace_providers.txt
https://awesome-dtrace.com/
https://www.codenong.com/35752326/ 关于macos：Dtrace打印父进程命令
https://www.cnblogs.com/daizhj/articles/3178257.html MAC OS 内核跟踪监视工具dtrace 使用示例说明

1
2
3
4
5
dtrace -n 'syscall::connect:entry { printf("%s is connect.\n", curpsinfo->pr_psargs); }'
curl -I baidu.com
dtrace -n 'syscall::execve:entry { printf("%s is execve.\n", curpsinfo->pr_psargs); }'
dtrace -n 'syscall::execve:entry { printf("%s %s", execname, copyinstr(arg0)); }'
dtrace -n 'syscall::open*:entry { printf("%s %s", execname, copyinstr(arg0)); }'

https://www.cnblogs.com/zengkefu/p/5632802.html Choosing a Linux Tracer–>Brendan Gregg’s Blog
https://www.cnblogs.com/zengkefu/p/5726530.html 利用DTrace实时检测MySQL
DTrace最好的一个功能是一个叫做“flowindent”的编译指示选项。这个函数会相应地显示entry和return并生成一个逻辑树，以便用户能了解其内部的函数调用关系。
http://myaut.github.io/dtrace-stap-book/ Dynamic Tracing with DTrace & SystemTap
https://www.cnblogs.com/zengkefu/p/6550976.html windows XP 下的DTRACE 跟踪 学习
https://github.com/prash-wghats/DTrace-win32

https://illumos.org/books/dtrace/preface.html Dynamic Tracing Guide
https://github.com/rmustacc/illumos-docbooks

TODO
JIT eBPF/cling mprotect确认
instruments
iprofiler (Older versions of MacOS (Sierra))
https://github.com/jfgiraldoa/moose/blob/master/modules/doc/content/application_development/profiling.md

https://www.cnblogs.com/zengkefu/p/6882921.html 白话火焰图—火丁笔记
https://www.cnblogs.com/zengkefu/p/7502433.html DTrace memory leak 内存泄露

dtrace -x ustackframes=100 -n ‘profile-97 /pid == 12345/ { @[ustack()] = count(); } tick-60s { exit(0); }’ -o out.user_stacks

dtrace -l -n profile-*
dtrace -x ustackframes=100 -n ‘profile-997 /execname == “test”/ { @[ustack()] = count(); }’ -o user_stacks.txt -c ‘./test’

gprof2dot -f dtrace user_stacks.txt | dot -Tpng -o output.png

Instruments
instruments -t “Time Profiler” ./a.out foo.txt 20
instruments -t “Time Profiler” ./test
instruments -l 30000 -t “Time Profiler” -p 5773

MTuner is a C/C++ memory profiler and memory leak finder for Windows, PlayStation 4 and 3, Android and other platforms

分析系统自带CPU采样格式
macOS sample/活动监视器
Windows dump/任务管理器
Linux perf?

Userspace eBPF VM
ubpf
ebpf-for-windows
DTrace-book-scripts
DTrace-tools
awesome-ebpf

dtrace-stap-book
toolkit
FlameGraph
frida