diff --git a/source/tools/detect/surftrace/Makefile b/source/tools/detect/surftrace/Makefile new file mode 100644 index 0000000000000000000000000000000000000000..acd7346152803efd15878c2d9bf538e803e594e8 --- /dev/null +++ b/source/tools/detect/surftrace/Makefile @@ -0,0 +1,3 @@ +target := surftrace +DEPEND := "prev{default python};" +include $(SRC)/mk/py.mk diff --git a/source/tools/detect/surftrace/ReadMe.md b/source/tools/detect/surftrace/ReadMe.md new file mode 100644 index 0000000000000000000000000000000000000000..0b8cc8043452ae250867d74575b4268e291b60fd --- /dev/null +++ b/source/tools/detect/surftrace/ReadMe.md @@ -0,0 +1,355 @@ +# 1、产生背景 + +​ 我们可以采用以下手段来trace内核调用，只说缺点： + +## 1.1、kprobe/jprobe/kretprobe + +- 侵入式插入ko，危险系数高 +- 需要编写内核代码，难度系数大 + +## 1.2、systemtap + +- 需要编写stp代码，步骤较多 + +## 1.3、bpf（含bcc和libebpf） + +- 需要高版本内核支持 +- 需要编写两处代码，步骤较多 + +## 1.4、ftrace-kprobe + +- 配置步骤繁琐，从配置到看出效果，至少要经历五个以上的步骤 +- 功能受限，对知识点要求较高 + +## 1.5、perf-tools kprobe + +后来我发现了greg 写的一个kprobe 封装工具：https://github.com/brendangregg/perf-tools/blob/master/kernel/kprobe，它可以把繁杂的ftrace 一个 kprobe event 缩略为一个命令，极大拓展了我对ftrace的了解。然而这个工具使用起来仍有以下困难： + +- 只能追踪一个kprobe点，我往往需要追踪多个kprobe点； +- 深入追踪困难：比如我们要在__netif_receive_skb_core 函数中打出skb参数中ip头里面的protocol成员，对应的表达式是 **proto=+0x9(+0xf0(%di)):s8**，光推导这个表达式的过程或许要耗费我们10分钟左右的时间。而且这个表达式并非固定不变，在不同的内核上还需要重新计算； + +上述两点成为了我使用ftrace的拦路虎，一直想对它改造优化，但受限于自己的蹩脚的bash能力，进展比较慢。于是换了一个思路，改用python。 + +# 2、surftrace 准备工作 + +## 2.1、命名约定 + +在后面的使用中，会用到两类表达式，一种是程序员可以直观通过结构体定义理解的，比如： + +```bash +p __netif_receive_skb_core proto=@(struct iphdr *)l3%0->protocol ip_src=@(struct iphdr *)l3%0->saddr ip_dst=@(struct iphdr *)l3%0->daddr data=@(struct iphdr *)l3%0->sdata[1] f:proto==1&&ip_src==127.0.0.1 +``` + +称为**结构化表达式** + +这类表达式不能被ftrace识别，需要在surftrace中进行转换，转换后的 + +```bash +p __netif_receive_skb_core proto=+0x9(+0xf0(%di)):x8 ip_src=+0xc(+0xf0(%di)):x32 ip_dst=+0x10(+0xf0(%di)):x32 type=+0x14(+0xf0(%di)):x8 seq=+0x1c(+0xf0(%di)):s16 f:common_pid==0&&proto==1&&ip_src==0x100007f +``` + +称为**ftrace表达式** + +# 2.2、依赖条件 + +如果你想使用surftrace的完整功能，至少需要以下条件： + +- 内核支持ftrace、已经mount了debugfs、root权限 +- python2.7或更高，推荐3 以上 +下面的条件三选一即可 +- 1、公开发行版内核，可以访问 pylcc.openanolis.cn +- 2、公开发行版内核，已经从 http://pylcc.openanolis.cn/db/ 下载了 对应内核的db文件 +- 3.1、环境上安装了gdb 版本大于 9，如果是x86 平台，可以直接从 http://pylcc.openanolis.cn/gdb/x64/gdb 下载 +- 3.2、安装了对应内核的 vmlinux （结构化表达式依赖，非必须） + + + +## 2.3、参数说明 + +```bash +usage: surftrace.py [-h] [-v VMLINUX] [-m MODE] [-r RIP] [-f FILE] [-g GDB] + [-F FUNC] [-o OUTPUT] [-l LINE] [-a ARCH] [-s] [-S] + [traces [traces ...]] + +Trace ftrace kprobe events. + +positional arguments: + trace set trace args. + +optional arguments: + -h, --help show this help message and exit + -v VMLINUX, --vmlinux VMLINUX + set vmlinux path. + -m MODE, --mode MODE set arg parser, fro + -r RIP, --rip RIP set remote server ip, remote mode only. + -f FILE, --file FILE set input args path. + -g GDB, --gdb GDB set gdb exe file path. + -F FUNC, --func FUNC disasassemble function. + -o OUTPUT, --output OUTPUT + set output bash file + -l LINE, --line LINE get file disasemble info + -a ARCH, --arch ARCH set architecture. + -s, --stack show call stacks. + -S, --show only show expressions. + +``` + + +-f: 从文件中读取表达式，适合大量配置的场景 + +-o: 将执行过程导出到脚本中 + +-a:指定cpu架构，涉及到寄存器转换，目前只支持x86_64/aarch64，不指定的话，会根据lscpu获取 + +-s:打印probe点调用栈，这是个全局开关 + +-S:只生成ftrace表达式，不下发到ftrace。该模式适合交叉调试场景，比如我们要想在树莓派上去probe 钩子，但是树莓派的资源空间有限，不可能去安装gdb和vmlinux。因此我们可以在宿主机上将结构化表达式转成ftrace表达式。然后在树莓派下发即可。 + +# 3、实战 + +我们以open anolis为例，将surftrace.py取下来 + +``` +sudo sh -c su +chmod +x surftrace.py +``` + +## 3.1、追踪函数入口和返回位置 + +按Ctrl + C 停止 + +```bash +#./surftrace.py 'p _do_fork' 'r _do_fork' +echo 'p:f0 _do_fork ' >> /sys/kernel/debug/tracing/kprobe_events +echo 1 > /sys/kernel/debug/tracing/events/kprobes/f0/enable +echo 'r:f1 _do_fork ' >> /sys/kernel/debug/tracing/kprobe_events +echo 1 > /sys/kernel/debug/tracing/events/kprobes/f1/enable +echo 0 > /sys/kernel/debug/tracing/options/stacktrace +echo 1 > /sys/kernel/debug/tracing/tracing_on + <...>-1637241 [000] d... 7462686.335257: f0: (_do_fork+0x0/0x3a0) + <...>-1637241 [000] d... 7462686.335323: f1: (SyS_clone+0x36/0x40 <- _do_fork) + systemd-1 [004] d... 7462686.854375: f0: (_do_fork+0x0/0x3a0) + systemd-1 [004] d... 7462686.854446: f1: (SyS_clone+0x36/0x40 <- _do_fork) + …… + systemd-1 [004] d... 7462688.104383: f0: (_do_fork+0x0/0x3a0) + systemd-1 [004] d... 7462688.104464: f1: (SyS_clone+0x36/0x40 <- _do_fork) +^Cecho 0 > /sys/kernel/debug/tracing/events/kprobes/f0/enable + <...>-1637241 [000] d... 7462688.134451: f0: (_do_fork+0x0/0x3a0) + <...>-1637241 [000] d... 7462688.135278: f1: (SyS_clone+0x36/0x40 <- _do_fork) +echo 0 > /sys/kernel/debug/tracing/events/kprobes/f1/enable + <...>-1637241 [000] d... 7462688.155188: f1: (SyS_clone+0x36/0x40 <- _do_fork) +echo > /sys/kernel/debug/tracing/kprobe_events +echo 0 > /sys/kernel/debug/tracing/tracing_on +``` + +可以看到 surftrace支持多个probe，所有表达式要用单引号括起来，表达式中，第一段字母p 表示probe函数入口，r表示probe函数返回位置，第二段为函数符号，该符号必须要在 tracing/available_filter_functions 中可以查找到的 + +## 3.2、 获取函数入参 + +还是以_do_fork为例，我们可以查找到它的入参是： + +```c +#ifdef CONFIG_FORK2 +long _do_fork(struct task_struct *parent, + struct task_struct *source, + unsigned long clone_flags, +#else + long _do_fork(unsigned long clone_flags, +#endif + unsigned long stack_start, + unsigned long stack_size, + int __user *parent_tidptr, + int __user *child_tidptr, + unsigned long tls) +``` + +我们可以确认它的第一个入参类型是 struct task_struct，如果要获取任务名，即common，可以采用以下方法： + +```bash +#./surftrace.py 'p _do_fork comm=%0->comm' +echo 'p:f0 _do_fork comm=+0xafc(%di):string ' >> /sys/kernel/debug/tracing/kprobe_events +echo 1 > /sys/kernel/debug/tracing/events/kprobes/f0/enable +echo 0 > /sys/kernel/debug/tracing/options/stacktrace +echo 1 > /sys/kernel/debug/tracing/tracing_on + <...>-1642046 [001] d... 7463503.175187: f0: (_do_fork+0x0/0x3a0) comm="surftrace.py" + systemd-1 [002] d... 7463503.606161: f0: (_do_fork+0x0/0x3a0) comm="systemd" + python-16819 [003] d... 7463504.383400: f0: (_do_fork+0x0/0x3a0) comm="python" + systemd-1 [002] d... 7463504.856166: f0: (_do_fork+0x0/0x3a0) comm="systemd" + <...>-1642087 [002] d... 7463506.031046: f0: (_do_fork+0x0/0x3a0) comm="sh" + <...>-1642087 [002] d... 7463506.031363: f0: (_do_fork+0x0/0x3a0) comm="sh" + systemd-1 [004] d... 7463506.106159: f0: (_do_fork+0x0/0x3a0) comm="systemd" +^Cecho 0 > /sys/kernel/debug/tracing/events/kprobes/f0/enable + <...>-1642046 [001] d... 7463506.356102: f0: (_do_fork+0x0/0x3a0) comm="surftrace.py" +echo > /sys/kernel/debug/tracing/kprobe_events +echo 0 > /sys/kernel/debug/tracing/tracing_on +``` + +参数表达式中，第一个 comm是变量名，可以自己定义，%0 表示第一个入参，%1 表示第二入参，以此类推。~连接符号表示的是后面会紧跟结构化成员，surftrace会根据解析结果得到comm成员类型是string并显示出来。 + +## 3.3 结构体级联和扩展 + +仍以 _do_fork 和 struct task_struct为例，在一个结构化表达式中，uesrs=%0**S**~(struct task_struct)->mm->mm_users，入参编号%0与连接符~中间增加了一个S字母来指定整数显示格式，共有SUX三种类型，分别对应有符号十进制、无符号十进制和十六进制。如果不指定，默认是X，16进制 + +```bash +#级联指针和指定整数数据格式 +./surftrace.py 'p _do_fork comm=%0->comm uesrs=S%0->mm->mm_users' +echo 'p:f0 _do_fork comm=+0xafc(%di):string uesrs=+0x48(+0x858(%di)):s32 ' >> /sys/kernel/debug/tracing/kprobe_events +echo 1 > /sys/kernel/debug/tracing/events/kprobes/f0/enable +echo 0 > /sys/kernel/debug/tracing/options/stacktrace +echo 1 > /sys/kernel/debug/tracing/tracing_on + <...>-1650321 [005] d... 7464948.730210: f0: (_do_fork+0x0/0x3a0) comm="surftrace.py" uesrs=1 + systemd-1 [000] d... 7464949.359231: f0: (_do_fork+0x0/0x3a0) comm="systemd" uesrs=1 + <...>-1650361 [005] d... 7464949.424381: f0: (_do_fork+0x0/0x3a0) comm="sh" uesrs=1 + <...>-1650361 [005] d... 7464949.424606: f0: (_do_fork+0x0/0x3a0) comm="sh" uesrs=1 + python-16819 [004] d... 7464950.235552: f0: (_do_fork+0x0/0x3a0) comm="python" uesrs=4 + \.... + #级联结构体成员 + ./surftrace.py 'p _do_fork comm=%0->comm node=%0->pids[1].node.next' +echo 'p:f0 _do_fork comm=+0xafc(%di):string node=+0x988(%di):x64 ' >> /sys/kernel/debug/tracing/kprobe_events +echo 1 > /sys/kernel/debug/tracing/events/kprobes/f0/enable +echo 0 > /sys/kernel/debug/tracing/options/stacktrace +echo 1 > /sys/kernel/debug/tracing/tracing_on + <...>-1652761 [000] d... 7465380.899543: f0: (_do_fork+0x0/0x3a0) comm="surftrace.py" node=0x0 + systemd-1 [000] d... 7465381.610144: f0: (_do_fork+0x0/0x3a0) comm="systemd" node=0x0 + python-16819 [001] d... 7465382.017634: f0: (_do_fork+0x0/0x3a0) comm="python" node=0xffff88062231be08 +``` + +## 3.4 设置过滤器 + +过滤器需要放在表达式最后，以f:开头，可以使用括号和&& ||逻辑表达式进行组合，具体写法可以参考ftrace文档说明 + +```bash +./surftrace.py 'p _do_fork comm=%0->comm uesrs=S%0->mm->mm_users f:comm==systemd' +echo 'p:f0 _do_fork comm=+0xafc(%di):string uesrs=+0x48(+0x858(%di)):s32' >> /sys/kernel/debug/tracing/kprobe_events +echo 'comm==systemd' > /sys/kernel/debug/tracing/instances/surftrace/events/kprobes/f0/filter +echo 1 > /sys/kernel/debug/tracing/instances/surftrace/events/kprobes/f0/enable +echo 0 > /sys/kernel/debug/tracing/instances/surftrace/options/stacktrace +echo 1 > /sys/kernel/debug/tracing/instances/surftrace/tracing_on + systemd-1 [002] d... 4817737.060026: f0: (_do_fork+0x0/0x3a0) comm="systemd" uesrs=1 + systemd-1 [002] d... 4817738.310035: f0: (_do_fork+0x0/0x3a0) comm="systemd" uesrs=1 + systemd-1 [002] d... 4817739.560046: f0: (_do_fork+0x0/0x3a0) comm="systemd" uesrs=1 + systemd-1 [001] d... 4817740.607892: f0: (_do_fork+0x0/0x3a0) comm="systemd" uesrs=1 + systemd-1 [001] d... 4817741.810041: f0: (_do_fork+0x0/0x3a0) comm="systemd" uesrs=1 + + #./surftrace.py 'p _do_fork comm=%0->comm users=S%0->mm->mm_users f:comm==python||users<4' +echo 'p:f0 _do_fork comm=+0xafc(%di):string users=+0x48(+0x858(%di)):s32 ' >> /sys/kernel/debug/tracing/kprobe_events +echo 'comm==python||users<4' > /sys/kernel/debug/tracing/events/kprobes/f0/filter +echo 1 > /sys/kernel/debug/tracing/events/kprobes/f0/enable +echo 0 > /sys/kernel/debug/tracing/options/stacktrace +echo 1 > /sys/kernel/debug/tracing/tracing_on + <...>-1655729 [004] d... 7465872.793430: f0: (_do_fork+0x0/0x3a0) comm="surftrace.py" users=1 + systemd-1 [000] d... 7465872.861176: f0: (_do_fork+0x0/0x3a0) comm="systemd" users=1 + systemd-1 [000] d... 7465874.111185: f0: (_do_fork+0x0/0x3a0) comm="systemd" users=1 + <...>-1655773 [003] d... 7465874.123909: f0: (_do_fork+0x0/0x3a0) comm="sh" users=1 + <...>-1655773 [003] d... 7465874.124134: f0: (_do_fork+0x0/0x3a0) comm="sh" users=2 + python-16819 [002] d... 7465874.909735: f0: (_do_fork+0x0/0x3a0) comm="python" users=4 + systemd-1 [000] d... 7465875.361189: f0: (_do_fork+0x0/0x3a0) comm="systemd" users=1 +``` + +我们还会常用common_pid作为current tid进行过滤，该变量由系统提供，无需定义 + +```bash +#./surftrace.py 'p _do_fork comm=%0->comm users=S%0->mm->mm_users f:common_pid==1&&(comm==python||users<4)' +echo 'p:f0 _do_fork comm=+0xafc(%di):string users=+0x48(+0x858(%di)):s32 ' >> /sys/kernel/debug/tracing/kprobe_events +echo 'common_pid==1&&(comm==python||users<4)' > /sys/kernel/debug/tracing/events/kprobes/f0/filter +echo 1 > /sys/kernel/debug/tracing/events/kprobes/f0/enable +echo 0 > /sys/kernel/debug/tracing/options/stacktrace +echo 1 > /sys/kernel/debug/tracing/tracing_on + systemd-1 [004] d... 7466113.361698: f0: (_do_fork+0x0/0x3a0) comm="systemd" users=1 + systemd-1 [004] d... 7466114.611729: f0: (_do_fork+0x0/0x3a0) comm="systemd" users=1 + systemd-1 [004] d... 7466115.861707: f0: (_do_fork+0x0/0x3a0) comm="systemd" users=1 + systemd-1 [004] d... 7466117.111716: f0: (_do_fork+0x0/0x3a0) comm="systemd" users=1 +``` + +## 3.5 函数内部追踪 + +以一下汇编代码为例，要获取偏移21位置时的%r12值 + +``` +disas _do_fork +Dump of assembler code for function _do_fork: + 0xffffffff8108a560 <+0>: callq 0xffffffff8174db10 <__fentry__> + 0xffffffff8108a565 <+5>: push %rbp + 0xffffffff8108a566 <+6>: mov %rsp,%rbp + 0xffffffff8108a569 <+9>: push %r15 + 0xffffffff8108a56b <+11>: push %r14 + 0xffffffff8108a56d <+13>: push %r13 + 0xffffffff8108a56f <+15>: push %r12 + 0xffffffff8108a571 <+17>: xor %r14d,%r14d + 0xffffffff8108a574 <+20>: push %rbx + 0xffffffff8108a575 <+21>: mov %rdx,%r12 +``` + +``` +#./surftrace.py 'p _do_fork+21 r=%r12' +echo 'p:f0 _do_fork+21 r=%r12 ' >> /sys/kernel/debug/tracing/kprobe_events +echo 1 > /sys/kernel/debug/tracing/events/kprobes/f0/enable +echo 0 > /sys/kernel/debug/tracing/options/stacktrace +echo 1 > /sys/kernel/debug/tracing/tracing_on + <...>-1659986 [000] d... 7466637.258116: f0: (_do_fork+0x15/0x3a0) r=0x38 + python-16819 [004] d... 7466637.577201: f0: (_do_fork+0x15/0x3a0) r=0x38 + <...>-497835 [004] d... 7466638.002734: f0: (_do_fork+0x15/0x3a0) r=0x3a + <...>-497835 [004] d... 7466638.003674: f0: (_do_fork+0x15/0x3a0) r=0x38 +``` + +同样的，也可以对寄存器对应的指针进行解析和过滤，这里就不在详细展开了 + +## 3.6、函数返回值获取 + +这个和原kprobe方法一致，返回值用$retval 表示 + +``` +#./surftrace.py 'r _do_fork r=$retval' +echo 'r:f0 _do_fork r=$retval ' >> /sys/kernel/debug/tracing/kprobe_events +echo 1 > /sys/kernel/debug/tracing/events/kprobes/f0/enable +echo 0 > /sys/kernel/debug/tracing/options/stacktrace +echo 1 > /sys/kernel/debug/tracing/tracing_on + <...>-1661938 [000] d... 7466977.685020: f0: (SyS_clone+0x36/0x40 <- _do_fork) r=0x195bfe + systemd-1 [004] d... 7466977.863627: f0: (SyS_clone+0x36/0x40 <- _do_fork) r=0x195bff + systemd-1 [004] d... 7466979.113626: f0: (SyS_clone+0x36/0x40 <- _do_fork) r=0x195c00 + <...>-497835 [000] d... 7466979.731526: f0: (sys_vfork+0x3c/0x40 <- _do_fork) r=0x195c01 +``` + +## 3.7、skb解析处理 + +sk_buff 是linux网络协议栈重要的结构体，但是通过上面的方法，并不能直接解析到我们关注的报文内容，需要进行特殊处理。以追踪icmp接收ping报文为例，我们在__netif_receive_skb_core 函数中进行probe和过滤 + +```bash +#./surftrace.py 'p __netif_receive_skb_core proto=@(struct iphdr *)l3%0->protocol ip_src=@(struct iphdr *)%0->saddr ip_dst=@(struct iphdr *)l3%0->daddr data=X@(struct iphdr *)l3%0->sdata[1] f:proto==1&&ip_src==127.0.0.1' +echo 'p:f0 __netif_receive_skb_core proto=+0x9(+0xf0(%di)):u8 ip_src=+0xc(+0xf0(%di)):u32 ip_dst=+0x10(+0xf0(%di)):u32 data=+0x16(+0xf0(%di)):x16' >> /sys/kernel/debug/tracing/kprobe_events +echo 'proto==1&&ip_src==0x100007f' > /sys/kernel/debug/tracing/instances/surftrace/events/kprobes/f0/filter +echo 1 > /sys/kernel/debug/tracing/instances/surftrace/events/kprobes/f0/enable +echo 0 > /sys/kernel/debug/tracing/instances/surftrace/options/stacktrace +echo 1 > /sys/kernel/debug/tracing/instances/surftrace/tracing_on + <...>-2076163 [000] d.s1 4818041.743856: f0: (__netif_receive_skb_core+0x0/0xa80) proto=1 ip_src=127.0.0.1 ip_dst=127.0.0.1 data=0x4dec + <...>-2076163 [000] d.s1 4818041.743905: f0: (__netif_receive_skb_core+0x0/0xa80) proto=1 ip_src=127.0.0.1 ip_dst=127.0.0.1 data=0x4df4 + <...>-2076163 [000] d.s1 4818042.767865: f0: (__netif_receive_skb_core+0x0/0xa80) proto=1 ip_src=127.0.0.1 ip_dst=127.0.0.1 data=0xef26 + <...>-2076163 [000] d.s1 4818042.767914: f0: (__netif_receive_skb_core+0x0/0xa80) proto=1 ip_src=127.0.0.1 ip_dst=127.0.0.1 data=0xef2e + <...>-2076163 [000] d.s1 4818043.791858: f0: (__netif_receive_skb_core+0x0/0xa80) proto=1 ip_src=127.0.0.1 ip_dst=127.0.0.1 data=0x9069 + <...>-2076163 [000] d.s1 4818043.791905: f0: (__netif_receive_skb_core+0x0/0xa80) proto=1 ip_src=127.0.0.1 ip_dst=127.0.0.1 data=0x9071 + <...>-2076163 [000] d.s1 4818044.815861: f0: (__netif_receive_skb_core+0x0/0xa80) proto=1 ip_src=127.0.0.1 ip_dst=127.0.0.1 data=0x31a6 + <...>-2076163 [000] d.s1 4818044.815911: f0: (__netif_receive_skb_core+0x0/0xa80) proto=1 ip_src=127.0.0.1 ip_dst=127.0.0.1 data=0x31ae +``` + +协议的获取表达式为 @(struct iphdr *)l3%0->protocol，和之前不一样的是，寄存器的结构体名左括号加了@符号进行特殊标记，表示需要用该结构体来解析skb->data指针数据，结构体名和右括号后加了l3标记（命名为右标记），表示当前skb->data指向了TCP/IP 层3位置。 + +右标记有l2、l3、l4三个选项，也可以不标记，默认为l3，如 ip_src=@(struct iphdr *)%0->saddr，没有右标记。 + +报文结构体有 'struct ethhdr', 'struct iphdr', 'struct icmphdr', 'struct tcphdr', 'struct udphdr'五类，如果协议栈层级和报文结构体对应不上，解析器会报参数错误，如右标记为l3，但是报文结构体是 struct ethhdr类型； + +'struct icmphdr', 'struct tcphdr', 'struct udphdr'这三个4层结构体增加了xdata成员，用于获取协议对应报文内容。xdata有 cdata. sdata, ldata, qdata 四类场景，位宽对应 1 2 4 8. 数组下标是按照位宽进行对齐的，如实例表达式中的 data=%0~$(struct icmphdr)l3->sdata[1],sdata[1]表示要提取icmp报文中的2~3字节内容 + +surftrace 会对以 ip_xx开头的变量进行ipv4<->u32 ，如 ip_src=@(struct iphdr *)%0->saddr，会转成对应的IP格式。对B16\_、B32\_、B64\_、b16\_、b32\_、b64\_开头的变量也会进行大小端转换，B开头按照16进制输出，b以10进制输出。 + +## 3.8 event 事件处理 +trace event 信息参考 /sys/kernel/debug/tracing/events目录下的事件 描述，以追踪wakeup等待超过10ms任务为例， +```bash +#./surftrace.py 'e sched/sched_stat_wait f:delay>1000000' +echo 'delay>1000000' > /sys/kernel/debug/tracing/instances/surftrace/events/sched/sched_stat_wait/filter +echo 1 > /sys/kernel/debug/tracing/instances/surftrace/events/sched/sched_stat_wait/enable +echo 0 > /sys/kernel/debug/tracing/instances/surftrace/options/stacktrace +echo 1 > /sys/kernel/debug/tracing/instances/surftrace/tracing_on + -0 [001] dN.. 11868700.419049: sched_stat_wait: comm=h2o pid=3046552 delay=87023763 [ns] + -0 [005] dN.. 11868700.419049: sched_stat_wait: comm=h2o pid=3046617 delay=87360020 [ns] + +``` \ No newline at end of file diff --git a/source/tools/detect/surftrace/surftrace.py b/source/tools/detect/surftrace/surftrace.py new file mode 100755 index 0000000000000000000000000000000000000000..799cb5a2034c02b5c90e9b884bc6a35a9cc3271b --- /dev/null +++ b/source/tools/detect/surftrace/surftrace.py @@ -0,0 +1,1637 @@ +#!/usr/bin/python +# -*- coding: utf-8 -*- +""" +------------------------------------------------- + File Name： surftrace + Description : + Author : liaozhaoyan + date： 2021/1/4 +------------------------------------------------- + Change Activity: + 2021/1/4: +------------------------------------------------- +""" +__author__ = 'liaozhaoyan' + +import re +import signal +import sys +import os +import socket +import shlex +import argparse +import struct +import json +import hashlib +from subprocess import PIPE, Popen +from threading import Thread +import select +import traceback +import sqlite3 +ON_POSIX = 'posix' in sys.builtin_module_names + +LBC_COMPILE_PORT = 7654 +LBCBuffSize = 80 * 1024 * 1024 +save2File = False +cmdStrings = "trap ':' INT QUIT TERM PIPE HUP\n" + +def saveCmd(s): + global save2File + global cmdStrings + if save2File: + cmdStrings += s + "\n" + +def getValueFromStr(s): + if s.startswith('0x') or s.startswith('0X'): + return int(s, 16) + else: + return int(s) + +def headTrans(head, value): + type = head.split('_', 1)[0] + if type == "ip": + v = getValueFromStr(value) + value = socket.inet_ntoa(struct.pack('>I', socket.htonl(v))) + elif type == "B16": + v = getValueFromStr(value) + v = struct.unpack('H', struct.pack('>H', v)) + value = "%x" % v + elif type == "B32": + v = getValueFromStr(value) + v = struct.unpack('I', struct.pack('>I', v)) + value = "%x" % v + elif type == "B64": + v = getValueFromStr(value) + v = struct.unpack('Q', struct.pack('>Q', v)) + value = "%x" % v + elif type == "b16": + v = getValueFromStr(value) + v = struct.unpack('H', struct.pack('>H', v)) + value = "%d" % v + elif type == "b32": + v = getValueFromStr(value) + v = struct.unpack('I', struct.pack('>I', v)) + value = "%d" % v + elif type == "b64": + v = getValueFromStr(value) + v = struct.unpack('Q', struct.pack('>Q', v)) + value = "%d" % v + return "%s=%s" % (head, value) + +def invHeadTrans(head, value): #为 filter 翻转 + type = head.split('_', 1)[0] + if type == "ip": + v = struct.unpack('I',socket.inet_aton(value))[0] + value = "0x%x" % v + elif type == "B16" or type == "b16": + v = getValueFromStr(value) + v = struct.unpack('H', struct.pack('>H', v)) + value = "%x" % v + elif type == "B32" or type == 'b32': + v = getValueFromStr(value) + v = struct.unpack('I', struct.pack('>I', v)) + value = "%x" % v + elif type == "B64" or type == 'b64': + v = getValueFromStr(value) + v = struct.unpack('Q', struct.pack('>Q', v)) + value = "%x" % v + return head, value + +class RootRequiredException(Exception): + def __init__(self, msg=""): + super(RootRequiredException, self).__init__(msg) + +class FileNotExistException(Exception): + def __init__(self, msg=""): + super(FileNotExistException, self).__init__(msg) + +class FileNotEmptyException(Exception): + def __init__(self, msg=""): + super(FileNotEmptyException, self).__init__(msg) + +class InvalidArgsException(Exception): + def __init__(self, msg=""): + super(InvalidArgsException, self).__init__(msg) + +class DbException(Exception): + def __init__(self, message): + super(DbException, self).__init__(message) + self.message = message + +class ExprException(Exception): + def __init__(self, message): + super(ExprException, self).__init__(message) + self.message = message + +# copy from surf expression.py +probeReserveVars = ('common_pid', 'common_preempt_count', 'common_flags', 'common_type') +archRegd = {'x86': ('di', 'si', 'dx', 'cx', 'r8', 'r9'), + 'aarch64': ('x0', 'x1', 'x2', 'x3', 'x4', 'x5', 'x6', 'x7'),} + +def maxNameString(cells, t): + rs = cells[0][t] + ret = "" + for i, r in enumerate(rs): + for f in cells[1:]: + sFunc = f[t] + if len(sFunc) < i or sFunc[i] != r: + return ret + ret += r + return ret + +def isStruct(text): + strips = ("const ", "volatile ") + for s in strips: + text = text.replace(s, "") + text = text.strip() + if text.startswith("struct ") or text.startswith("union "): + return text + return None + +def splitExpr(text): + es = [] + e = "" + count = 0 + for c in text: + if c == '(': + count += 1 + elif c == ")": + count -= 1 + if count < 0: + return None + elif count == 0 and c == " ": + if e != "": es.append(e) + e = "" + continue + e += c + if e != "": es.append(e) + return es + +def spiltInputLine(line): + res = {} + es = splitExpr(line) + esLen = len(es) + if esLen < 2: + raise ExprException("%s is a single expression" % line) + if es[0] not in ('p', 'r', 'e'): + raise ExprException("not support %s event." % es[0]) + res['type'] = es[0] + res['symbol'] = es[1] + if esLen >= 3: + if es[-1].startswith("f:"): + res['filter'] = es[-1][2:] + res['args'] = " ".join(es[2:-1]) + else: + res['filter'] = "" + res['args'] = " ".join(es[2:]) + else: + res['filter'] = ""; res['args'] = "" + return res + +def unpackRes(res): + s = "%s %s" % (res['type'], res['symbol']) + if res['filter'] == "": + if res['args'] == "": + return s + else: + return s + " %s" % res['args'] + else: + if res['args'] == "": + return s + " f:%s" % res['filter'] + else: + return s + " %s f:%s" % (res['args'], res['filter']) + +def stripPoint(sStruct): + return sStruct.strip("*").strip() + +def regIndex(reg, arch='x86'): + regs = archRegd[arch] + try: + return regs.index(reg) + except ValueError: + raise ExprException('%s is not a %s register.' % (reg, arch)) + +def transReg(i, arch='x86'): + try: + return archRegd[arch][i] + except IndexError: + raise ExprException('reg index %d overflow.' % i) + +class CasyncPipe(Thread): + def __init__(self, f, func): + if not os.path.exists(f): + FileNotExistException("%s is not exist." % f) + self.__callBack = func + super(CasyncPipe, self).__init__() + self.daemon = True # thread dies with the program + self.__pipe = open(f, 'r') + self.__loop = True + self.start() + + def newCb(self, func): + self.__callBack = func + + def run(self): + while self.__loop: + line = self.__pipe.readline().strip() + self.__callBack(line) + + def terminate(self): + self.__loop = False + self.join(1) + +class CexecCmd(object): + def __init__(self): + pass + + def cmd(self, cmds): + p = Popen(shlex.split(cmds), stdout=PIPE) + return p.stdout.read().strip() + + def system(self, cmds): + cmds = cmds.replace('\0', '').strip() + return os.popen(cmds) + +class CasyncCmdQue(object): + def __init__(self, cmd): + super(CasyncCmdQue, self).__init__() + self.daemon = True # thread dies with the program + self.__p = Popen(shlex.split(cmd), stdout=PIPE, stdin=PIPE, close_fds=ON_POSIX) + self.__e = select.epoll() + self.__e.register(self.__p.stdout.fileno(), select.EPOLLIN) + + def __del__(self): + self.__p.kill() + + def write(self, cmd): + try: + self.__p.stdin.write(cmd.encode()) + self.__p.stdin.flush() + except IOError: + return -1 + + def writeLine(self, cmd): + self.write(cmd + "\n") + + def read(self, tmout=0.2, l=16384): + while True: + es = self.__e.poll(tmout) + if not es: + return "" + for f, e in es: + if e & select.EPOLLIN: + s = os.read(f, l).decode() + return s + + def readw(self, want, tries=100): + i = 0 + r = "" + while i < tries: + line = self.read() + if want in line: + return r + line + r += line + i += 1 + raise Exception("get want args %s overtimes" % want) + + def terminate(self): + self.__p.terminate() + return self.__p.wait() + +class ftrace(object): + def __init__(self, show=False): + super(ftrace, self).__init__() + self._c = CexecCmd() + if show: + self.baseDir = "" + else: + self.__checkRoot() + self.baseDir = self.__getMountDir() + self.pipe = None + self._stopHook = [] + self._single = True + self.__ps = [] + + def __getMountDirStr(self): + cmd = "mount" + lines = self._c.cmd(cmd) + for l in lines.split('\n'): + if "type debugfs" in l: + return l + return None + + def __checkRoot(self): + cmd = 'whoami' + line = self._c.cmd(cmd).strip() + if line != "root": + raise RootRequiredException('this app need run as root') + + def __getMountDir(self): + s = self.__getMountDirStr() + if s is None: + return None + else: + return s.split(' ')[2] + + def tracePre(self, buffSize=2048): + pBuffersize = self.baseDir + "/tracing/instances/surftrace/buffer_size_kb" + self._echoPath(pBuffersize, "%d" % buffSize) + pTrace = self.baseDir + "/tracing/instances/surftrace/trace" + self._echoPath(pTrace) + + def _echoPath(self, path, value=""): + cmd = "echo %s > %s" % (value, path) + saveCmd(cmd) + print(cmd) + return self._c.system(cmd) + + def _echoDPath(self, path, value=""): + cmd = "echo %s >> %s" % (value, path) + saveCmd(cmd) + print(cmd) + return self._c.system(cmd) + + def procLine(self, line): + print(line) + return 0 + + def __stopTracing(self): + pOn = self.baseDir + "/tracing/instances/surftrace/tracing_on" + self._echoPath(pOn, "0") + + def _start(self): + pOn = self.baseDir + "/tracing/instances/surftrace/tracing_on" + self._echoPath(pOn, "1") + self._stopHook.insert(0, self.__stopTracing) + signal.signal(signal.SIGINT, self.signalHandler) + + def start(self): + self._single = True + self._start() + pipe = self.baseDir + "/tracing/instances/surftrace/trace_pipe" + self.pipe = CasyncPipe(pipe, self.procLine) + saveCmd("cat %s" % pipe) + + def signalHandler(self, signalNumber, frame): + if signalNumber == signal.SIGINT: + self.stop() + + def stop(self): + if self._single: + self.pipe.terminate() + else: + for p in self.__ps: + print("stop %d" % p.pid) + p.join() + for hook in self._stopHook: + hook() + + def loop(self): + signal.pause() + +class ClbcClient(object): + def __init__(self, ver="", server="pylcc.openanolis.cn"): + super(ClbcClient, self).__init__() + if ver == "": + c = CexecCmd() + ver = c.cmd('uname -r') + self._server = server + self._ver = ver + self._fastOff = False + + def _setupSocket(self): + addr = (self._server, LBC_COMPILE_PORT) + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.connect(addr) + return s + + @staticmethod + def _send_lbc(s, send): + send = "LBC%08x" % (len(send)) + send + s.send(send.encode()) + + @staticmethod + def _recv_lbc(s): + d = s.recv(LBCBuffSize).decode("utf-8") + if d[:3] != "LBC": + print("not lbc") + return None + size = d[3:11] + try: + size = int(size, 16) + 11 + except: + raise DbException("bad lbc Exception, %s" % size) + if size > LBCBuffSize: + return None + while len(d) < size: + d += s.recv(LBCBuffSize).decode("utf-8") + res = json.loads(d[11:]) + if res['log'] != "ok.": + raise DbException('db set return %s' % res["log"]) + return res + + def getFunc(self, func, ret=None, arg=None): + s = self._setupSocket() + dSend = {"ver": self._ver, + "cmd": "func", + "func": func} + if ret: dSend['ret'] = ret + if arg: dSend['arg'] = arg + self._send_lbc(s, json.dumps(dSend)) + return self._recv_lbc(s) + + def getStruct(self, sStruct): + s = self._setupSocket() + dSend = {"ver": self._ver, + "cmd": "struct", + "struct": sStruct} + self._send_lbc(s, json.dumps(dSend)) + return self._recv_lbc(s) + + def getType(self, t): + s = self._setupSocket() + if "*" in t: + t = '_' + dSend = {"ver": self._ver, + "cmd": "type", + "type": t} + self._send_lbc(s, json.dumps(dSend)) + return self._recv_lbc(s) + +class CdbParser(object): + def __init__(self, dbPath=""): + super(CdbParser, self).__init__() + if dbPath == "": + c = CexecCmd() + ver = c.cmd('uname -r') + dbPath = "info-%s.db" % ver + if not os.path.exists(dbPath): + raise DbException("db %s is not exist." % dbPath) + self._db = sqlite3.connect(dbPath) + self._fastOff = False + + def _dbCheckArg(self, k, v): + if ";" in v: + return {"log": "bad %s key." % k} + + def _genFuncRet(self, cur, res): + lR = [] + for r in res: + args = json.loads(r[1]) + fid = r[4] + sql = "SELECT file FROM files WHERE id = '%d'" % fid + res = cur.execute(sql) + txt = res.fetchone()[0] + d = {"func": r[0], "args": args, "ret": r[2], "line": r[3], "file": txt} + lR.append(d) + return lR + + def _chekcFunfilter(self, func): + if "%" not in func: + raise ValueError("bad arg %s, args should contains %." % func) + + def _getFuncs(self, func, limit=20): + self._chekcFunfilter(func) + cur = self._db.cursor() + sql = "SELECT func, args, ret, line, fid FROM funs WHERE func LIKE '%s' LIMIT %d" % (func, limit) + res = cur.execute(sql).fetchall() + lR = self._genFuncRet(cur, res) + cur.close() + return lR + + def _getFunc(self, func): + cur = self._db.cursor() + sql = "SELECT func, args, ret, line, fid FROM funs WHERE func = '%s'" % func + res = cur.execute(sql).fetchall() + lR = self._genFuncRet(cur, res) + cur.close() + return lR + + def _getFuncFilterRet(self, ret, func="", limit=100): + cur = self._db.cursor() + sql = "SELECT func, args, ret, line, fid FROM funs WHERE (ret = '%s' OR ret = 'static %s')" % (ret, ret) + if func != "": + self._chekcFunfilter(func) + sql += " AND func LIKE '%s'" % func + sql += " LIMIT %d" % limit + res = cur.execute(sql) + lR = self._genFuncRet(cur, res) + cur.close() + return lR + + def _getFuncFilterArg(self, arg, func="", limit=100): + cur = self._db.cursor() + sql = "SELECT func, args, ret, line, fid FROM funs, json_each(funs.args) WHERE json_each.value = '%s'" % arg + if func != "": + self._chekcFunfilter(func) + sql += " AND func LIKE '%s'" % func + sql += " LIMIT %d" % limit + res = cur.execute(sql) + lR = self._genFuncRet(cur, res) + cur.close() + return lR + + def __getStruct(self, cur, sStruct, limit=10): + if '%' in sStruct: + sql = "SELECT id, name, members, bytes FROM structs WHERE name LIKE '%s' LIMIT %d" % (sStruct, limit) + res = cur.execute(sql) + rDs = [] + for r in res.fetchall(): + rD = {"name": r[1], "members": r[2], "size": r[3]} + rDs.append(rD) + return rDs + + sql = "SELECT id, name, members, bytes FROM structs WHERE name = '%s'" % sStruct + res = cur.execute(sql) + r = res.fetchone() + if r is None: return None + fid = r[0] + rD = {"name": r[1], "members": r[2], "size": r[3]} + sql = "SELECT types, offset, bytes, bits, name FROM members WHERE fid = %d" % fid + res = cur.execute(sql) + cells = [] + for r in res.fetchall(): + dCell = {"type": r[0], "offset": r[1], "size": r[2], + "bits": r[3], "name": r[4]} + cells.append(dCell) + rD["cell"] = cells + return rD + + def _getStruct(self, sStruct): + cur = self._db.cursor() + r = self.__getStruct(cur, sStruct) + cur.close() + return r + + def _getType(self, t): + cur = self._db.cursor() + if '*' not in t: + sql = "SELECT name, alias, bytes FROM types WHERE name = '%s'" % t + else: + sql = "SELECT name, alias, bytes FROM types WHERE id = 1" + res = cur.execute(sql) + r = res.fetchone() + if r is None: return None + return {"name": r[0], "type": r[1], "size": r[2]} + + def getFunc(self, func, ret=None, arg=None): + dSend = {"log": "ok."} + if ret is not None: + r = self._dbCheckArg("ret", ret) + if r: return r + try: + dSend['res'] = self._getFuncFilterRet(ret, func) + return dSend + except ValueError: + return {"log": "query value error."} + if arg is not None: + r = self._dbCheckArg("arg", arg) + if r: return r + try: + dSend['res'] = self._getFuncFilterArg(arg, func) + return dSend + except ValueError: + return {"log": "query value error."} + if "%" in func: + dSend['res'] = self._getFuncs(func) + else: + dSend['res'] = self._getFunc(func) + return dSend + + def getStruct(self, sStruct): + dSend = {"log": "ok."} + dSend['res'] = self._getStruct(sStruct) + return dSend + + def getType(self, t): + dSend = {"log": "ok."} + dSend['res'] = self._getType(t) + return dSend + +class CgdbParser(object): + def __init__(self, vmlinuxPath="", gdb=None): + self._res = {} + + self._reBrackets = re.compile(r"(?<=\().+?(?=\))") + self._reSquareBrackets = re.compile(r"(?<=\[).+?(?=\])") + self._reAngleBrackets = re.compile(r"(?<=\<).+?(?=\>)") + self._reGdbVersion = re.compile(r"[\d]+\.[\d]+") + self._rePaholeRem1 = re.compile(r"\/\* *[\d]+( *|: *[\d] *)\| *[\d]+ *\*\/") + self._rePaholeRem2 = re.compile(r"\/\* *[\d]+ *\*\/") + self._reRem = re.compile(r"\/\*.*\*\/") + + self._cmd = CexecCmd() + if gdb is None: + gdb = self._checkGdbExist() + if vmlinuxPath == "": + vmlinuxPath = self.__getVmlinuxPath() + if not os.path.exists(vmlinuxPath): + raise FileNotExistException("vmlinux %s is not found." % (vmlinuxPath)) + + cmd = '%s %s' % (gdb, vmlinuxPath) + self.__want = "(gdb)" + self.__aCmd = CasyncCmdQue(cmd) + self._read() + self._write('set pagination off') + self._read() + self._pSize = self.__showTypeSize("void *") + self._iSize = self.__showTypeSize("int") + self._checkGdbVer() + + def __del__(self): + self.__aCmd.terminate() + + def _write(self, l): + self.__aCmd.writeLine(l) + + def _read(self, tries=100): + return self.__aCmd.readw(self.__want, tries) + + def _checkGdbExist(self): + cmd = 'which gdb' + line = self._cmd.cmd(cmd) + if line == "": + raise FileNotExistException("gdb is not install") + return "gdb" + + def _checkGdbVer(self): + self._write("show version") + lines = self._read().split('\n') + res = self._reGdbVersion.search(lines[0]) + if res is None: + raise FileNotExistException("%s, unknown gdb version." % lines[0]) + major, minor = res.group().split(".") + if int(major) < 9: + s = "you gdb version is %s, lower than 9.x." % res.group() + s += " A high version of the gdb is required to achieve full functionality.\n" + s += "if your arch is x86_64, you can wget http://pylcc.openanolis.cn/gdb/x64/gdb then set gdb path args." + raise FileNotExistException(s) + + def __getVmlinuxPath(self): + name = self._cmd.cmd('uname -r') + return "/usr/lib/debug/usr/lib/modules/" + name + "/vmlinux" + + def _showStruct(self, sStruct): + self._write("ptype /o %s" % sStruct) + return self._read() + + def __showTypeSize(self, sType): + if "..." in sType: + return 0 + cmd = "p sizeof(%s)" % sType + self._write(cmd) + readStr = self._read().split('\n')[0] + try: + nStr = readStr.split('=')[1].strip() + except IndexError: + return 0 + return int(nStr) + + def _showTypeSize(self, sType): + if "*" in sType: + return self._pSize + elif sType.startswith("enum"): + return self._iSize + return self.__showTypeSize(sType) + + def _showMemberOffset(self, member, structName): + structName = structName.replace("*", "") + cmd = "p &((%s*)0)->%s" % (structName, member) + self._write(cmd) + s = self._read() + if s is None: + raise InvalidArgsException("%s is not in %s" % (member, structName)) + readStr = s.strip() + nStr = self._reAngleBrackets.findall(readStr)[0] + if "+" not in nStr: + return 0 + return int(nStr.split("+")[1]) + + def showMemberOffset(self, member, structName): + self._showMemberOffset(member, structName) + + def _setupRes(self): + self._res = {'log': 'ok.'} + + def _argFuncSplit(self, argStr): + args = [] + arg = "" + count = 0 + + for a in argStr: + if count == 0 and a == ",": + args.append(arg.strip()) + arg = "" + continue + elif a == "(": + count += 1 + elif a == ")": + count -= 1 + arg += a + if arg != "": + args.append(arg.strip()) + return args + + def _showFuncs(self, func): + cmd = "i func %s" % func + self._write(cmd) + return self._read() + + def getFunc(self, func, ret=None, arg=None): + self._setupRes() + if len(func) < 2: + return {"log": "func len should bigger than 2.", "res": None} + + self._res['res'] = [] + func = func.replace("%", "*") + func = "^" + func + lines = self._showFuncs(func).split('\n') + File = "" + funcs = 0 + for line in lines[1:]: + if line == "": + continue + elif line.startswith("(gdb)"): + break + elif line.startswith("File "): + _, sFile = line.split(" ", 1) + File = sFile[:-1] + elif line.endswith(");"): + # 8: static int __paravirt_pgd_alloc(struct mm_struct *); + line = line[:-2] + if ':' in line: + lineNo, body = line.split(":", 1) + else: + lineNo = '0' + body = line + head, args = body.split("(", 1) + # args = [x.strip() for x in args.split(",")] + args = self._argFuncSplit(args) + if "*" in head: + ret, func = head.rsplit("*", 1) + ret += "*" + else: + ret, func = head.rsplit(" ", 1) + funcd = {'func': func, 'args': args, 'ret': ret, 'line': int(lineNo), 'file': File} + self._res['res'].append(funcd) + + funcs += 1 + if funcs > 20: + break + return self._res + + def _stripRem(self, line): + return self._reRem.sub("", line).strip() + + def _splitStructLine(self, line): + rd = {"offset": None, "size": None, "bits": None} + + res = self._rePaholeRem1.search(line) + if res: + l = res.group()[2:-2].strip() + # /* 19: 0 | 1 */ unsigned char skc_reuse : 4; + # /* 19: 4 | 1 */ unsigned char skc_reuseport : 1; + off, size = l.split('|', 1) + rd["size"] = int(size.strip()) + if ":" in off: + off, bits = off.split(":", 1) + rd['bits'] = bits.strip() # offset + rd["offset"] = int(off.strip()) + else: + res = self._rePaholeRem2.search(line) + if res: + l = res.group()[2:-2].strip() + # /* 8 | 4 */ union { + # /* 4 */ unsigned int skc_hash; + # /* 4 */ __u16 skc_u16hashes[2]; + size = l.strip() + rd["size"] = int(size.strip()) + rd["line"] = self._stripRem(line) + return rd + + def _parseMember(self, sStruct, line, pre="", off=0): + """struct list_head * next;""" + """void (*func)(struct callback_head *);""" + """unsigned int p:1;""" + if ";" not in line: + return + rd = self._splitStructLine(line) + l = rd['line'] + bits = "" + if ':' in l: + l, bits = l.rsplit(" : ", 1) + bits = "%s:%s" % (rd["bits"], bits) + if l[-1] == ')': + _, func = l.split("(*", 1) + func, _ = func.split(")", 1) + types = line.replace(" (*%s)(" % func, " (*)(", 1) + types = re.sub(" +", " ", types) + name = func + elif '*' in l: + types, name = l.rsplit("*", 1) + types = types + "*" + name = name.strip("; ") + else: + types, name = l.rsplit(" ", 1) + types = types.strip() + name = name.strip("; ") + name = pre + name + + if rd["offset"] is None: + rd["offset"] = off + cell = {"type": types, "name": name, "offset": rd["offset"], + "size": rd["size"], "bits": bits} + self._res['res']['cell'].append(cell) + + def _parseBox(self, sStruct, lines, pre): + """union {""" + """} pci;""" + rd = self._splitStructLine(lines[0]) + t = rd['line'].split(" ", 1)[0] + if t in ["union", "struct"]: + lastLine = lines[-1].strip() + if not lastLine.startswith("};"): + npre, _ = lastLine[1:].split(";", 1) + _, npre = npre.rsplit(" ", 1) + pre += npre.strip() + "." + self._parseLoop(sStruct, lines, pre, rd["offset"]) + + def _parseLoop(self, sStruct, lines, pre, off=0): + qCount = 0 + box = [] + for line in lines[1:-1]: + lCount = line.count("{") + rCount = line.count("}") + qCount += lCount - rCount + if qCount > 0: + box.append(line) + elif len(box) > 0: + box.append(line) + self._parseBox(sStruct, box, pre) + box = [] + else: + self._parseMember(sStruct, line, pre, off) + + def getStruct(self, sStruct): + self._setupRes() + lines = self._showStruct(sStruct).split('\n') + l = self._splitStructLine(lines[0])['line'] + name = l.split('=', 1)[1] + name = name.split('{', 1)[0].strip() + self._res['res'] = {"name": name, "size": self._showTypeSize(name), "cell": []} + self._parseLoop(name, lines, "") + self._res['res']['members'] = len(self._res['res']['cell']) + return self._res + + def testStructs(self, sStruct): + self._setupRes() + with open("struct.txt", 'r') as f: + lines = f.read().split('\n') + l = self._splitStructLine(lines[0])['line'] + name = l.split('=', 1)[1] + name = name.split('{', 1)[0].strip() + self._res['res'] = {"name": name, "size": self._showTypeSize(name), "cell": []} + self._parseLoop(name, lines, "") + self._res['res']['members'] = len(self._res['res']['cell']) + return self._res + + def getType(self, sType): + self._setupRes() + lines = self._showStruct(sType).split('\n') + _, alias = lines[0].split("=", 1) + alias = alias.strip() + size = self._showTypeSize(sType) + res = {'name': sType, 'type': alias, 'size': size} + self._res['res'] = res + return self._res + + def __endAdd1(self, s): + if "+" not in s: + return s + start, num = s.split("+", 1) + num = int(num) + 1 + return "%s+%d" % (start, num) + + def parserLine(self, line): + cmd = "i line %s" % line + self._write(cmd) + s = self._read(tries=100) + md5s = [] + cnt = 0 + for l in s.split("\n")[:-1]: + res = self._reAngleBrackets.findall(l) + if res is None: + raise InvalidArgsException("%s is not a func head, eg. fs/stat.c:145", line) + elif len(res) == 1: + beg = end = res[0] + else: + beg = res[0]; + end = res[1] + cmd = "disas %s,%s" % (beg, end) + self._write(cmd) + try: + show = self._read() + except: + continue + md5 = hashlib.md5(show).hexdigest() + if md5 not in md5s: + cnt += 1 + md5s.append(md5) + print("show disas %d, %s:" % (cnt, md5)) + print(self._stripGdb(show)) + + def _stripGdb(self, line): + ls = line.split('\n') + return '\n'.join(ls[:-1]) + + def disasFun(self, func): + cmd = "disas /m %s" % func + self._write(cmd) + show = self._read() + print(self._stripGdb(show)) + +class surftrace(ftrace): + def __init__(self, args, parser, show=False, arch="", stack=False, cb=None, cbOrig=None): + super(surftrace, self).__init__() + self._parser = parser + self._c = CexecCmd() + self._probes = [] + self._events = [] + if not show: + self._checkIsEmpty() + self._arch = arch + if self._arch == "": + self._arch = self._getArchitecture() + self.__args = args + self._show = show + self._stack = stack + self._reSurfProbe = re.compile(r"[a-zA-z][a-zA-z0-9_]*=[SUX]?(@\(struct .*\*\)(l[234]|)|\!\(.*\)|)%") + self._reSurfRet = re.compile(r"[a-zA-z][a-zA-z0-9_]*=[SUX]?(@\(struct .*\*\)(l[234]|)|\!\(.*\)|)\$retval") + self._reLayer = re.compile(r"l[234]") + self._reBrackets = re.compile(r"(?<=\().+?(?=\))") + self._reSquareBrackets = re.compile(r"(?<=\[).+?(?=\])") + self._netStructs = ('struct ethhdr', 'struct iphdr', 'struct icmphdr', 'struct tcphdr', 'struct udphdr') + self._netDatas = {'cdata': (1, "unsigned char"), + 'sdata': (2, "unsigned short"), + 'ldata': (4, "unsigned int"), + 'qdata': (8, "unsigned long"), + 'Sdata': (1, "char"), + } + self._strFxpr = "" + self.__format = 'u' + self._func = None + + self._cb = cb + self._cbOrig = cbOrig + + def _getArchitecture(self): + lines = self._c.cmd('lscpu').split('\n') + for line in lines: + if line.startswith("Architecture"): + arch = line.split(":", 1)[1].strip() + if arch.startswith("arm"): + return "arm" + if arch.startswith('x86'): + return "x86" + return arch + return "Unkown" + + def _clearProbes(self): + if self._show: + return + for p in self._probes: + fPath = self.baseDir + "/tracing/instances/surftrace/events/kprobes/" + p + "/enable" + self._echoPath(fPath, 0) + cmd = '-:%s' % p + self._echoDPath(self.baseDir + "/tracing/kprobe_events", cmd) + self._probes = [] + for ePath in self._events: + self._echoPath(ePath, 0) + self._events = [] + + def __transFilter(self, filter, i, beg): + decisions = ('==', '!=', '~', '>=', '<=', '>', '<') + s = filter[beg:i] + for d in decisions: + if d in s: + k, v = s.split(d) + if '_' in k: + k, v = invHeadTrans(k, v) + return "%s%s%s" % (k, d, v) + raise InvalidArgsException("bad filter format %s" % s) + + def __checkFilter(self, filter): + cpStr = "()|&" + beg = 0; ret = "" + l = len(filter) + for i in range(l): + if filter[i] in cpStr: + if i and beg != i: + ret += self.__transFilter(filter, i, beg) + beg = i + 1 + ret += filter[i] + if beg != l: + ret += self.__transFilter(filter, l, beg) + return ret + + def __showExpression(self, head, cmd): + c = cmd.split(' ', 1)[1] + print("%s %s" % (head, c)) + + def __setupEvent(self, res): + e = res['symbol'] + eBase = os.path.join(self.baseDir, "tracing/instances/surftrace/events") + eDir = os.path.join(eBase, e) + if not os.path.exists(eDir): + raise InvalidArgsException("event %s is not an available event, see %s" % (e, eBase)) + if res['filter'] != "": + filter = res['filter'] + try: + filter = self.__checkFilter(filter) + except Exception as e: + print(e.message) + raise InvalidArgsException('bad filter：%s' % filter) + if self._show: + self.__showExpression('e', ' ' + e + 'f:%s' % filter) + return + else: + fPath = os.path.join(eDir, 'filter') + self._echoPath(fPath, "'" + filter + "'") + if self._show: + self.__showExpression('e', ' ' + e) + return + else: + ePath = os.path.join(eDir, 'enable') + self._echoPath(ePath, 1) + self._events.append(ePath) + + def _memINStruct(self, mem, tStruct): + if tStruct is None: + return None + arrMem = False + if '[' in mem: + mem = mem.split('[', 1)[0] + arrMem = True + for cell in tStruct["res"]["cell"]: + name = cell['name'] + if arrMem: + if '[' not in name: + continue + if '[' in name: + name = name.split('[', 1)[0] + if name == mem: + return cell + elif not arrMem and name.startswith(mem + "."): + return {"type": "struct"} + return None + + def __checkFormat(self, e): + _, flag = e.split("=") + self.__format = 'u' + if flag[0] in "SUX": + self.__format = str.lower(flag[0]) + + def __checkBegExpr(self, e): + if self._res['type'] == 'p': + res = self._reSurfProbe.match(e) + else: + res = self._reSurfRet.match(e) + if res is None: + raise ExprException("error in expr %s." % e) + return res + + def _splitXpr(self, xpr): + for i, c in enumerate(xpr): + if c in ('.', '-'): + return xpr[:i], xpr[i:] + return xpr, '' + + def __checkVar(self, var): + if var in probeReserveVars: + raise ExprException('%s is reserve word, can not used for args' % var) + + def __checkSkbStruct(self, sStrcut): + if sStrcut.strip("* ") not in self._netStructs: + raise ExprException("type: %s is no not a legal struct." % sStrcut) + + def __filtType(self, s): + try: + return self._reBrackets.findall(s)[0] + except (TypeError, IndexError): + raise ExprException("%s may not match" % s) + + def __netParse(self, sType, pr): + self.__checkSkbStruct(sType) + if self._reLayer.match(pr): + pr = pr[2:] + return pr + + def showTypeSize(self, sType): + multi = 1 + if "[" in sType: + sType, multi = sType.split("[", 1) + multi = int(multi[:-1]) + if '*' in sType: + sType = "_" + if isStruct(sType): + dRes = self._getVmStruct(sType) + else: + dRes = self._getVmType(sType) + return multi * dRes['res']['size'] + + def showMemberOffset(self, member, structName): + add = 0 + if '[' in structName: + structName = structName.split('[')[0].strip() + + if '[' in member: + add = int(self._reSquareBrackets.findall(member)[0]) + member = member.split('[')[0] + structName = structName.replace("*", "").strip() + dRes = self._getVmStruct(structName) + for cell in dRes['res']['cell']: + name = cell['name'] + indexMax = 0 + if '[' in name: + indexMax = int(self._reSquareBrackets.findall(name)[0]) + name = name.split("[")[0] + if name == member: + if add > 0: + if add >= indexMax: + raise ExprException("member %s max index is %d, input %d, overflow" % (name, indexMax, add)) + add *= int(cell['size'] / indexMax) + return cell['offset'] + add + raise ExprException("there is not member named %s in %s" % (member, structName)) + + def _getVmStruct(self, sStruct): + size = 1536 # for skb_buff + try: + res = self._parser.getStruct(stripPoint(sStruct)) + except DbException as e: + raise ExprException('db get %s return %s' % (sStruct, e.message)) + if 'res' in res and res['res'] is not None and res['res']['name'] in self._netStructs: + offset = res['res']['size'] + for k, v in self._netDatas.items(): + name = "%s[%d]" % (k, size/v[0]) + cell = {"type": v[1], 'offset': offset, "size": size, "bits": "", "name": name} + res['res']['cell'].append(cell) + return res + + def _getVmType(self, sType): + try: + res = self._parser.getType(sType) + except DbException as e: + raise ExprException('db get %s return %s' % (sType, e.message)) + return res + + def _getVmFunc(self, func): + try: + res = self._parser.getFunc(func) + except DbException as e: + raise ExprException('db get %s return %s' % (func, e.message)) + return res + + def __getExprArgi(self, e): + # expression: a=@(struct iphdr *)l4%1->saddr uesrs=!(struct task_struct *)%0->mm->mm_users + # e is already checked at self.__checkBegExpr + var, expr = e.split("=", 1) + self.__checkVar(var) + + showType = '' + if expr[0] in ('S', 'U', 'X'): + expr = expr[1:] + showType = expr[0] + if self._res['type'] == 'p': + types, xpr = expr.split('%', 1) + reg, xpr = self._splitXpr(xpr) + if reg.isdigit(): + argi = int(reg) + else: + argi = regIndex(reg) + if types == '': + argt = self._func['args'][argi] + else: + argt = types + regArch = transReg(argi) + else: + types, xpr = expr.split('$retval') + if types == '': + argt = self._func['ret'] + else: + argt = types + regArch = '$retval' + return showType, regArch, argt, xpr + + def _splitPr(self, argt, prs): + cells = [] + beg = 0 + for i, c in enumerate(prs): + if c == ".": + cells.append(prs[beg:i]) + cells.append(".") + beg = i + 1 + elif c == '-': + if prs[i + 1] != '>': + raise ExprException("bad point mode, should be '->'") + cells.append(prs[beg:i]) + cells.append("->") + beg = i + 2 + if beg < len(prs): + cells.append(prs[beg:]) + if (len(cells)): + cells[0] = argt + else: + cells.append(argt) + return cells + + def _cellCheckArray(self, sMem, res): + name = res['name'] + if res['type'] == "char": + return + if '[' in name: + if '[' not in sMem: + raise ExprException("member %s is an array, should add [, member is %s" % (name, sMem)) + try: + iMem = self._reSquareBrackets.findall(sMem)[0] + iName = self._reSquareBrackets.findall(name)[0] + except TypeError: + raise ExprException("%s or %s is not in []" % (sMem, name)) + try: + iMem = int(iMem) + except ValueError: + raise ExprException("%s in %s is not int" % (iMem, sMem)) + if iName == "": + return + else: + try: + iName = int(iName) + except ValueError: + raise ExprException("remote type %s error" % name) + if iMem >= iName: + raise ExprException("%s max index is %d, you set %d, overflow" % (name, iName, iMem)) + + def _fxprAddPoint(self, off): + self._strFxpr = "+0x%x(" % off + self._strFxpr + ')' + + def _fxprAddMem(self, off): + # +0xa(%di) + try: + prev, cmd = self._strFxpr.split("(", 1) + except: + raise ExprException("regs should start with ->.") + last = int(prev[1:], 16) + self._strFxpr = "+0x%x(" % (off + last) + cmd + + def _fxprAddSuffix(self, lastCell): + sType = lastCell["type"] + if "char *" in sType: + self._strFxpr = "+0x0(%s):string" % self._strFxpr + elif (sType == "char" or " char" in sType ) and '[' in lastCell['name']: + self._strFxpr += ":string" + elif sType == "struct": + raise ExprException("lastCell type %s, which is incompletely." % lastCell["type"]) + else: + formDict = {1: 8, 2: 16, 4: 32, 8: 64} + if 'name' in lastCell: + name = lastCell['name'] + if "[" in name: + res = self._parser.getType(sType) + if res['log'] == "ok." and 'res' in res and res['res'] is not None: + cell = res['res'] + size = cell['size'] + else: + raise DbException("get type %s failed" % sType) + else: + size = lastCell['size'] + else: + res = self._parser.getType(sType) + if res['log'] == "ok." and 'res' in res and res['res'] is not None: + cell = res['res'] + size = cell['size'] + else: + raise DbException("get type %s failed" % sType) + if size in formDict: + self._strFxpr += ":%s%d" % (self.__format, formDict[size]) + else: + raise ExprException( + "last cell type: %s, can not show." % (lastCell["type"])) + + def _procSkb(self, member, sStruct, layer): + # struct is already checked in _cellCheck __getCellMem, func + off = self.showMemberOffset('data', 'struct sk_buff') + self._fxprAddPoint(off) + + sStruct = stripPoint(sStruct) + off = 0 + #layer 2 + if sStruct == 'struct ethhdr': + if layer != 2: + raise ExprException("can not get ethhdr at layer%d" % layer) + off += self.showMemberOffset(member, sStruct) + self._fxprAddPoint(off) + return + if layer == 2: + off += self.showTypeSize('struct ethhdr') + #layer 3 + if sStruct == 'struct iphdr': + if layer > 3: + raise ExprException("can not get iphdr at layer%d" % layer) + off += self.showMemberOffset(member, sStruct) + self._fxprAddPoint(off) + return + if layer < 4: + off += self.showTypeSize('struct iphdr') + #layer 4 + off += self.showMemberOffset(member, sStruct) + self._fxprAddPoint(off) + + def _procFxpr(self, member, structName, mode): + first = structName[0] + orig = structName + if first in ('(', '!', '@'): + structName = self.__filtType(structName) + if first == '@': + try: + layer = int(self._reLayer.match(orig)[0][1]) + except (TypeError, IndexError): + layer = 3 + if mode != '->': + raise ExprException("net struct process should in -> mode.") + self._procSkb(member, structName, layer) + return + off = self.showMemberOffset(member, structName) + if mode == '.': + self._fxprAddMem(off) + else: + self._fxprAddPoint(off) + + def __getCellMem(self, s): + sym = s[0] + sType = None; v = s + if sym in ('@', '(', '!'): + sType = self.__filtType(s) + _, v = s.split(')') + if sym == '@': + v = self.__netParse(sType, v) + return sType, v + + def _cellCheck(self, cells, reg): + if reg == "$retval": + self._strFxpr = reg + else: + self._strFxpr = "%" + reg + + i = 0; end = len(cells); lastCell = None + sMem = ""; origType = sType = "unkown"; tStruct = None; origMode = '->' + + if end <= 2: + sType = cells[0] + if '(' in sType: + lastCell = {"type": self.__filtType(sType)} + else: + lastCell = {"type": sType} + while i + 2 < end: + if sMem == "": + origType = sType = cells[i] + sMode = cells[i + 1] + tMem, sMem = self.__getCellMem(cells[i + 2]) + if sType[0] in ('(', '@', '!'): + sType = self.__filtType(sType) + if sMode == ".": + if sType.endswith("*"): + raise ExprException("%s is a point, should use '->' to get member" % sType) + elif sMode == "->": + if not sType.endswith("*"): + raise ExprException("%s is a type, should use '.' to get member" % sType) + origMode = sMode + sType = isStruct(sType) + if sType is None: + raise ExprException("%s is not a legal struct types." % sType) + tStruct = self._getVmStruct(sType) + if tStruct['res'] is None: + raise ExprException("%s is not a valid struct in database." % sType) + else: + sMode = cells[i + 1] + if sMode != ".": + ExprException("%s:nested structure members should be '.' mode, %s" % (sType, sMem)) + tMem, v = self.__getCellMem(cells[i + 2]) + sMem = "%s.%s" % (sMem, v) + lastCell = res = self._memINStruct(sMem, tStruct) + if res is None: + raise ExprException("%s is not a member in %s" % (sMem, sType)) + elif res['type'] != "struct": # not nested structs mode, if in nested, then clear. + self._cellCheckArray(sMem, res) + self._procFxpr(sMem, origType, origMode) + sMem = "" + if tMem is None: + cells[i + 2] = res['type'] + else: + cells[i + 2] = tMem + lastCell = {"type": tMem} + i += 2 + self._fxprAddSuffix(lastCell) + return lastCell + + def _checkExpr(self, e): + self.__checkBegExpr(e) + self.__checkFormat(e) + + showType, reg, argt, xpr = self.__getExprArgi(e) + + cells = self._splitPr(argt, xpr) + res = self._cellCheck(cells, reg) + if res['type'] == "struct": + raise ExprException("last member is nested struct type, which is not completed.") + return res + + def __initEvents(self, i, arg): + arg = arg.strip('\n') + if len(arg) == 0: + return + try: + res = spiltInputLine(arg) + except ExprException as e: + raise ExprException("expression %s error, %s" % (arg, e.message)) + if res['type'] == 'e': + self.__setupEvent(res) + return + + name = "f%d" % i + cmd = "%s:f%d " % (res['type'], i) + symbol = res['symbol'] + try: + self._func = self._getVmFunc(symbol)['res'][0] + except (TypeError, KeyError): + raise DbException("no %s debuginfo in file." % symbol) + if '+' in symbol: + func = symbol.split('+')[0] + else: + func = symbol + if not self._checkAvailable(func): + raise InvalidArgsException("%s is not in available_filter_functions" % func) + cmd += "%s" % symbol + + vars = [] + for expr in splitExpr(res['args']): + if expr == "": + continue + self._res = res + self._checkExpr(expr) + var, _ = expr.split("=", 1) + if var in vars: + raise ExprException("var %s is already used at previous expression" % var) + vars.append(var) + cmd += " %s=" % var + self._strFxpr + self._echoDPath(self.baseDir + "/tracing/kprobe_events", "'" + cmd + "'") + if res['filter'] != "": + try: + filter = self.__checkFilter(res['filter']) + except Exception as e: + print(e.message) + raise InvalidArgsException('bad filter：%s' % a[-1]) + if self._show: + self.__showExpression(head, cmd + 'f:%s' % filter) + else: + fPath = self.baseDir + "/tracing/instances/surftrace/events/kprobes/%s/filter" % name + self._echoPath(fPath, "'%s'" % filter) + fPath = self.baseDir + "/tracing/instances/surftrace/events/kprobes/" + name + "/enable" + self._echoPath(fPath, 1) + + self._probes.append(name) + + def _initEvents(self, args): + if len(args) < 1: + raise InvalidArgsException("no args.") + for i, arg in enumerate(args): + self.__initEvents(i, arg) + if self._show: + return + + fPath = self.baseDir + "/tracing/instances/surftrace/options/stacktrace" + if not os.path.exists(fPath): + fPath = self.baseDir + "/tracing/options/stacktrace" + if self._stack: + self._events.append(fPath) + self._echoPath(fPath, 1) + else: + self._echoPath(fPath, 0) + + def _checkIsEmpty(self): + if not os.path.exists(self.baseDir + '/tracing/instances/' + 'surftrace'): + os.mkdir(self.baseDir + '/tracing/instances/' + 'surftrace') + return + cmd = 'cat %s/tracing/kprobe_events' % (self.baseDir) + line = self._c.cmd(cmd).strip() + if line != "": + raise FileNotEmptyException("kprobe_events is not empty. should clear other kprobe at first.") + + def _checkAvailable(self, name): + cmd = "cat " + self.baseDir + "/tracing/available_filter_functions |grep " + name + ss = self._c.system(cmd).read().strip() + for res in ss.split('\n'): + if ':' in res: + res = res.split(":", 1)[1] + if ' [' in res: #for ko symbol + res = res.split(" [", 1)[0] + if res == name: + return True + return False + + def _cbLine(self, line): + print("%s" % line) + + def procLine(self, line): + ss = line.split(' ') + o = ' ' + for s in ss: + if '=' in s: + head, value = s.split('=', 1) + if '_' in head: + s = headTrans(head, value) + o += s + ' ' + self._cb(o) + + def start(self): + try: + self._initEvents(self.__args) + except (InvalidArgsException, ExprException, FileNotEmptyException) as e: + self._clearProbes() + del self._parser + print(e.message) + traceback.print_exc() + raise InvalidArgsException("input error") + del self._parser + if not self._show: + super(surftrace, self).start() + if self._cbOrig: + self.pipe.newCb(self._cbOrig) + if self._cb is None: + self._cb = self._cbLine + + def stop(self): + super(surftrace, self).stop() + self._clearProbes() + +def setupParser(mode="remote", remote_ip="pylcc.openanolis.cn", gdb="./gdb", vmlinux=""): + if mode not in ("remote", "local", "gdb"): + raise InvalidArgsException("bad parser mode: %s" % mode) + + if mode == "local": + return CdbParser() + if mode == "remote": + return ClbcClient(server=remote_ip) + elif mode == "gdb": + gdbPath = gdb + return CgdbParser(vmlinuxPath=vmlinux, gdb=gdbPath) + +examples = """examples:""" +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description="Trace ftrace kprobe events.", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=examples + ) + parser.add_argument('-v', '--vmlinux', type=str, dest='vmlinux', default="", help='set vmlinux path.') + parser.add_argument('-m', '--mode', type=str, dest='mode', default="remote", help='set arg parser, fro') + parser.add_argument('-r', '--rip', type=str, dest='rip', default="pylcc.openanolis.cn", help='set remote server ip, remote mode only.') + parser.add_argument('-f', '--file', type=str, dest='file', help='set input args path.') + parser.add_argument('-g', '--gdb', type=str, dest='gdb', default="./gdb", help='set gdb exe file path.') + parser.add_argument('-F', '--func', type=str, dest='func', help='disasassemble function.') + parser.add_argument('-o', '--output', type=str, dest='output', help='set output bash file') + parser.add_argument('-l', '--line', type=str, dest='line', help='get file disasemble info') + parser.add_argument('-a', '--arch', type=str, dest='arch', help='set architecture.') + parser.add_argument('-s', '--stack', action="store_true", help="show call stacks.") + parser.add_argument('-S', '--show', action="store_true", help="only show expressions.") + parser.add_argument(type=str, nargs='*', dest='traces', help='set trace args.') + args = parser.parse_args() + traces = args.traces + + localParser = setupParser(args.mode, args.rip, args.gdb, args.vmlinux) + + if args.line: + localParser.parserLine(args.line) + sys.exit(0) + if args.func: + localParser.disasFun(args.func) + sys.exit(0) + if args.file: + with open(args.file, 'r') as f: + ts = f.readlines() + traces += ts + if args.output: + save2File = True + + arch = "" + if args.arch: + if arch not in ('x86', 'x86_64', 'arm', 'aarch64'): + raise InvalidArgsException('not support architecture %s' % args.arch) + arch = args.arch + if arch.startswith('x86'): + arch = 'x86' + k = surftrace(traces, localParser, args.show, arch, args.stack) + k.start() + if not args.show: + k.loop() + if args.output: + with open(args.output, 'w') as f: + f.write(cmdStrings)