文章总结: 文章讲述了作者使用逆向工程方法解决Golang程序CPU占用过高问题的过程。通过IDA、go_parser、perf和strace等工具,作者定位到问题出现在netlink.LinkByIndex函数的无限调用,原因是通道被关闭后select语句立即返回导致循环。文章展示了在无调试信息情况下如何通过逆向分析定位性能问题的实用技巧,提供了具体的工具使用方法和代码示例,对类似问题的解决有很好的参考价值。
综合评分: 91
文章分类: 逆向分析,二进制安全,安全工具,应用安全,其他
记录工作中解决bug用到的逆向知识
hexdeep
看雪学苑
2025年11月4日 17:59
上海
问题现象
在使用rk3588开发云手机过程中,有部分核心板随机出现hostserver进程占用cpu过高问题。
因为golang编译时候去除了调试信息,没有加上pprof,无法看到goroutine的工作状态,同时hostserver进程为服务进程。
kill -QUIT $(pidof host_server)
无法使用上面命令打印堆栈
同时环境不容易复现,无法通过反复修改代码来验证问题,只能在不破坏环境的情况下,尽可能多收集信息,然后定位问题。
分析过程
分析工具
◆ida7.5
◆https://github.com/0xjiayu/go_parser
◆strace
◆perf
环境配置
打开ida,导入goparser
太清晰了,简直就是明文[呲牙],以后用golang写程序要上加壳了
perf record -F 99 -p $(pidof host_server) -g -- sleep 30
如果出现以上错误执行以下命令:
echo 1 > /proc/sys/kernel/kptr_restrict
运行查看结果
perf report
Samples: 1K of event 'cycles:P', Event count (approx.): 24575912110
Children Self Command Shared Object Symbol
+ 50.49% 0.00% host_server host_server [.] 0x0000000000086ba4
+ 42.66% 0.00% host_server host_server [.] 0x00000000004309f0
+ 42.25% 0.00% host_server host_server [.] 0x00000000003ab5ac
+ 42.07% 0.00% host_server host_server [.] 0x00000000003ab5f4
+ 41.87% 0.00% host_server host_server [.] 0x00000000003a5f2c
+ 37.75% 0.00% host_server [kernel.kallsyms] [k] el0t_64_sync
+ 37.75% 0.00% host_server [kernel.kallsyms] [k] el0t_64_sync_handler
+ 36.42% 0.00% host_server host_server [.] 0x000000000008485c
+ 30.67% 0.00% host_server [kernel.kallsyms] [k] el0_svc
+ 27.78% 0.00% host_server [kernel.kallsyms] [k] do_el0_svc
+ 27.72% 1.95% host_server [kernel.kallsyms] [k] el0_svc_common.constprop.0
+ 25.63% 0.06% host_server [kernel.kallsyms] [k] invoke_syscall
+ 25.23% 0.00% host_server host_server [.] 0x00000000000190a0
+ 17.42% 0.00% host_server host_server [.] 0x00000000000ca99b
+ 15.47% 0.00% host_server host_server [.] 0x000000000002d914
+ 15.28% 0.00% host_server host_server [.] 0x00000000003a63c8
+ 14.77% 0.00% host_server host_server [.] 0x0000000000031884
+ 11.29% 0.00% host_server host_server [.] 0x00000000000548c4
+ 11.11% 0.00% host_server host_server [.] 0x00000000003a6248
+ 11.01% 0.00% host_server host_server [.] 0x000000000002445c
+ 10.55% 0.00% host_server host_server [.] 0x00000000000847c4
+ 9.54% 0.00% host_server host_server [.] 0x00000000003a765c
+ 9.54% 0.00% host_server host_server [.] 0x000000000007ce70
+ 9.46% 0.00% host_server host_server [.] 0x0000000000054dc0
+ 8.59% 0.00% host_server host_server [.] 0x0000000000024018
+ 8.53% 0.00% host_server host_server [.] 0x00000000000243d3
+ 8.50% 0.00% host_server host_server [.] 0x00000000003a62fc
+ 8.23% 0.00% host_server host_server [.] 0x00000000000cb4fc
+ 7.97% 0.00% host_server host_server [.] 0x000000000002d8e8
+ 7.70% 0.00% host_server host_server [.] 0x00000000000caa7c
+ 7.52% 0.00% host_server host_server [.] 0x0000000000087430
+ 7.01% 0.18% host_server [kernel.kallsyms] [k] el0_da
+ 6.89% 0.00% host_server host_server [.] 0x0000000000031ae4
+ 6.73% 0.07% host_server [kernel.kallsyms] [k] do_mem_abort
+ 6.72% 0.00% host_server host_server [.] 0x00000000003a741c
+ 6.72% 0.00% host_server host_server [.] 0x00000000000ee794
+ 6.71% 0.00% host_server host_server [.] 0x000000000002aff8
+ 6.66% 0.10% host_server [kernel.kallsyms] [k] do_translation_fault
+ 6.50% 0.00% host_server host_server [.] 0x00000000000e54d4
+ 6.50% 0.00% host_server host_server [.] 0x00000000003a752c
+ 6.46% 0.08% host_server [kernel.kallsyms] [k] do_page_fault
+ 6.44% 0.00% host_server host_server [.] 0x0000000000163d94
+ 6.44% 0.00% host_server host_server [.] 0x000000000016522c
+ 6.07% 0.07% host_server [kernel.kallsyms] [k] __arm64_sys_sendto
+ 6.00% 0.00% host_server [kernel.kallsyms] [k] __sys_sendto
现在我们已经知道消耗cpu的代码地址,0x0000000000086ba4 0x00000000004309f0 0x00000000003ab5ac
0x0000000000086ba4 代码:
.text:0000000000086BA0 ; =============== S U B R O U T I N E =======================================
.text:0000000000086BA0
.text:0000000000086BA0 ; Attributes: noreturn
.text:0000000000086BA0
.text:0000000000086BA0 runtime_goexit ; DATA XREF: runtime_oneNewExtraM+38↑o
.text:0000000000086BA0 ; runtime_newproc1+E0↑o
.text:0000000000086BA0 MOV X0, X0
.text:0000000000086BA4
.text:0000000000086BA4 loc_86BA4
.text:0000000000086BA4 BL runtime_goexit1_0
0x00000000004309f0 代码:
.text:00000000004309D4 SUB X29, SP, #0x18
.text:00000000004309D8 BL sub_87260
.text:00000000004309DC
.text:00000000004309DC loc_4309DC ; DATA XREF: host_server_netlink__Netlink_Init_func1+10C↑o
.text:00000000004309DC SUB X29, SP, #8
.text:00000000004309E0 LDR X1, [SP,#0x250+var_150]
.text:00000000004309E4 ADRP X27, #off_AD3828@PAGE
.text:00000000004309E8 LDR X0, [X27,#off_AD3828@PAGEOFF]
.text:00000000004309EC BL github_com_vishvananda_netlink__Handle_LinkByIndex
.text:00000000004309F0 CBNZ X2, loc_4309FC
.text:00000000004309F4 MOV X4, XZR
0x00000000003ab5ac 代码:
.text:00000000003AB59C STR X2, [X25,#8]
.text:00000000003AB5A0
.text:00000000003AB5A0 loc_3AB5A0 ; CODE XREF: github_com_vishvananda_netlink__Handle_LinkByIndex+29C↑j
.text:00000000003AB5A0 STR X0, [X1,#8]
.text:00000000003AB5A4 MOV X0, X5
.text:00000000003AB5A8 BL github_com_vishvananda_netlink_execGetLink
.text:00000000003AB5AC LDP X29, X30, [SP,#0x120+var_128]
.text:00000000003AB5B0 ADD SP, SP, #0x120
从以上片段可以得出以下结论
◆0x0000000000086ba4
runtime_goexit1 是 Go runtime 在销毁 goroutine 时的内部函数,说明有大量的goroutine创建和销毁
◆0x00000000004309f0
问题应该出在github_com_vishvananda_netlink__Handle_LinkByIndex这个函数
验证
找到我们自己的代码
case addrUpdate := <-addrCh:
link, err = netlink.LinkByIndex(addrUpdate.LinkIndex)
应该是上面这行代码导致的
用strace工具再次验证下
strace -p $(pidof host_server) -f
满屏的输出
[pid 20362] <... epoll_ctl resumed>) = 0
[pid 575] <... epoll_pwait resumed>[], 128, 0, NULL, 0) = 0
[pid 572] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 20362] close(3) = 0
[pid 575] epoll_pwait(4, <unfinished ...>
[pid 20362] socket(AF_NETLINK, SOCK_RAW|SOCK_CLOEXEC, NETLINK_ROUTE) = 3
[pid 20362] fcntl(3, F_GETFL) = 0x2 (flags O_RDWR)
[pid 20362] fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid 20362] fcntl(3, F_GETFL <unfinished ...>
[pid 572] <... nanosleep resumed>NULL) = 0
[pid 20362] <... fcntl resumed>) = 0x802 (flags O_RDWR|O_NONBLOCK)
[pid 572] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 20362] epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data=0x7f6eb111fec2ad}) = 0
[pid 20362] bind(3, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12 <unfinished ...>
[pid 575] <... epoll_pwait resumed>[{events=EPOLLOUT, data=0x7f6eb111fec2ad}], 128, 780, NULL, 0) = 1
[pid 572] <... nanosleep resumed>NULL) = 0
[pid 20362] <... bind resumed>) = 0
[pid 572] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 20362] sendto(3, [{nlmsg_len=40, nlmsg_type=RTM_GETLINK, nlmsg_flags=NLM_F_REQUEST|NLM_F_ACK, nlmsg_seq=3681469127, nlmsg_pid=0}, {ifi_family=AF_UNSPEC, ifi_type=ARPHRD_NETROM, ifi_index=0, ifi_flags=0, ifi_change=0}, [{nla_len=8, nla_type=IFLA_EXT_MASK}, RTEXT_FILTER_VF]], 40, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12 <unfinished ...>
[pid 575] epoll_pwait(4, <unfinished ...>
[pid 20362] <... sendto resumed>) = 40
[pid 572] <... nanosleep resumed>NULL) = 0
[pid 20362] getsockname(3, <unfinished ...>
[pid 575] <... epoll_pwait resumed>[{events=EPOLLIN|EPOLLOUT, data=0x7f6eb111fec2ad}], 128, 0, NULL, 0) = 1
[pid 20362] <... getsockname resumed>{sa_family=AF_NETLINK, nl_pid=-1395142807, nl_groups=00000000}, [112 => 12]) = 0
[pid 572] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 575] epoll_pwait(4, <unfinished ...>
[pid 20362] recvfrom(3, [{nlmsg_len=60, nlmsg_type=NLMSG_ERROR, nlmsg_flags=0, nlmsg_seq=3681469127, nlmsg_pid=-1395142807}, {error=-EINVAL, msg=[{nlmsg_len=40, nlmsg_type=RTM_GETLINK, nlmsg_flags=NLM_F_REQUEST|NLM_F_ACK, nlmsg_seq=3681469127, nlmsg_pid=0}, {ifi_family=AF_UNSPEC, ifi_type=ARPHRD_NETROM, ifi_index=0, ifi_flags=0, ifi_change=0}, [{nla_len=8, nla_type=IFLA_EXT_MASK}, RTEXT_FILTER_VF]]}], 65536, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, [112 => 12]) = 60
[pid 572] <... nanosleep resumed>NULL) = 0
[pid 20362] epoll_ctl(4, EPOLL_CTL_DEL, 3, 0x40002a4850 <unfinished ...>
[pid 572] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 20362] <... epoll_ctl resumed>) = 0
[pid 20362] close(3) = 0
[pid 20362] socket(AF_NETLINK, SOCK_RAW|SOCK_CLOEXEC, NETLINK_ROUTE <unfinished ...>
[pid 572] <... nanosleep resumed>NULL) = 0
[pid 20362] <... socket resumed>) = 3
[pid 572] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 20362] fcntl(3, F_GETFL) = 0x2 (flags O_RDWR)
[pid 20362] fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid 20362] fcntl(3, F_GETFL <unfinished ...>
[pid 572] <... nanosleep resumed>NULL) = 0
[pid 20362] <... fcntl resumed>) = 0x802 (flags O_RDWR|O_NONBLOCK)
[pid 572] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 20362] epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data=0x7f6eb111fec2ae}) = 0
[pid 575] <... epoll_pwait resumed>[{events=EPOLLOUT, data=0x7f6eb111fec2ae}], 128, 779, NULL, 0) = 1
[pid 20362] bind(3, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12 <unfinished ...>
[pid 572] <... nanosleep resumed>NULL) = 0
[pid 20362] <... bind resumed>) = 0
[pid 575] epoll_pwait(4, <unfinished ...>
[pid 572] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 20362] sendto(3, [{nlmsg_len=40, nlmsg_type=RTM_GETLINK, nlmsg_flags=NLM_F_REQUEST|NLM_F_ACK, nlmsg_seq=3681469128, nlmsg_pid=0}, {ifi_family=AF_UNSPEC, ifi_type=ARPHRD_NETROM, ifi_index=0, ifi_flags=0, ifi_change=0}, [{nla_len=8, nla_type=IFLA_EXT_MASK}, RTEXT_FILTER_VF]], 40, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12 <unfinished ...>
[pid 575] <... epoll_pwait resumed>[], 128, 0, NULL, 0) = 0
[pid 572] <... nanosleep resumed>NULL) = 0
[pid 20362] <... sendto resumed>) = 40
[pid 575] epoll_pwait(4, <unfinished ...>
[pid 20362] getsockname(3, <unfinished ...>
[pid 572] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 20362] <... getsockname resumed>{sa_family=AF_NETLINK, nl_pid=-65674427, nl_groups=00000000}, [112 => 12]) = 0
[pid 575] <... epoll_pwait resumed>[{events=EPOLLIN|EPOLLOUT, data=0x7f6eb111fec2ae}], 128, 778, NULL, 0) = 1
[pid 572] <... nanosleep resumed>NULL) = 0
[pid 20362] recvfrom(3, <unfinished ...>
[pid 572] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 20362] <... recvfrom resumed>[{nlmsg_len=60, nlmsg_type=NLMSG_ERROR, nlmsg_flags=0, nlmsg_seq=3681469128, nlmsg_pid=-65674427}, {error=-EINVAL, msg=[{nlmsg_len=40, nlmsg_type=RTM_GETLINK, nlmsg_flags=NLM_F_REQUEST|NLM_F_ACK, nlmsg_seq=3681469128, nlmsg_pid=0}, {ifi_family=AF_UNSPEC, ifi_type=ARPHRD_NETROM, ifi_index=0, ifi_flags=0, ifi_change=0}, [{nla_len=8, nla_type=IFLA_EXT_MASK}, RTEXT_FILTER_VF]]}], 65536, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, [112 => 12]) = 60
[pid 575] epoll_pwait(4, <unfinished ...>
[pid 20362] epoll_ctl(4, EPOLL_CTL_DEL, 3, 0x40002a4850 <unfinished ...>
[pid 572] <... nanosleep resumed>NULL) = 0
[pid 20362] <... epoll_ctl resumed>) = 0
[pid 575] <... epoll_pwait resumed>[], 128, 0, NULL, 0) = 0
[pid 20362] close(3 <unfinished ...>
[pid 572] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 20362] <... close resumed>) = 0
[pid 575] epoll_pwait(4, <unfinished ...>
[pid 20362] socket(AF_NETLINK, SOCK_RAW|SOCK_CLOEXEC, NETLINK_ROUTE) = 3
基本可以肯定是上面这行代码导致的
case addrUpdate := <-addrCh:
link, err = netlink.LinkByIndex(addrUpdate.LinkIndex)
#
结论
addrCh应该在什么场景下被close了,golang select 一个close的channel会导致立即返回,导致netlink.LinkByIndex(addrUpdate.LinkIndex)被无限调用,知道原因就好办了。
#
看雪ID:hexdeep
https://bbs.kanxue.com/user-home-974190.htm
*本文为看雪论坛优秀文章,由 hexdeep 原创,转载请注明来自看雪社区
往期推荐
静态程序分析之数据流分析(Foundations + LiveVar Analysis Code)续
基于Minifilter实现目录保护软件,自定义保护目录,用户可选择是否允许文件行为
球分享
球点赞
球在看
点击阅读原文查看更多
免责声明:
本文所载程序、技术方法仅面向合法合规的安全研究与教学场景,旨在提升网络安全防护能力,具有明确的技术研究属性。
任何单位或个人未经授权,将本文内容用于攻击、破坏等非法用途的,由此引发的全部法律责任、民事赔偿及连带责任,均由行为人独立承担,本站不承担任何连带责任。
本站内容均为技术交流与知识分享目的发布,若存在版权侵权或其他异议,请通过邮件联系处理,具体联系方式可点击页面上方的联系我。
本文转载自:看雪学苑 hexdeep《记录工作中解决bug用到的逆向知识》