内存泄漏排查工具-memleak

memleak

memleak是BCC工具中的内存分析工具，它用来跟踪(trace)内存和释放请求，并且收集进程的调用堆栈，并展示进程中每个函数的调用情况。

实践

这里使用《Linux 性能优化实战》中倪朋飞老师的例子来对memleak进行一个简单上手展示:

1
$ docker run --name=app -itd feisky/app:mem-leak

这是一个计算斐波那契数列的进程。之后运行起来可以看到计算结果:

1
2
3
4
5
6
7
$ docker logs app
2th => 1
3th => 2
4th => 3
5th => 5
6th => 8
7th => 13

之后通过vmstat来观察内存变化:

1
2
3
4
5
6
7
8
9
10
11
12
# 每隔3秒输出一组数据
$ vmstat 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
0  0      0 6601824  97620 1098784    0    0     0     0   62  322  0  0 100  0  0
0  0      0 6601700  97620 1098788    0    0     0     0   57  251  0  0 100  0  0
0  0      0 6601320  97620 1098788    0    0     0     3   52  306  0  0 100  0  0
0  0      0 6601452  97628 1098788    0    0     0    27   63  326  0  0 100  0  0
2  0      0 6601328  97628 1098788    0    0     0    44   52  299  0  0 100  0  0
0  0      0 6601080  97628 1098792    0    0     0     0   56  285  0  0 100  0  0

free在不断下降，但是buffer和cache没有发生变化，说明系统中使用内存一直在升高，但是目前还是无法确定是否为内存泄露问题。

这个时候可以借助memleak来查看是否发生泄漏:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
/usr/share/bcc/tools/memleak -p $(pidof app)

Top 10 stacks with outstanding allocations:
       addr = 7fe4b8193bc0 size = 8192
       addr = 7fe4b81afca0 size = 8192
       addr = 7fe4b822a070 size = 8192
       addr = 7fe4b81b9cf0 size = 8192
       addr = 7fe4b822e090 size = 8192
       addr = 7fe4b81d9df0 size = 8192
       addr = 7fe4b820df90 size = 8192
       addr = 7fe4b8189b70 size = 8192
       addr = 7fe4b81fdf10 size = 8192
       addr = 7fe4b81cfda0 size = 8192
       addr = 7fe4b81ede90 size = 8192
       addr = 7fe4b81a9c70 size = 8192
       addr = 7fe4b81b3cc0 size = 8192
       addr = 7fe4b818fba0 size = 8192
       addr = 7fe4b81c1d30 size = 8192
       addr = 7fe4b81adc90 size = 8192
       addr = 7fe4b81e7e60 size = 8192
       addr = 7fe4b81bdd10 size = 8192
       addr = 7fe4b81c7d60 size = 8192
       addr = 7fe4b81dbe00 size = 8192
       addr = 7fe4b81cdd90 size = 8192
       addr = 7fe4b8222030 size = 8192
       addr = 7fe4b81d3dc0 size = 8192
       addr = 7fe4b8213fc0 size = 8192
       addr = 7fe4b81f9ef0 size = 8192
       addr = 7fe4b8199bf0 size = 8192
       addr = 7fe4b818bb80 size = 8192
       addr = 7fe4b8224040 size = 8192
       addr = 7fe4b81d1db0 size = 8192
       addr = 7fe4b8207f60 size = 8192
       addr = 7fe4b81efea0 size = 8192
       addr = 7fe4b8220020 size = 8192
       addr = 7fe4b81f3ec0 size = 8192
       addr = 7fe4b81dde10 size = 8192
       addr = 7fe4b8217fe0 size = 8192
       addr = 7fe4b820ffa0 size = 8192
       addr = 7fe4b8187b60 size = 8192
       addr = 7fe4b81b5cd0 size = 8192
       addr = 7fe4b81f5ed0 size = 8192
       addr = 7fe4b8201f30 size = 8192
       addr = 7fe4b81b1cb0 size = 8192
       addr = 7fe4b82300a0 size = 8192
       addr = 7fe4b8211fb0 size = 8192
       addr = 7fe4b81fff20 size = 8192
       addr = 7fe4b81cbd80 size = 8192
       addr = 7fe4b81d5dd0 size = 8192
       addr = 7fe4b8191bb0 size = 8192
       addr = 7fe4b822c080 size = 8192
       addr = 7fe4b81f7ee0 size = 8192
       addr = 7fe4b81c3d40 size = 8192
       addr = 7fe4b8209f70 size = 8192
       addr = 7fe4b81c9d70 size = 8192
       addr = 7fe4b81e1e30 size = 8192
       addr = 7fe4b81a3c40 size = 8192
       addr = 7fe4b81b7ce0 size = 8192
       addr = 7fe4b821e010 size = 8192
       addr = 7fe4b81abc80 size = 8192
       addr = 7fe4b81a5c50 size = 8192
       addr = 7fe4b8219ff0 size = 8192
       addr = 7fe4b81e3e40 size = 8192
       addr = 7fe4b818db90 size = 8192
       addr = 7fe4b81ebe80 size = 8192
       addr = 7fe4b8226050 size = 8192
       addr = 7fe4b8228060 size = 8192
       addr = 7fe4b821c000 size = 8192
       addr = 7fe4b8197be0 size = 8192
       addr = 7fe4b819bc00 size = 8192
       addr = 7fe4b81a1c30 size = 8192
       addr = 7fe4b81dfe20 size = 8192
       addr = 7fe4b81fbf00 size = 8192
       addr = 7fe4b81bbd00 size = 8192
       addr = 7fe4b81a7c60 size = 8192
       addr = 7fe4b81bfd20 size = 8192
       addr = 7fe4b8195bd0 size = 8192
       addr = 7fe4b81c5d50 size = 8192
       addr = 7fe4b8215fd0 size = 8192
       addr = 7fe4b81e5e50 size = 8192
       addr = 7fe4b81e9e70 size = 8192
       addr = 7fe4b8203f40 size = 8192
       addr = 7fe4b819fc20 size = 8192
       addr = 7fe4b81d7de0 size = 8192
       addr = 7fe4b8205f50 size = 8192
       addr = 7fe4b819dc10 size = 8192
       addr = 7fe4b820bf80 size = 8192
       addr = 7fe4b81f1eb0 size = 8192
       696320 bytes in 85 allocations from stack
               0x0000560700f44879      fibonacci+0x1f [app]
               0x0000560700f448ea      child+0x4f [app]
               0x00007fe4c09966db      start_thread+0xdb [libpthread-2.27.so]

可以看到这里再大量的重复分配内存，并且堆栈内存增长不断，根据最后结果可以发现是child和fibonacci问题，所以需要到代码层面去查看这里的问题:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
long long *fibonacci(long long *n0, long long *n1)
{
    //分配1024个长整数空间方便观测内存的变化情况
    long long *v = (long long *) calloc(1024, sizeof(long long));
    *v = *n0 + *n1;
    return v;
}


void *child(void *arg)
{
    long long n0 = 0;
    long long n1 = 1;
    long long *v = NULL;
    for (int n = 2; n > 0; n++) {
        v = fibonacci(&n0, &n1);
        n0 = n1;
        n1 = *v;
        printf("%dth => %lld\n", n, *v);
        sleep(1);
    }
}

可以看到再child函数中调用了fibonacci，fibonacci中使用了calloc进行了内存分配，但是并没有使用free进行回收，所以会造成内存泄漏；解决这个问题的方法就是在child函数中使用free函数就可以了。

1
2
3
4
5
6
7
8
9
10
11
12
void *child(void *arg)
{
    ...
    for (int n = 2; n > 0; n++) {
        v = fibonacci(&n0, &n1);
        n0 = n1;
        n1 = *v;
        printf("%dth => %lld\n", n, *v);
        free(v);    // 释放内存
        sleep(1);
    }
}

再来查看一下结果:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
[11:35:06] Top 10 stacks with outstanding allocations:
[11:35:11] Top 10 stacks with outstanding allocations:
[11:35:16] Top 10 stacks with outstanding allocations:
[11:35:21] Top 10 stacks with outstanding allocations:
[11:35:26] Top 10 stacks with outstanding allocations:
[11:35:31] Top 10 stacks with outstanding allocations:
[11:35:36] Top 10 stacks with outstanding allocations:
[11:35:41] Top 10 stacks with outstanding allocations:
[11:35:46] Top 10 stacks with outstanding allocations:
[11:35:51] Top 10 stacks with outstanding allocations:
[11:35:56] Top 10 stacks with outstanding allocations:
[11:36:01] Top 10 stacks with outstanding allocations:
[11:36:06] Top 10 stacks with outstanding allocations:
[11:36:11] Top 10 stacks with outstanding allocations:
[11:36:16] Top 10 stacks with outstanding allocations:
[11:36:21] Top 10 stacks with outstanding allocations:
[11:36:26] Top 10 stacks with outstanding allocations:
[11:36:31] Top 10 stacks with outstanding allocations:
[11:36:36] Top 10 stacks with outstanding allocations:
[11:36:41] Top 10 stacks with outstanding allocations:
[11:36:46] Top 10 stacks with outstanding allocations:
[11:36:51] Top 10 stacks with outstanding allocations:
[11:36:56] Top 10 stacks with outstanding allocations:
[11:37:01] Top 10 stacks with outstanding allocations:
[11:37:06] Top 10 stacks with outstanding allocations:
[11:37:11] Top 10 stacks with outstanding allocations:
[11:37:16] Top 10 stacks with outstanding allocations:
[11:37:21] Top 10 stacks with outstanding allocations:
[11:37:26] Top 10 stacks with outstanding allocations:
[11:37:31] Top 10 stacks with outstanding allocations:
[11:37:36] Top 10 stacks with outstanding allocations:
[11:37:41] Top 10 stacks with outstanding allocations:
[11:37:46] Top 10 stacks with outstanding allocations:
[11:37:51] Top 10 stacks with outstanding allocations:
[11:37:56] Top 10 stacks with outstanding allocations:
[11:38:01] Top 10 stacks with outstanding allocations:
[11:38:06] Top 10 stacks with outstanding allocations:
[11:38:11] Top 10 stacks with outstanding allocations:
[11:38:16] Top 10 stacks with outstanding allocations:
[11:38:21] Top 10 stacks with outstanding allocations:
[11:38:26] Top 10 stacks with outstanding allocations:
[11:38:31] Top 10 stacks with outstanding allocations:
[11:38:36] Top 10 stacks with outstanding allocations:
[11:38:41] Top 10 stacks with outstanding allocations:
[11:38:46] Top 10 stacks with outstanding allocations:
[11:38:51] Top 10 stacks with outstanding allocations:
[11:38:56] Top 10 stacks with outstanding allocations:
[11:39:01] Top 10 stacks with outstanding allocations:
[11:39:06] Top 10 stacks with outstanding allocations:
[11:39:11] Top 10 stacks with outstanding allocations:
[11:39:16] Top 10 stacks with outstanding allocations:
[11:39:21] Top 10 stacks with outstanding allocations:
[11:39:26] Top 10 stacks with outstanding allocations:
[11:39:31] Top 10 stacks with outstanding allocations:
[11:39:36] Top 10 stacks with outstanding allocations:
[11:39:41] Top 10 stacks with outstanding allocations:
[11:39:46] Top 10 stacks with outstanding allocations:
[11:39:51] Top 10 stacks with outstanding allocations:
[11:39:56] Top 10 stacks with outstanding allocations:
[11:40:01] Top 10 stacks with outstanding allocations:
[11:40:06] Top 10 stacks with outstanding allocations:
[11:40:11] Top 10 stacks with outstanding allocations:
[11:40:16] Top 10 stacks with outstanding allocations:
[11:40:21] Top 10 stacks with outstanding allocations:
[11:40:26] Top 10 stacks with outstanding allocations:
[11:40:31] Top 10 stacks with outstanding allocations:
[11:40:36] Top 10 stacks with outstanding allocations:
[11:40:41] Top 10 stacks with outstanding allocations:
[11:40:46] Top 10 stacks with outstanding allocations:
[11:40:51] Top 10 stacks with outstanding allocations:
[11:40:56] Top 10 stacks with outstanding allocations:
[11:41:01] Top 10 stacks with outstanding allocations:
[11:41:06] Top 10 stacks with outstanding allocations:
[11:41:11] Top 10 stacks with outstanding allocations:
[11:41:16] Top 10 stacks with outstanding allocations:
[11:41:21] Top 10 stacks with outstanding allocations:
[11:41:26] Top 10 stacks with outstanding allocations:
[11:41:31] Top 10 stacks with outstanding allocations:

可以看到在outstanding allocations中，并没出现一些尚未释放的内存分配等相关信息。

#eBPF
内存泄漏排查工具-memleak
http://example.com/2024/02/05/内存泄漏排查工具-memleak/
Author
John Doe
Posted on
February 5, 2024
Licensed under