ceph記憶體剖析工具--pprof

一、參考文檔路徑：

http://docs.ceph.org.cn/rados/troubleshooting/memory-profiling/

https://gperftools.github.io/gperftools/heapprofile.html

二、google-perftools工具下載下傳路徑：

https://sourceforge.net/projects/goog-perftools/

gperftools-master.zip <見百度雲常用工具>

三、工具使用方法

gperftools适合發現運作過程中記憶體差異，提供洩露的函數範圍。而且程序響影響小，适合高壓力快速複現。

詳細使用方法請參考http://docs.ceph.org.cn/rados/troubleshooting/memory-profiling/，下面僅對該文檔做一些補充說明

1、單獨檢視一個記憶體堆分析檔案

pprof --text /usr/bin/ceph-osd /var/log/ceph/osd.25.profile.0003.heap

2、比較兩個記憶體堆分析檔案

pprof --text --base /var/log/ceph/osd.34.profile.0011.heap /usr/bin/ceph-osd /var/log/ceph/osd.34.profile.0029.heap

3、輸出結果說明

[[email protected] gperftools-master]# pprof --text /usr/bin/ceph-osd /var/log/ceph/osd.34.profile.0011.heap |less

Using local file /usr/bin/ceph-osd.

Using local file /var/log/ceph/osd.34.profile.0011.heap.

Total: 3.4 MB

3.3 94.8% 94.8% 3.3 94.8% ceph::logging::Log::create_entry

0.1 1.7% 96.6% 0.1 1.7% AsyncConnection::AsyncConnection

0.0 0.7% 97.2% 0.0 0.7% ceph::buffer::list::[email protected]

0.0 0.6% 97.8% 0.0 0.6% ceph::buffer::create_aligned

0.0 0.5% 98.3% 0.1 1.5% AsyncMessenger::add_accept

0.0 0.4% 98.7% 0.0 0.4% EventCenter::create_time_event

0.0 0.3% 99.0% 0.0 1.0% AsyncMessenger::create_connect

0.0 0.3% 99.2% 2.9 83.2% EventCenter::process_events

0.0 0.1% 99.4% 0.0 0.1% ceph::buffer::list::[email protected]

0.0 0.1% 99.5% 0.5 15.3% OSD::heartbeat

0.0 0.1% 99.6% 0.0 0.1% decode_message

0.0 0.1% 99.7% 0.0 0.1% std::basic_string::_Rep::_S_create

0.0 0.0% 99.7% 0.0 0.0% std::_Rb_tree::_M_emplace_hint_unique

0.0 0.0% 99.7% 0.0 0.1% OSD::ms_verify_authorizer

0.0 0.0% 99.8% 0.0 0.0% std::_Rb_tree::_M_copy

0.0 0.0% 99.8% 0.0 0.0% std::_Rb_tree::M_insert

0.0 0.0% 99.8% 1.0 27.9% AsyncConnection::send_message

0.0 0.0% 99.8% 0.0 0.0% PORT_Alloc_Util

0.0 0.0% 99.9% 0.0 0.0% CryptoAES::get_key_handler

0.0 0.0% 99.9% 0.0 0.0% std::vector::_M_emplace_back_aux

0.0 0.0% 99.9% 0.0 0.0% ceph::buffer::create_static

0.0 0.0% 99.9% 0.0 0.0% ceph::buffer::list::[email protected]

0.0 0.0% 99.9% 0.0 0.0% ceph::buffer::list::[email protected]

Total：後面顯示總共申請的記憶體

第一列值顯示的是該函數内，不包括調用的其他函數，總共申請的堆記憶體

第四列值顯示的是該函數内，包括自己申請的堆記憶體，和調用外部函數申請的堆記憶體

第二列和第五列，是第一列和第四例相對于Total的比值

第六列顯示具體申請堆記憶體的函數

4、堆記憶體釋放方法

a、首先使用指令：ceph tell {daemon-type}.{daemon-id} heap stats 檢視是否有未使用的堆記憶體。（注意這裡的daemon-type可以是mon、osd、mds，daemon-id是對應服務編号）

舉例：

[[email protected] bn_cli]# ceph tell osd.4 heap stats
osd.4 tcmalloc heap stats:------------------------------------------------
MALLOC:       21704640 (   20.7 MiB) Bytes in use by application
MALLOC: +            0 (    0.0 MiB) Bytes in page heap freelist
MALLOC: +      1406560 (    1.3 MiB) Bytes in central cache freelist
MALLOC: +      1310656 (    1.2 MiB) Bytes in transfer cache freelist
MALLOC: +      3316256 (    3.2 MiB) Bytes in thread cache freelists
MALLOC: +      1175712 (    1.1 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =     28913824 (   27.6 MiB) Actual memory used (physical + swap)
MALLOC: +       573440 (    0.5 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =     29487264 (   28.1 MiB) Virtual address space used
MALLOC:
MALLOC:           1039              Spans in use
MALLOC:             36              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
           

檢視是否有：Bytes in page heap freelist （未釋放的堆記憶體）

b、如果Bytes in page heap freelist 不為0，則執行如下指令，可以釋放掉未使用的堆記憶體

ceph tell osd.4 heap release