公司的一款產(chǎn)品（linux平臺），最近一段時(shí)間經(jīng)常出現莫名其妙的死機，開(kāi)始懷疑是某個(gè)虛擬設備的驅動(dòng)有問(wèn)題，后來(lái)修改了代碼還是會(huì )死機，再后來(lái)我就寫(xiě)了個(gè)腳本，每隔一個(gè)小時(shí)將系統的各種信息寫(xiě)到日志文件，直到再次死機后分析日志發(fā)現，系統的可用內存從開(kāi)始的900M多逐漸減少，直到最后一次記錄顯示可用內存為100M左右，那么死機是不是由于內存耗盡引起的呢，還不能確定，我決定寫(xiě)個(gè)小程序來(lái)測一下，內核在內存耗盡時(shí)會(huì )是怎樣的狀況。
#include
int main()
{
char *p = NULL;
int count = 1;
while(1){
p = (char *)malloc(1024*1024*10);
if(!p) {
printf("malloc error!\n");
return -1;
}
memset(p, 0, 1024*1024*10);
printf("malloc %dM memory\n",10*count++);
usleep(500000);
}
}
把這段程序分別在兩個(gè)版本的linux平臺上跑，得到的結果卻完全不同:
平臺1： Red Hat Linux release 8.0（2.4.18-14），物理內存 1G,當程序malloc到2890M 時(shí)被系統的oom killer干掉,Out of Memory: Killed process 6448 (loop_malloc)
平臺2： Red Hat Enterprise Linux WS release 3（2.4.21-50.EL），物理內存 1G,當程序malloc到1460M時(shí)，系統死翹翹了，ssh和http都無(wú)法訪(fǎng)問(wèn)，但是可以ping通
而出現死機的產(chǎn)品就是用的平臺2版本的內核，這么說(shuō)來(lái)由于內存耗盡導致死機的可能性越來(lái)越大了，只是不明白為什么oom killer在平臺2上沒(méi)有起作用，是沒(méi)有被調用呢，還是沒(méi)有找到合適的進(jìn)程來(lái)殺？這恐怕要看內核源碼了，鑒于我對內核方面的無(wú)知，只好拜托蘿卜同學(xué)幫忙了。。
從另一方面看，可用內存在短短2天減少了這么多很是有內存泄露的嫌疑，由于原來(lái)腳本收集的信息有限，于是又修改了腳本，增加記錄每個(gè)進(jìn)程的VSZ和RSS，這次發(fā)現果然有一個(gè)進(jìn)程的VSZ和RSS超大，并且在不斷增長(cháng)，用valgrand 測了一下果然是內存泄露，這樣情況就比較明朗了，某進(jìn)程內存泄露->內存耗光->死機，下面把這幾天搜集的資料整理一下。
【關(guān)于/proc/meminfo】
MemTotal: Total usable ram (i.e. physical ram minus a few reserved bits and the kernel binary code)
MemFree: The sum of LowFree+HighFree
Buffers: Relatively temporary storage for raw disk blocks shouldn't get tremendously large (20MB or so)
Cached: in-memory cache for files read from the disk (the pagecache). Doesn't include SwapCached
SwapCached: Memory that once was swapped out, is swapped back in but still also is in the swapfile (if memory is needed it doesn't need to be swapped out AGAIN because it is already in the swapfile. This saves I/O)
Active: Memory that has been used more recently and usually not reclaimed unless absolutely necessary.
Inactive: Memory which has been less recently used. It is more eligible to be reclaimed for other purposes
HighTotal:
HighFree: Highmem is all memory above ~860MB of physical memory Highmem areas are for use by userspace programs, or for the pagecache. The kernel must use tricks to access this memory, making it slower to access than lowmem.
LowTotal:
LowFree: Lowmem is memory which can be used for everything that highmem can be used for, but it is also availble for the kernel's use for its own data structures. Among many other things, it is where everything from the Slab is allocated. Bad things happen when you're out of lowmem.
SwapTotal: total amount of swap space available
SwapFree: Memory which has been evicted from RAM, and is temporarily on the disk
Slab: in-kernel data structures cache
CommitLimit: Based on the overcommit ratio ('vm.overcommit_ratio'),this is the total amount of memory currently available to be allocated on the system. This limit is only adhered to if strict overcommit accounting is enabled (mode 2 in 'vm.overcommit_memory').The CommitLimit is calculated with the following formula: CommitLimit = ('vm.overcommit_ratio' * Physical RAM) + Swap For example, on a system with 1G of physical RAM and 7G of swap with a `vm.overcommit_ratio` of 30 it would yield a CommitLimit of 7.3G. For more details, see the memory overcommit documentation in vm/overcommit-accounting.
Committed_AS: The amount of memory presently allocated on the system. The committed memory is a sum of all of the memory which has been allocated by processes, even if it has not been "used" by them as of yet. A process which malloc()'s 1G of memory, but only touches 300M of it will only show up as using 300M of memory even if it has the address space allocated for the entire 1G. This 1G is memory which has been "committed" to by the VM and can be used at any time by the allocating application. With strict overcommit enabled on the system (mode 2 in 'vm.overcommit_memory'), allocations which would exceed the CommitLimit (detailed above) will not be permitted. This is useful if one needs to guarantee that processes will not fail due to lack of memory once that memory has been successfully allocated.
【關(guān)于free】
在Linux下查看內存我們一般用free命令：
[root@scs-2 tmp]# free
total used free shared buffers cached
Mem: 3266180 3250004 16176 0 110652 2668236
-/+ buffers/cache: 471116 2795064
Swap: 2048276 80160 1968116
區別：第二行(mem)的used/free與第三行(-/+ buffers/cache) used/free的區別。這兩個(gè)的區別在于使用的角度來(lái)看，第一行是從OS的角度來(lái)看，因為對于OS，buffers/cached 都是屬于被使用，所以他的可用內存是16176KB,已用內存是3250004KB,其中包括，內核（OS）使用+Application(X, oracle,etc)使用的+buffers+cached.
第三行所指的是從應用程序角度來(lái)看，對于應用程序來(lái)說(shuō)，buffers/cached 是等于可用的，因為buffer/cached是為了提高文件讀取的性能，當應用程序需在用到內存的時(shí)候，buffer/cached會(huì )很快地被回收。所以從應用程序的角度來(lái)說(shuō)，可用內存=系統free memory+buffers+cached。如上例：2795064=16176+110652+2668236
我們通過(guò)free命令查看機器空閑內存時(shí)，會(huì )發(fā)現free的值很小。這主要是因為，在linux中有這么一種思想，內存不用白不用，因此它盡可能的cache和buffer一些數據，以方便下次使用。但實(shí)際上這些內存也是可以立刻拿來(lái)使用的。所以，空閑內存=free+buffers+cached=total-used
【關(guān)于/proc/ pid/status】
我們可以通過(guò)ps –aux或者top查看某個(gè)進(jìn)程占用的虛擬內存VSZ和物理內存RSS，也可以直接查看/proc/pid/status文件得到這些信息。
VmSize(KB) 任務(wù)虛擬地址空間的大小 (total_vm-reserved_vm)，其中total_vm為進(jìn)程的地址空間的大小，reserved_vm：進(jìn)程在預留或特殊的內存間的物理頁(yè)
VmLck(KB) 任務(wù)已經(jīng)鎖住的物理內存的大小。鎖住的物理內存不能交換到硬盤(pán) (locked_vm)
VmRSS(KB) 應用程序正在使用的物理內存的大小，就是用ps命令的參數rss的值 (rss)
VmData(KB) 程序數據段的大?。ㄋ继摂M內存的大?。?，存放初始化了的數據； (total_vm-shared_vm-stack_vm)
VmStk(KB) 任務(wù)在用戶(hù)態(tài)的棧的大小 (stack_vm)
VmExe(KB) 程序所擁有的可執行虛擬內存的大小，代碼段，不包括任務(wù)使用的庫 (end_code-start_code)
VmLib(KB) 被映像到任務(wù)的虛擬內存空間的庫的大小 (exec_lib)
VmPTE 該進(jìn)程的所有頁(yè)表的大小，單位：kb
【關(guān)于oom killer】
Out-of-Memory (OOM) Killer，就是一層保護機制，用于避免 Linux 在內存不足的時(shí)候不至于出太嚴重的問(wèn)題，把無(wú)關(guān)緊要的進(jìn)程殺掉。
在 32 位CPU 架構下尋址是有限制的。Linux 內核定義了三個(gè)區域：
# DMA: 0x00000000 - 0x00999999 (0 - 16 MB)
# LowMem: 0x01000000 - 0x037999999 (16 - 896 MB) - size: 880MB
# HighMem: 0x038000000 - <硬件特定>
什么時(shí)候會(huì )觸發(fā)oom killer？根據我的搜查，大概就兩種情況：
1 當 low memory 被耗盡的時(shí)候，即使high memory還有很大的空閑內存
2 low memory里都是碎片，請求不到連續的內存區域
通常的問(wèn)題是high memory很大仍然會(huì )觸發(fā)oom killer，或者由于碎片觸發(fā)oom killer，解決辦法：
1、升級至64位的 Linux 版本，這是最好的解決方案。
2、如果是32位的 Linux 版本，最好的解決辦法就是應用 hugemem kernel，還有一個(gè)解決方法設置 /proc/sys/vm/lower_zone_protection 值為250或更高。
# echo "250" > /proc/sys/vm/lower_zone_protection
設置成啟動(dòng)加載，在 /etc/sysctl.conf 中加入
vm.lower_zone_protection = 250
3、最無(wú)力的解決辦法，就是禁用 oom-killer ，這可能會(huì )導致系統掛起，所以要慎重使用。
使 oom-killer 關(guān)閉/開(kāi)啟:
# echo "0" > /proc/sys/vm/oom-kill
# echo "1" > /proc/sys/vm/oom-kill
使設置啟動(dòng)時(shí)生效，需要在 /etc/sysctl.conf 中加入
vm.oom-kill = 0
而我的問(wèn)題是oom killer在該被觸發(fā)的時(shí)候沒(méi)有被觸發(fā)，所以這些方法對我沒(méi)用-_-|
【關(guān)于overcommit_memory】
The Linux kernel supports the following overcommit handling modes
0 - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slighly more memory in this mode. This is the default.
1 - Always overcommit. Appropriate for some scientific applications.
2 - Don't overcommit. The total address space commit for the system is not permitted to exceed swap + a configurable percentage (default is 50) of physical RAM. Depending on the percentage you use, in most situations this means a process will not be killed while accessing pages but will receive errors on memory allocation as appropriate.
The overcommit policy is set via the sysctl `vm.overcommit_memory'.
The overcommit percentage is set via `vm.overcommit_ratio'.
The current overcommit limit and amount committed are viewable in
/proc/meminfo as CommitLimit and Committed_AS respectively.
------------------------------------------------------------------------------------------
#echo 2>/proc/sys/vm/overcommit_memory
#echo 0>/proc/sys/vm/overcommit_ratio
-------------------------------------------------------------------------------------------
實(shí)際測試：
overcommit_memory ==2 ，物理內存使用完后，打開(kāi)任意一個(gè)程序均顯示“內存不足”；
overcommit_memory ==1，會(huì )從buffer中釋放較多物理內存，適合大型科學(xué)應用軟件，但oom-killer機制仍然起作用；
overcommit_memory ==0，系統默認設置，釋放物理內存較少，使得oom-killer機制運作很明顯。
本站僅提供存儲服務(wù)，所有內容均由用戶(hù)發(fā)布，如發(fā)現有害或侵權內容，請
點(diǎn)擊舉報。
