Debian Linux 8, Jessie, Managing and Troubleshooting Memory Consumption

I ran into some issues running Debian 8.6 on Hyper-V recently – my VM would run out of memory, run through a series of activities kill processes to try and self-heal and then, eventually, lock up entirely.

On reviewing the syslog I started to understand what was going on – the system was running out of memory:

sudo cat /var/log/syslog | grep memory

It looked as if, for whatever reason, wfica (the Citrix Receiver Client) couldn’t be “completely” killed to free-up memory; once killed more processes seemed to spawn which in turn would be killed, leading to more processes. Eventually more important system processes were killed to try and free-up memory:

Sep 30 11:42:15 debian kernel: [55556.844790]  [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0
Sep 30 11:42:15 debian kernel: [55556.844978] Out of memory: Kill process 9405 (clamd) score 72 or sacrifice child
Sep 30 11:52:15 debian kernel: [56156.791644]  [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0
Sep 30 11:52:15 debian kernel: [56156.791823] Out of memory: Kill process 8831 (wfica) score 30 or sacrifice child
Sep 30 11:52:15 debian kernel: [56156.792333]  [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0
Sep 30 11:52:15 debian kernel: [56156.792498] Out of memory: Kill process 8834 (threaded-ml) score 30 or sacrifice child
Sep 30 12:02:16 debian kernel: [56757.847752]  [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0
Sep 30 12:02:16 debian kernel: [56757.848397] Out of memory: Kill process 8853 (wfica) score 31 or sacrifice child
[...]
[...] # many more wfica processes killed, different PIDs
[...]
process 9238 (wfica) score 45 or sacrifice child
Sep 30 13:52:39 debian kernel: [63381.448595]  [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0
Sep 30 13:52:39 debian kernel: [63381.448762] Out of memory: Kill process 1147 (dropbox) score 18 or sacrifice child
Sep 30 13:53:17 debian kernel: [63419.152094]  [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0
Sep 30 13:53:17 debian kernel: [63419.152256] Out of memory: Kill process 8655 (shutter) score 10 or sacrifice child
Sep 30 13:53:21 debian kernel: [63423.078535]  [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0
Sep 30 13:53:21 debian kernel: [63423.078699] Out of memory: Kill process 1071 (Xvnc) score 10 or sacrifice child
Sep 30 13:57:26 debian kernel: [63666.800723]  [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0
Sep 30 13:57:26 debian kernel: [63666.800914] Out of memory: Kill process 852 (lightdm-gtk-gre) score 3 or sacrifice child
Sep 30 13:57:26 debian kernel: [63666.820658]  [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0
Sep 30 13:57:26 debian kernel: [63666.820787] Out of memory: Kill process 581 (Xorg) score 2 or sacrifice child
Sep 30 13:57:26 debian kernel: [63668.058072]  [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0
Sep 30 13:57:26 debian kernel: [63668.058324] Out of memory: Kill process 501 (freshclam) score 0 or sacrifice child
Sep 30 13:57:26 debian kernel: [63668.065610]  [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0
Sep 30 13:57:27 debian kernel: [63668.065906] Out of memory: Kill process 3727 (AuthManagerDaem) score 0 or sacrifice child
Sep 30 13:57:29 debian kernel: [63668.256015]  [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0
Sep 30 13:57:29 debian kernel: [63668.256217] Out of memory: Kill process 9143 (AuthManagerDaem) score 0 or sacrifice child
Sep 30 13:57:29 debian kernel: [63668.872898]  [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0
Sep 30 13:57:30 debian kernel: [63668.873008] Out of memory: Kill process 192 (systemd-journal) score 0 or sacrifice child
Sep 30 13:57:30 debian kernel: [63669.939297]  [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0
Sep 30 13:57:30 debian kernel: [63669.939437] Out of memory: Kill process 552 (polkitd) score 0 or sacrifice child
Sep 30 13:57:30 debian kernel: [63669.943290]  [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0
Sep 30 13:57:30 debian kernel: [63669.943387] Out of memory: Kill process 502 (NetworkManager) score 0 or sacrifice child

In terms of cause, this was simple, Debian 8 doesn’t support Hyper-V Dynamic Memory: https://technet.microsoft.com/en-us/windows-server-docs/compute/hyper-v/supported-debian-virtual-machines-on-hyper-v

However, this got me thinking, coming from a Windows background, about how to monitor / review consumption of resources.

To view memory consumption, by process, sorted high > low:

top -o %MEM

Review syslog for out of memory events:

sudo cat /var/log/syslog | grep memory

To view virtual memory consumption:

vmstat

To view CPU utilisation, by process, sorted high > low:

top -o %CPU

To view disk utilisation:

sudo apt-get install sysstat
iostat -d #for physical disks
iostat -N #for LVM

[more to come]