I ran into some issues running Debian 8.6 on Hyper-V recently – my VM would run out of memory, run through a series of activities kill processes to try and self-heal and then, eventually, lock up entirely.
On reviewing the syslog I started to understand what was going on – the system was running out of memory:
sudo cat /var/log/syslog | grep memory
It looked as if, for whatever reason, wfica (the Citrix Receiver Client) couldn’t be “completely” killed to free-up memory; once killed more processes seemed to spawn which in turn would be killed, leading to more processes. Eventually more important system processes were killed to try and free-up memory:
Sep 30 11:42:15 debian kernel: [55556.844790] [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0 Sep 30 11:42:15 debian kernel: [55556.844978] Out of memory: Kill process 9405 (clamd) score 72 or sacrifice child Sep 30 11:52:15 debian kernel: [56156.791644] [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0 Sep 30 11:52:15 debian kernel: [56156.791823] Out of memory: Kill process 8831 (wfica) score 30 or sacrifice child Sep 30 11:52:15 debian kernel: [56156.792333] [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0 Sep 30 11:52:15 debian kernel: [56156.792498] Out of memory: Kill process 8834 (threaded-ml) score 30 or sacrifice child Sep 30 12:02:16 debian kernel: [56757.847752] [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0 Sep 30 12:02:16 debian kernel: [56757.848397] Out of memory: Kill process 8853 (wfica) score 31 or sacrifice child [...] [...] # many more wfica processes killed, different PIDs [...] process 9238 (wfica) score 45 or sacrifice child Sep 30 13:52:39 debian kernel: [63381.448595] [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0 Sep 30 13:52:39 debian kernel: [63381.448762] Out of memory: Kill process 1147 (dropbox) score 18 or sacrifice child Sep 30 13:53:17 debian kernel: [63419.152094] [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0 Sep 30 13:53:17 debian kernel: [63419.152256] Out of memory: Kill process 8655 (shutter) score 10 or sacrifice child Sep 30 13:53:21 debian kernel: [63423.078535] [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0 Sep 30 13:53:21 debian kernel: [63423.078699] Out of memory: Kill process 1071 (Xvnc) score 10 or sacrifice child Sep 30 13:57:26 debian kernel: [63666.800723] [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0 Sep 30 13:57:26 debian kernel: [63666.800914] Out of memory: Kill process 852 (lightdm-gtk-gre) score 3 or sacrifice child Sep 30 13:57:26 debian kernel: [63666.820658] [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0 Sep 30 13:57:26 debian kernel: [63666.820787] Out of memory: Kill process 581 (Xorg) score 2 or sacrifice child Sep 30 13:57:26 debian kernel: [63668.058072] [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0 Sep 30 13:57:26 debian kernel: [63668.058324] Out of memory: Kill process 501 (freshclam) score 0 or sacrifice child Sep 30 13:57:26 debian kernel: [63668.065610] [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0 Sep 30 13:57:27 debian kernel: [63668.065906] Out of memory: Kill process 3727 (AuthManagerDaem) score 0 or sacrifice child Sep 30 13:57:29 debian kernel: [63668.256015] [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0 Sep 30 13:57:29 debian kernel: [63668.256217] Out of memory: Kill process 9143 (AuthManagerDaem) score 0 or sacrifice child Sep 30 13:57:29 debian kernel: [63668.872898] [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0 Sep 30 13:57:30 debian kernel: [63668.873008] Out of memory: Kill process 192 (systemd-journal) score 0 or sacrifice child Sep 30 13:57:30 debian kernel: [63669.939297] [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0 Sep 30 13:57:30 debian kernel: [63669.939437] Out of memory: Kill process 552 (polkitd) score 0 or sacrifice child Sep 30 13:57:30 debian kernel: [63669.943290] [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0 Sep 30 13:57:30 debian kernel: [63669.943387] Out of memory: Kill process 502 (NetworkManager) score 0 or sacrifice child
In terms of cause, this was simple, Debian 8 doesn’t support Hyper-V Dynamic Memory: https://technet.microsoft.com/en-us/windows-server-docs/compute/hyper-v/supported-debian-virtual-machines-on-hyper-v
However, this got me thinking, coming from a Windows background, about how to monitor / review consumption of resources.
To view memory consumption, by process, sorted high > low:
top -o %MEM
Review syslog for out of memory events:
sudo cat /var/log/syslog | grep memory
To view virtual memory consumption:
vmstat
To view CPU utilisation, by process, sorted high > low:
top -o %CPU
To view disk utilisation:
sudo apt-get install sysstat iostat -d #for physical disks iostat -N #for LVM
[more to come]