{"id":756,"date":"2018-07-10T10:37:44","date_gmt":"2018-07-10T10:37:44","guid":{"rendered":"https:\/\/www.nicktailor.com\/?p=756"},"modified":"2022-10-21T11:38:59","modified_gmt":"2022-10-21T11:38:59","slug":"how-to-diagnose-a-kernel-panic-caused-by-a-killed-process","status":"publish","type":"post","link":"https:\/\/nicktailor.com\/tech-blog\/how-to-diagnose-a-kernel-panic-caused-by-a-killed-process\/","title":{"rendered":"How to diagnose a kernel panic caused by a killed process"},"content":{"rendered":"<p>You should install <strong>atop<\/strong> on your server as this is top on steroids and can help diagnose all kinds of server issues such as.<\/p>\n<p><strong><a href=\"https:\/\/lwn.net\/Articles\/387202\/\">https:\/\/lwn.net\/Articles\/387202\/<\/a> &#8211; Atop usage<\/strong><\/p>\n<ul>\n<li><em>CPU load<\/em><\/li>\n<li><em>IO load<\/em><\/li>\n<li><em>Memory usage<\/em><\/li>\n<li><em>Process utilization of resources<\/em><\/li>\n<li><i>Paging\/swaping<\/i><\/li>\n<li>etc&#8230;<\/li>\n<\/ul>\n<ul>\n<li><strong>How you install atop on ubuntu\/debian<\/strong>\n<ul>\n<li><em>\u2018apt-get install atop\u2019<\/em><\/li>\n<\/ul>\n<\/li>\n<li><strong>Then you want to start the atop logging<\/strong>\n<ul>\n<li><em>\u2018\/etc\/init.d\/atop start<br \/>\n<\/em><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><strong><em>Note : by default the atop logs every 10mins<\/em><\/strong><\/p>\n<p>Now lets say you console your VM or blade server. You see a message that states the server killed a process or ran out of memory or something.<\/p>\n<p><strong><u>Example:<\/u><\/strong><\/p>\n<ul>\n<li><em>Out of memory: Kill process <strong>11970 (php<\/strong>) score 80 or sacrifice child<br \/>\nKilled process <strong>11970 (php)<\/strong> total-vm:1957108kB<\/em><\/li>\n<\/ul>\n<p>When you reboot the server you will want to find out exactly how it happened. &nbsp;How you do this is by checking the kernel log. Now if you have kdump installed you can use that to get a dump of the kernel log and if not you can do this.<\/p>\n<ul>\n<li><em>dmesg | egrep -i &#8216;killed process&#8217; <\/em>\n<ul>\n<li><strong><em>this will provide a log as indicated below<\/em><\/strong><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><strong><u>Kernel log<\/u><\/strong><\/p>\n<ul>\n<li><em>[Wed July 10 13:27:30 2018] Out of memory: Kill process <strong>11970 (php)<\/strong> score 80 or sacrifice child]<\/em><\/li>\n<li><em>[Wed July 10 13:27:30 2018] Killed process <strong>11970 (php)<\/strong> total-vm:123412108kB, anon-rss:1213410764kB, file-rss:2420k]<\/em><\/li>\n<\/ul>\n<p>Now once you have this log you can see the time stamp of when it occurred and you can use atop logs to drill down and find the process id, and see if you can see which daemon and or script caused the issue.<\/p>\n<p>From the log \u2018July 10 13:27:30 2018\u2019 we can see the time stamp. Inside \/var\/log\/atop you can do the following.<\/p>\n<p style=\"padding-left: 30px;\"><em><strong>Run the following:<\/strong><\/em><\/p>\n<ul>\n<li><em>\u2018atop -r atop_20180710\u2019<br \/>\n<\/em><em>this will bring up a screen and you can toggle through the time intervals by using lowercase \u2018T\u2019 to move forward in time or Capital \u2018T\u2019 to go backward in time.<\/em><\/li>\n<li><em>Once you find the time stamp you can<br \/>\n<strong>press &#8211;<\/strong><\/em> <em><strong>&#8216;c&#8217; &#8211; full command-line per process<\/strong> to see which processes were running at that time stamp and you should be able to locate the id process from the kernel log<\/em><\/p>\n<figure id=\"attachment_772\" aria-describedby=\"caption-attachment-772\" style=\"width: 1643px\" class=\"wp-caption alignnone\"><a href=\"http:\/\/www.nicktailor.com\/wp-content\/uploads\/2018\/07\/atop-e1531224537845.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-772\" src=\"http:\/\/www.nicktailor.com\/wp-content\/uploads\/2018\/07\/atop-e1531224537845.jpg\" alt=\"\" width=\"1643\" height=\"793\" srcset=\"https:\/\/nicktailor.com\/tech-blog\/wp-content\/uploads\/2018\/07\/atop-e1531224537845.jpg 1643w, https:\/\/nicktailor.com\/tech-blog\/wp-content\/uploads\/2018\/07\/atop-e1531224537845-300x145.jpg 300w, https:\/\/nicktailor.com\/tech-blog\/wp-content\/uploads\/2018\/07\/atop-e1531224537845-768x371.jpg 768w, https:\/\/nicktailor.com\/tech-blog\/wp-content\/uploads\/2018\/07\/atop-e1531224537845-1024x494.jpg 1024w, https:\/\/nicktailor.com\/tech-blog\/wp-content\/uploads\/2018\/07\/atop-e1531224537845-733x354.jpg 733w, https:\/\/nicktailor.com\/tech-blog\/wp-content\/uploads\/2018\/07\/atop-e1531224537845-600x290.jpg 600w\" sizes=\"auto, (max-width: 1643px) 100vw, 1643px\" \/><\/a><figcaption id=\"caption-attachment-772\" class=\"wp-caption-text\">atop -r atop_20180710\u2019<\/figcaption><\/figure>\n<p>&nbsp;<\/li>\n<\/ul>\n<p><strong><u>Example<\/u><\/strong><\/p>\n<ul>\n<li><em>3082&nbsp; 27% php<\/em><\/li>\n<li><em>15338&nbsp; 27% php<\/em><\/li>\n<li><em>26639&nbsp; 25% php<\/em><\/li>\n<li><em>8520&nbsp;&nbsp; 8% php<\/em><\/li>\n<li><em>8796&nbsp;&nbsp; 8% php<\/em><\/li>\n<li><em>2157&nbsp;&nbsp; 8% \/usr\/sbin\/apache2 -k start<\/em><\/li>\n<li><strong><em>11970&nbsp;&nbsp; 1% php &#8211;&nbsp;<\/em><\/strong><strong><em>This is the process ID from the kernel log above and what appears to what was running. So we know it was a php script. Atop doesn\u2019t always provide the exact script. However from the kernel log and this we can determine what was some type of rss feed. From this you can also see that it wasnt using very much CPU. This helps us determine that the php code is causing a memory leak and needs to be updated and or disabled.<\/em><\/strong><\/li>\n<li><em>10493&nbsp;&nbsp; 1% php<\/em><\/li>\n<li><em>10942&nbsp;&nbsp; 1% php<\/em><\/li>\n<li><em>5335&nbsp;&nbsp; 1% php<\/em><\/li>\n<li><em>9964&nbsp;&nbsp; 0% php<\/em><\/li>\n<\/ul>\n<p><em>Written by Nick Tailor<\/em><\/p>\n<p><em>&nbsp;<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>You should install atop on your server as this is top on steroids and can help diagnose all kinds of server issues such as. https:\/\/lwn.net\/Articles\/387202\/ &#8211; Atop usage CPU load IO load Memory usage Process utilization of resources Paging\/swaping etc&#8230; How you install atop on ubuntu\/debian \u2018apt-get install atop\u2019 Then you want to start the atop logging \u2018\/etc\/init.d\/atop start Note<a href=\"https:\/\/nicktailor.com\/tech-blog\/how-to-diagnose-a-kernel-panic-caused-by-a-killed-process\/\" class=\"read-more\">Read More &#8230;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[58,52,138],"tags":[],"class_list":["post-756","post","type-post","status-publish","format-standard","hentry","category-centos","category-kernel-stuff","category-linux"],"_links":{"self":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/756","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/comments?post=756"}],"version-history":[{"count":18,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/756\/revisions"}],"predecessor-version":[{"id":1603,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/756\/revisions\/1603"}],"wp:attachment":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/media?parent=756"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/categories?post=756"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/tags?post=756"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}