{"id":1272,"date":"2021-04-30T16:29:38","date_gmt":"2021-04-30T16:29:38","guid":{"rendered":"https:\/\/www.nicktailor.com\/?p=1272"},"modified":"2023-05-12T02:47:47","modified_gmt":"2023-05-12T02:47:47","slug":"how-to-check-cpu-memory-disks-mounts-on-an-array-of-hosts","status":"publish","type":"post","link":"https:\/\/nicktailor.com\/tech-blog\/how-to-check-cpu-memory-disks-mounts-on-an-array-of-hosts\/","title":{"rendered":"HOW TO CHECK CPU, MEMORY, & DISKS THRESHHOLDS on an ARRAY of HOSTS."},"content":{"rendered":"

So I was tinkering around as usual. I thought this will come in handy for other engineers<\/strong><\/p>\n

If you a large cluster of servers that can suddenly over night loose all its MEM,CPU<\/span>,DISK <\/span>due to the nature of your businesses. Its difficult to monitor that from a GUI and on an array of hosts more often  than not.<\/strong><\/p>\n

Cloud Scenario\u2026…<\/strong><\/p>\n

Say you find a node that is dying because too many clients are using resources and you need migrate instances off to another node, only you don\u2019t know which nodes have the needed resources without having to go look at all the nodes individually. <\/strong><\/p>\n

This tends be every engineers pain point. So I decide to come up with quick easy solution for emergency situations, where you don\u2019t have time to sifting through alert systems that only show you data on a per host basis, that tend to load very slowly.<\/strong><\/p>\n

This bash script will check the CPU, MEM, DISK MOUNTS (including NFS) and tell which ones are okay and which ones are<\/strong><\/p>\n

CPU – calculated by the = 100MaxThrottle \u2013 Cpu-idle = CPU-usage<\/span><\/strong>
\nnote: it also creates a log \/opt\/cpu.log on each host<\/span><\/strong><\/p>\n

MEM \u2013 calculate by Total Mem \/ Used Memory * 100 = Percentage of Used Memory<\/span><\/strong>
\nnote: it also creates a log \/opt\/mem.log on each host<\/span><\/strong><\/p>\n

Disk \u2013 Any mount that reaches the warn threshold\u2026 COMPLAIN<\/span><\/strong><\/p>\n

.<\/span><\/span><\/strong><\/p>\n

Now, itemised the bash script so you can just comment out item you don\u2019t want to use at the bottom of the script if you wanted to say just check CPU\/MEM<\/span><\/strong><\/p>\n

#Written By Nick Tailor<\/strong><\/p>\n

#!\/bin\/bash<\/strong><\/p>\n

now=`date -u -d”+8 hour” +’%Y-%m-%d %H:%M:%S’`<\/strong><\/p>\n

#cpu use threshold<\/strong><\/p>\n

cpu_warn=’75’<\/strong><\/p>\n

#disk use threshold<\/strong><\/p>\n

disk_warn=’80’<\/strong><\/p>\n

#—cpu<\/strong><\/p>\n

item_cpu () {<\/strong><\/p>\n

cpu_idle=`top -b -n 1 | grep Cpu | awk ‘{print $8}’|cut -f 1 -d “.”`<\/strong><\/p>\n

cpu_use=`expr 100 – $cpu_idle`<\/strong><\/p>\n

echo “now current cpu utilization rate of $cpu_use $(hostname) as on $(date)” >> \/opt\/cpu.log<\/strong><\/p>\n

if [ $cpu_use -gt $cpu_warn ]<\/strong><\/p>\n

then<\/strong><\/p>\n

echo “cpu warning!!! $cpu_use Currently HIGH $(hostname)”<\/strong><\/p>\n

else<\/strong><\/p>\n

echo “cpu ok!!! $cpu_use% use Currently LOW $(hostname)”<\/strong><\/p>\n

fi<\/strong><\/p>\n

}<\/strong><\/p>\n

#—mem<\/strong><\/p>\n

item_mem () {<\/strong><\/p>\n

#MB units<\/strong><\/p>\n

LOAD=’80.00′<\/strong><\/p>\n

mem_free_read=`free -h | grep “Mem” | awk ‘{print $4+$6}’`<\/strong><\/p>\n

MEM_LOAD=`free -t | awk ‘FNR == 2 {printf(“%.2f%”), $3\/$2*100}’`<\/strong><\/p>\n

echo “Now the current memory space remaining ${mem_free_read} GB $(hostname) as on $(date)” >> \/opt\/mem.log<\/strong><\/p>\n

if [[ $MEM_LOAD > $LOAD ]]<\/strong><\/p>\n

then<\/strong><\/p>\n

echo “$MEM_LOAD not good!! MEM USEAGE is HIGH – Free-MEM-${mem_free_read}GB $(hostname)”<\/strong><\/p>\n

else<\/strong><\/p>\n

echo “$MEM_LOAD ok!! MEM USAGE is beLOW 80% – Free-MEM-${mem_free_read}GB $(hostname)”<\/strong><\/p>\n

fi<\/strong><\/p>\n

}<\/strong><\/p>\n

#—disk<\/strong><\/p>\n

item_disk () {<\/strong><\/p>\n

df -H | grep -vE ‘^Filesystem|tmpfs|cdrom’ | awk ‘{ print $5 ” ” $1 }’ | while read output;<\/strong><\/p>\n

do<\/strong><\/p>\n

echo $output<\/strong><\/p>\n

  usep=$(echo $output | awk ‘{ print $1}’ | cut -d’%’ -f1 )<\/strong><\/p>\n

partition=$(echo $output | awk ‘{ print $2 }’ )<\/strong><\/p>\n

if [ $usep -ge $disk_warn ]; then<\/strong><\/p>\n

echo “AHH SHIT!, MOVE SOME VOLUMES IDIOT…. \\”$partition ($usep%)\\” on $(hostname) as on $(date)”<\/strong><\/p>\n

fi<\/strong><\/p>\n

done<\/strong><\/p>\n

}<\/strong><\/p>\n

item_cpu<\/strong><\/p>\n

item_mem<\/strong><\/p>\n

#item_disk \u2013 This is so you can comment out whole sections of the script<\/em><\/span> without having to do the whole section by individual lines.<\/em><\/span><\/strong><\/p>\n

Now the cool part.<\/strong><\/p>\n

Now if you have a centrally managed jump host that allows you to get out from your estate. Ideally you would want to setup ssh keys on the hosts and ensure you have sudo permissions on the those hosts.<\/p>\n

We want to loop this script through an array of hosts and have it run and then report back all the findings in once place. This is extremely handy if your in resource crunch.<\/p>\n

This assumes you have SSH KEYS SETUP & SUDO <\/span>for your user setup.<\/p>\n

Create the script<\/p>\n

\n
\n
1.<\/span>On your jump host as your \u201cuser\u201d not root<\/strong><\/div>\n
\n
\n
a.<\/span>vi coolchecks.sh<\/em><\/strong><\/div>\n<\/div>\n
\n
b.<\/span>Copy the above code and paste<\/strong><\/div>\n<\/div>\n
\n
c.<\/span>Save the file<\/strong><\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
2.<\/span>Next chmod the permission to executable<\/strong><\/div>\n
\n
\n
d.<\/span>chmod +x coolcheck.sh<\/em><\/strong><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

Next<\/p>\n

\n
\n
3.<\/span>Create a servers.txt file<\/strong><\/div>\n
\n
\n
e.<\/span>vi servers.txt<\/em><\/strong><\/div>\n<\/div>\n
\n
f.<\/span>List out servers in a column<\/strong><\/div>\n

Server1<\/span>
\nServer2<\/span>
\nServer3<\/span>
\nServer4<\/span><\/strong><\/p>\n<\/div>\n

\n
g.<\/span>Save the file.<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
4.<\/span>Now we want to loop that through the list of servers and then have it spit out the results and pipe the information to a file on the jumps host.<\/strong><\/div>\n

Run your forloop with ssh keys and sudo already setup.<\/strong><\/p>\n

.<\/span><\/span><\/p>\n

\n
\n
1.<\/span>for HOST in $(cat servers.txt); do ssh $HOST “sudo bash -s” < coolcheck.sh; done 2>&1 | tee -a cpumem.status.DEV<\/em><\/strong><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

Logfile – <\/span>cpumem.status.DEV<\/span> \u2013 will be the log file that has all the info<\/span><\/strong><\/p>\n

Output:<\/strong><\/p>\n

cpu<\/span> ok!!! 3% use Currently dev1.nicktailor.com<\/span><\/strong><\/p>\n

17.07% ok!! MEM USAGE is <\/span>beLOW<\/span> 80% – Free-MEM-312.7GB dev1.nicktailor.com<\/span><\/strong><\/p>\n

5% \/dev\/mapper\/VolGroup00-root<\/span><\/strong><\/p>\n

3% \/dev\/sda2<\/span><\/strong><\/p>\n

5% \/dev\/sda1<\/span><\/strong><\/p>\n

1% \/dev\/mapper\/VolGroup00-var_log<\/span><\/strong><\/p>\n

72% 192.168.1.101:\/data_1<\/span><\/strong><\/p>\n

28% 192.168.1.102:\/data_2<\/span><\/strong><\/p>\n

80% 192.168.1.103:\/data_3<\/span><\/strong><\/p>\n

AHH <\/span>SHIT!,<\/span> MOVE SOME VOLUMES IDIOT…. “192.168.1.104:\/data4 (80%)” on dev1.nicktailor.com as on Fri Apr 30 11:55:16 EDT 2021<\/span><\/strong><\/p>\n

.<\/span><\/span><\/p>\n

Okay so now I\u2019m gonna show you a dirty way to do it, because im just dirty. So say your in horrible place that doesn\u2019t use keys, because they’re waiting to be hacked by password. \ud83d\ude1b<\/strong><\/p>\n

.<\/span><\/span><\/strong><\/p>\n

DIRTY WAY \u2013 So this assumes you have sudo<\/span> permissions on the hosts<\/span>.<\/strong><\/p>\n

Note: I do not recommend doing this way if you are a newb. Doing it this way will basically log your password in the bash history and if you don\u2019t know how to clean up after yourself, well\u2026\u2026\u2026\u2026\u2026\u2026\u2026.you’re<\/span> going to get <\/span>owned.<\/span><\/strong><\/p>\n

I\u2019m only showing you this because<\/span> some<\/span> cyber security <\/span> \u201cfolks\u201d <\/span>believe that not using keys is easier to deal with in some <\/span>parallel realities iv visited<\/span>… You can do the exact same thing above<\/span>,<\/span> without keys. But leave massive trail behind you. <\/span> Hence why you should use <\/span>secure keys with passwords<\/span>.<\/span><\/strong><\/p>\n

.<\/span><\/span><\/p>\n

Not Recommended for <\/span>Newbies<\/span>:
\n<\/span>Forloop<\/span> AND passing your <\/span>ssh<\/span> password<\/span> inside it.<\/span><\/strong><\/p>\n

\n
\n
2.<\/span>for HOST in $(cat servers.txt); do sshpass -p’SHHPASSWORD<\/span>!<\/span>‘ ssh -o ‘StrictHostKeyChecking no’ -p 22 $HOST “sudo bash -s” < coolcheck.sh; done 2>&1 | tee -a cpumem.status.DEV<\/span><\/em><\/strong><\/div>\n

.<\/span><\/span><\/strong><\/p>\n<\/div>\n<\/div>\n

Log file – <\/span>cpumem.status.DEV<\/span> \u2013 will be the log file that has all the info<\/span><\/strong><\/p>\n

Output:<\/strong><\/p>\n

cpu<\/span> ok!!! 3% use Currently dev1.nicktailor.com<\/span><\/strong><\/p>\n

17.07% ok!! MEM USAGE is <\/span>beLOW<\/span> 80% – Free-MEM-312.7GB dev1.nicktailor.com<\/span><\/strong><\/p>\n

5% \/dev\/mapper\/VolGroup00-root<\/span><\/strong><\/p>\n

3% \/dev\/sda2<\/span><\/strong><\/p>\n

5% \/dev\/sda1<\/span><\/strong><\/p>\n

1% \/dev\/mapper\/VolGroup00-var_log<\/span><\/strong><\/p>\n

72% 192.168.1.101:\/data_1<\/span><\/strong><\/p>\n

28% 192.168.1.102:\/data_2<\/span><\/strong><\/p>\n

80% 192.168.1.103:\/data_3 <\/span><\/strong><\/p>\n

AHH <\/span>SHIT!,<\/span> MOVE SOME VOLUMES IDIOT…. “192.168.1.104:\/data4 (80%)” on dev1.nicktailor.com as on Fri Apr 30 11:55:16 EDT 2021
\n<\/span><\/strong>
\n.<\/span><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

So I was tinkering around as usual. I thought this will come in handy for other engineers If you a large cluster of servers that can suddenly over night loose all its MEM,CPU,DISK due to the nature of your businesses. Its difficult to monitor that from a GUI and on an array of hosts more often  than not. Cloud Scenario\u2026…Read More …<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[58,138,132],"tags":[],"class_list":["post-1272","post","type-post","status-publish","format-standard","hentry","category-centos","category-linux","category-monitoring"],"_links":{"self":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/1272","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/comments?post=1272"}],"version-history":[{"count":7,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/1272\/revisions"}],"predecessor-version":[{"id":1583,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/1272\/revisions\/1583"}],"wp:attachment":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/media?parent=1272"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/categories?post=1272"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/tags?post=1272"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}