{"id":940,"date":"2020-05-27T14:26:38","date_gmt":"2020-05-27T14:26:38","guid":{"rendered":"https:\/\/www.nicktailor.com\/?p=940"},"modified":"2022-10-21T11:34:14","modified_gmt":"2022-10-21T11:34:14","slug":"how-to-fix-in-the-infiniband-issue-when-migrating-multiple-bonded-nics-to-readhat-7","status":"publish","type":"post","link":"https:\/\/nicktailor.com\/tech-blog\/how-to-fix-in-the-infiniband-issue-when-migrating-multiple-bonded-nics-to-readhat-7\/","title":{"rendered":"How to fix in the infiniband issue when migrating multiple bonded nics to redhat 7"},"content":{"rendered":"

Okay so some of you be using malenox FPGA cards which basically bypasses the BUS to give lower latency on your network response time.<\/p>\n

Now if you have used an OS like SUSE and had a butt load of bonded nics and then want to migrate the OS and all the bonded nics configurations in an automated fashion using ansible or something configuration management tool.<\/p>\n

What some of you might run into is when the OS comes up for the first time, some of the Mellanox nics will boot up in infiniband mode. Which will result in the bonded nics showing up as down. I will show you how to determine this and fix this.<\/p>\n

.<\/span><\/span><\/p>\n

So the first thing you want to do is determine which bonds are showing down<\/p>\n

How to check which bonds are down.<\/p>\n

\n
\n
1.<\/span>grep -c down \/proc\/net\/bonding\/*<\/em><\/div>\n
\n
\n
\u25e6<\/span> <\/span>this will list out all the bonds that show an interface is down<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

Example<\/strong><\/p>\n

root@ansibleclient:~> grep -c down \/proc\/net\/bonding\/*<\/em><\/p>\n

.<\/span><\/span><\/p>\n

\/proc\/net\/bonding\/bond1:0<\/p>\n

\/proc\/net\/bonding\/bond2:0<\/p>\n

\/proc\/net\/bonding\/<\/em><\/span>bond3<\/em><\/span>:<\/em><\/span>1<\/em><\/span> (this indicates that one interface is down)<\/em><\/span><\/p>\n

.<\/span><\/span><\/p>\n

\n
\n
2.<\/span>Once you determine the bond has an interface that is down you want to figure out if it\u2019s the Mellanox card nic.<\/div>\n
\n
\n
\u2022<\/span> <\/span>cat \/proc\/net\/bonding\/bond3<\/em><\/div>\n
\n
\n
i.<\/span>this will give you the nic mac address that are inside the bond.<\/em><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

Example<\/strong><\/p>\n

Bonding Mode: fault-tolerance (active-backup)<\/em><\/p>\n

Primary Slave: None<\/em><\/p>\n

Currently Active Slave: eth4<\/em><\/p>\n

MII Status: up<\/em><\/p>\n

MII Polling Interval (ms): 100<\/em><\/p>\n

Up Delay (ms): 0<\/em><\/p>\n

Down Delay (ms): 0<\/em><\/p>\n

.<\/span><\/span><\/p>\n

Slave Interface: eth4<\/em><\/p>\n

MII Status: up<\/em><\/p>\n

Speed: 10000 Mbps<\/em><\/p>\n

Duplex: full<\/em><\/p>\n

Link Failure Count: 0<\/em><\/p>\n

Permanent HW addr: 00:02:c9:e9:e9:11<\/em><\/p>\n

Slave queue ID: 0<\/em><\/p>\n

.<\/span><\/span><\/p>\n

Slave Interface: eth5<\/em><\/p>\n

MII Status: up<\/em><\/p>\n

Speed: 10000 Mbps<\/em><\/p>\n

Duplex: full<\/em><\/p>\n

Link Failure Count: 0<\/em><\/p>\n

Permanent HW addr: 00:02:c9:e9:e9:12<\/em><\/p>\n

Slave queue ID: 0<\/em><\/p>\n

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)<\/em><\/p>\n

.<\/span><\/span><\/p>\n

\n
\n
3.<\/span>now what you want to do next is run \u2018ip a\u2019 and see if those interfaces are listed<\/div>\n<\/div>\n<\/div>\n

.<\/span><\/span><\/p>\n

Example \u2013 should look something like this. If you don\u2019t see the down nic here for our example lets say its eth5. This could mean its in infiniband mode and not ethernet mode. It also shows if the interface is up or down. Which is very important when troubleshooting the interface<\/em><\/strong><\/p>\n

.<\/span><\/span><\/p>\n

[root@nickansible]# ip a<\/p>\n

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000<\/p>\n

link\/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<\/p>\n

    inet 127.0.0.1\/8 scope host lo<\/p>\n

       valid_lft forever preferred_lft forever<\/p>\n

2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000<\/p>\n

link\/ether 08:00:26:9a:33:59 brd ff:ff:ff:ff:ff:ff<\/p>\n

    inet 10.0.2.15\/24 brd 10.0.2.255 scope global noprefixroute dynamic enp0s3<\/p>\n

       valid_lft 82770sec preferred_lft 82770sec<\/p>\n

3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000<\/p>\n

link\/ether 08:00:26:88:5a:fd brd ff:ff:ff:ff:ff:ff<\/p>\n

    inet 192.168.1.11\/24 brd 192.168.1.255 scope global noprefixroute dynamic enp0s8<\/p>\n

       valid_lft 82773sec preferred_lft 82773sec<\/p>\n

.<\/span><\/span><\/p>\n

\n
\n
4.<\/span>Okay now we need to determine if eth5 is infact the Mellanox card. So now we need the nic information<\/div>\n
\n
\n
\u2022<\/span> <\/span>Ethtool -I eth5<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

.<\/span><\/span><\/p>\n

Example.<\/strong>
\nIt will look something like this.<\/p>\n

.<\/span><\/span><\/p>\n

[root@nick ansible# ethtool -i eth5<\/p>\n

driver: e1000<\/p>\n

version: 7.3.21-k8-NAPI<\/p>\n

firmware-version:<\/p>\n

expansion-rom-version:<\/p>\n

bus-info: 0000:00:<\/span>1<\/span>8.<\/span>0<\/span> (<\/span>this is the important info you need)<\/span><\/strong><\/p>\n

supports-statistics: yes<\/p>\n

supports-test: yes<\/p>\n

supports-eeprom-access: yes<\/p>\n

supports-register-dump: yes<\/p>\n

supports-priv-flags: no<\/p>\n

.<\/span><\/span><\/p>\n

\n
\n
\u2022<\/span> <\/span>Now you want to take the bus info and determine if it is infact the Mellanox card<\/div>\n
\n
\n
\u25e6<\/span> <\/span>lspci<\/em><\/span> –<\/em><\/span>s<\/em><\/span> 0000:00:18.0<\/em><\/span>0<\/em><\/span>  <\/em><\/span><\/strong><\/div>\n<\/div>\n<\/div>\n

.<\/span><\/span><\/p>\n

Example<\/strong><\/p>\n

.<\/span><\/span><\/p>\n

[root@nick ansible]# lspci -s 0000:00:18.0.0<\/em><\/p>\n<\/div>\n<\/div>\n

00:18:00<\/span> Ethernet controller: Mellanox Technologies MT26448 [<\/span>ConnectX<\/span> EN 10GigE, PCIe 2.0 5GT\/s]<\/span><\/p>\n

.<\/span><\/span><\/p>\n

\n
\n
5.<\/span>Okay now we know for sure this nic is infact the Mellanox nic that is down. So now we went to manually force it into ethernet but first check to see what it says<\/div>\n
\n
\n
\u2022<\/span> <\/span>cat \/sys\/bus\/pci\/devices\/0000\\:18\\:00.0\/mlx4_port0<\/em><\/div>\n
\n
\n
ii.<\/span>if this doesn\u2019t return \u201cETH\u201d then its in infiniabnd mode<\/em><\/strong><\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\u2022<\/span> <\/span>cat \/sys\/bus\/pci\/devices\/0000\\:18\\:00.0\/mlx4_port1<\/em><\/div>\n
\n
\n
iii.<\/span>if this doesn\u2019t return \u201cETH\u201d then its in infiniabnd mode<\/em><\/strong><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

.<\/span><\/span><\/p>\n

\n
\n
6.<\/span>Now what we want to od is manually change the nic to ethernet mode<\/div>\n
\n
\n
\u2022<\/span> <\/span>echo eth > \/sys\/bus\/pci\/devices\/0000\\:18\\:00.0\/mlx4_port1<\/em><\/div>\n<\/div>\n
\n
\u2022<\/span> <\/span>echo eth > \/sys\/bus\/pci\/devices\/0000\\:18\\:00.0\/mlx4_port1<\/em><\/div>\n
\n
\n
iv.<\/span>If you cat them now it should say \u201cETH\u201d<\/strong><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

.<\/span><\/span><\/p>\n

Okay so now when you do \u2018ip a\u2019 you should should see the nics up and if you check the status of the bond there should be 0 bonds down. You might have to bring the bond down and up.<\/p>\n

.<\/span><\/span><\/p>\n

\n
\n
7.<\/span>You can do this simply by<\/div>\n
\n
\n
\u2022<\/span> <\/span>Ifdown eth5 & ifup eth5<\/em><\/div>\n
\n
\n
v.<\/span>If there are no errors, the cursor will simply move to the next line with a brief delay.<\/em><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

.<\/span><\/span><\/p>\n

.<\/span><\/span><\/p>\n

Now the issue here is that if you aren\u2019t able to get rpms from Mellanox that are supported by patching in your organisation. You\u2019re going to need a way to ensure that if the server reboots the nic will start up in ethernet mode, otherwise you could be in a very bad situation if the server boots and the nic came up in infiniband mode.<\/p>\n

.<\/span><\/span><\/p>\n

So there are a couple of ideas I came up with to solve this.<\/p>\n

Option:<\/p>\n

\n
\n
1.<\/span>You can simply add the echo lines in the \/etc\/rc.local<\/div>\n
\n
\n
\u2022<\/span> <\/span>echo eth > \/sys\/bus\/pci\/devices\/0000\\:18\\:00.0\/mlx4_port1<\/em><\/strong><\/div>\n<\/div>\n
\n
\u2022<\/span> <\/span>echo eth > \/sys\/bus\/pci\/devices\/0000\\:18\\:00.0\/mlx4_port1<\/em><\/strong><\/div>\n
\n
\n
i.<\/span>This should bring the interface back to “ETH”<\/strong>, however you might need to add some more lines to bring the interface up properly.<\/em><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

.<\/span><\/span><\/p>\n<\/div>\n<\/div>\n

\n
\n
\n

1.<\/span>This the approach I chose and the cooler way to go about it. In redhat 7 you can define a if-preup-local script which will run anytime \u201cifup\u201d is run.<\/p>\n

Here is how you set that up.<\/p>\n<\/div>\n

\n
\n
1.<\/span>Create a file called \u201c\/etc\/sysconfig\/network-scripts\/ifup-pre-local\u2019<\/div>\n
\n
\n
a.<\/span>vi \/etc\/sysconfig\/network-scripts\/ifup-pre-local<\/em><\/div>\n<\/div>\n<\/div>\n

.<\/span><\/span><\/p>\n<\/div>\n

\n
2.<\/span>Now you can add whatever script you want. My colleague and I came up with a script that determined based on mac and bus info and if it certain buses and mac showed up it would run the echo to move the ports into eth mode<\/div>\n

.<\/span><\/span><\/p>\n

ADD this inside and save the file<\/em><\/strong><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

#!\/bin\/bash<\/p>\n

.<\/span><\/span><\/p>\n

#<\/p>\n

.<\/span><\/span><\/p>\n

LID=”00:00:00:00″<\/p>\n

for i in `ls \/etc\/sysconfig\/network-scripts\/ifcfg-* 2> \/dev\/null`<\/p>\n

do<\/p>\n

for j in `grep HWADDR $i |awk -F\\” ‘{print $2}’`<\/p>\n

do<\/p>\n

ID1=$(echo $j | awk -F\\: ‘{print $2″:”$3}’)<\/p>\n

ID2=$(echo $j | awk -F\\: ‘{print $4″:”$5}’)<\/p>\n

ID=”$ID1:$ID2″<\/p>\n

PORT=$(echo $j | cut -c 16-17)<\/p>\n

  for k in `ls \/sys\/bus\/pci\/devices\/0000\\:*\\:00.0\/net\/ib[0-9]\/address 2> \/dev\/null`<\/p>\n

do<\/p>\n

grep “$ID1.*$ID2” $k 1> \/dev\/null<\/p>\n

if [ $? -eq 0 ]; then<\/p>\n

if [ “x$ID” != “x$LID” ]; then<\/p>\n

     mlxport=1<\/p>\n

else<\/p>\n

let “mlxport++”<\/p>\n

fi<\/p>\n

LID=$ID<\/p>\n

p=$(echo $k | awk -F\/ ‘{print “\/sys\/bus\/pci\/devices\/”$6″\/”}’)<\/p>\n

echo “Running: echo eth > ${p}mlx4_port${mlxport}”<\/p>\n

echo eth > ${p}mlx4_port${mlxport}<\/p>\n

fi<\/p>\n

done<\/p>\n

done<\/p>\n

done<\/p>\n

.<\/span><\/span><\/p>\n

\n
\n
3.<\/span>Next you want to create a symlink in side \/sbin<\/div>\n
\n
\n
b.<\/span>Move into \/sbin<\/div>\n
\n
\n
i.<\/span>cd \/sbin<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
c.<\/span>now create a symlink for ifup-pre-local<\/em><\/div>\n
\n
\n
ii.<\/span>ln -s \/etc\/sysconfig\/network-scripts\/ifup-pre-local ifup-pre-local<\/em><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

.<\/span><\/span><\/p>\n

Now when you run ifup<\/strong> it will run that script that check to see if the any of those bus and macs are in infinband mode and bring them into eth. It safer to do this way because if you restart the network and for some reason the nic goes back into infiniband and someone new had no idea. They would spend awhile trying to figure this out.<\/p>\n<\/div>\n<\/div>\n

.<\/span><\/span><\/p>\n

.<\/span><\/span><\/p>\n

.<\/span><\/span><\/p>\n

How do deploy this fix via anisble role coming soon\u2026\u2026<\/p>\n

.<\/span><\/span><\/p>\n

.<\/span><\/span><\/p>\n

.<\/span><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

Okay so some of you be using malenox FPGA cards which basically bypasses the BUS to give lower latency on your network response time. Now if you have used an OS like SUSE and had a butt load of bonded nics and then want to migrate the OS and all the bonded nics configurations in an automated fashion using ansibleRead More …<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[58,138,124],"tags":[],"class_list":["post-940","post","type-post","status-publish","format-standard","hentry","category-centos","category-linux","category-network-stuff"],"_links":{"self":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/940","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/comments?post=940"}],"version-history":[{"count":39,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/940\/revisions"}],"predecessor-version":[{"id":1599,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/940\/revisions\/1599"}],"wp:attachment":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/media?parent=940"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/categories?post=940"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/tags?post=940"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}