Skip to content

ConnectX-3 Set Port Mode to ETH/IB

I just swapped my TrueNAS from a Dell R720XD over to a Dell r730XD. During this migration, I moved over my NVMe drives, NICs, and everything else relevant.

However, after firing up TrueNAS on the new hardware, I noticed my 40G NIC was not listed in the list of devices!

After once again googling how to change the mode over, I figured I would just quickly write some documentation incase this happens again in the future.

Why would you change this?

Mellonax NICs can support traffic in either Ethernet mode, or Infiniband Mode.

In ethernet mode, they function exactly like you would expect a NIC to work. However, in Infiniband mode, they work quite a bit different, and may require special drivers and software to perform correctly.

For most homelab uses, you will generally want to ensure they are configured to ethernet mode.

How to change port mode of ConnectX-3 NIC

Step 1. Determine the PCI ID.

You can do this using lspci

root@truenas:~# lspci | grep Mell
09:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

In the above case, 09:00.0 is the information we are looking for.

Step 2. Determine current port mode.

root@truenas:~# cat /sys/bus/pci/devices/0000\:09\:00.0/mlx4_port1
root@truenas:~# cat /sys/bus/pci/devices/0000\:09\:00.0/mlx4_port2

Using the above command, we can determine port1 is in infiniband mode, and port2 is in ethernet mode.

Step 3. How to change port mode.

There are two ways you can change the port mode. You can either.... use a simple echo command, or, you can use mstflint.

Method 1. Using echo.

root@truenas:~# echo eth > /sys/bus/pci/devices/0000\:09\:00.0/mlx4_port1
-bash: echo: write error: Operation not supported

root@truenas:~# echo ib > /sys/bus/pci/devices/0000\:09\:00.0/mlx4_port1

# Works fine?

IF, this option does not work for you, then try using mstflint. If it does work (no error), reboot for it to take effect.

Method 2. Using mstflint.

apt-get install mstflint

Note- if you use truenas like me... you may need to Re-enable apt-get

Then use mstflint -d {your pci id} q to query info

root@truenas:~# mstconfig -d 09:00.0 q

Device #1:

Device type:    ConnectX3
Device:         09:00.0

Configurations:                              Next Boot
         SRIOV_EN                            True(1)
         NUM_OF_VFS                          8
         LINK_TYPE_P1                        IB(1)
         LINK_TYPE_P2                        ETH(2)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               False(0)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               False(0)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             PXE(1)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)

For the LINK_TYPE fields, IB/1 = infiniband, ETH/2 = ethernet

To change the mode, we can use mstconfig -d 09:00.0 set LINK_TYPE_P1=2, Just make sure to replace with your PCI device id.

root@truenas:~# mstconfig -d 09:00.0 set LINK_TYPE_P1=2

Device #1:

Device type:    ConnectX3
Device:         09:00.0

Configurations:                              Next Boot       New
         LINK_TYPE_P1                        IB(1)           ETH(2)

 Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.

Per the note- you will need to reboot before this takes effect.


Just a quick guide on how to change the port mode of your ConnectX-3 NICs. This works for both the 10G and 40G variants.

Also- the port-mode wasn't the issue as to why it wasn't being displayed- Instead- the NIC was at a differnet PCI address location. I just needed to move over the config. :-/