Adding more NVMe to the r730XD¶
Adding 4 more enterprise NVMe drives to my r730XD.
This is just a short post about its troubleshooting, and.... what will replace it.
Getting Started¶
For my ceph cluster project, I needed to add the proper amount of storage to each node, to make a decently performing, highly available ceph-cluster.
So, the first task, was to identify a good place to put them....
The problem is, my r730XD is already a tad on the full side.
The r730XD has 6 accessible PCIe slots. Three of which are half height, the rest are full height.
Current slot usage:
- x8 Half
- Coral TPU
- Empty x8 Half
- Empty x8 Half
- x16 Full
- ASUS Hyper M.2. Quad NVMe, Requires motherboard bifurcation
- 1T Samsung 970 evo
- 1T Samsung 970 evo
- 1T Samsung 970 evo plus
- 1T Samsung 970 evo plus
- (All four of these NVMe are assigned to a striped-mirrors ZFS pool.)
- ASUS Hyper M.2. Quad NVMe, Requires motherboard bifurcation
- x8 Full
- Quad NVMe Card. Does not require bifurcation. Fits 4x NVMe.
- 500G Samsung 970 Evo (Proxmox boot drive)
- empty
- empty
- empty
- Quad NVMe Card. Does not require bifurcation. Fits 4x NVMe.
- x16 Full
Instead of putting the new NVMe into the PLX switch which had three empty slots, I felt it would be best to use the empty slots 1 and 2, and enable bifurcation.
Since, this is being used for my ceph project, enterprise NVMes were a must here. I went with 4x used enterprise Samsung PM963 1T NVMe.
For mounting the NVMe, I went with Chinese Dual M.2 PCIe Adapter. Note- requires motherboard bifurcation.
After all of the parts had arrived, the first order of business was to mount the NVMe into the adapter.
After the cards were loaded, I pulled out the right sled from the r730xd, and inserted the cards into slots 2 and 3.
Another angle.
And... after adding everything, we just need to enable bifurcation on the selected slots. This is handled through the BIOS.
Finally- I had the fun task of assigning the drives within Proxmox.
The "fun" part- Four of the NVMe are assigned for a pool hosted inside of my Unraid VM.
Can- you spot which of the drives are the new ones, and which are the old?
All done!