Adding more NVMe to the r730XD¶
Adding 4 more enterprise NVMe drives to my r730XD.
This is just a short post about its troubleshooting, and.... what will replace it.
Amazon Affiliate Links Used
This post DOES include Amazon affiliate links. If you found this content useful, please consider buying the products displayed using the provided links.
You will pay the same amount as normal, however, it does provide a small benefit to me. This benefit is usually used to purchase other products and hardware for which I can review / blog about.
I do not display advertisements on this site. As such, the only compensation from this service, comes from affiliate links. I do not ask for, or even accept donations.
E-Bay Affiliate Links Used
This post DOES include EBay affiliate links. If you found this content useful, please consider buying the products displayed using the provided links.
You will pay the same amount as normal, however, it does provide a small benefit to me. This benefit is usually used to purchase other products and hardware for which I can review / blog about.
I do not display advertisements on this site. As such, the only compensation from this service, comes from affiliate links. I do not ask for, or even accept donations.
Getting Started¶
For my ceph cluster project, I needed to add the proper amount of storage to each node, to make a decently performing, highly available ceph-cluster.
So, the first task, was to identify a good place to put them....
The problem is, my r730XD is already a tad on the full side.
The r730XD has 6 accessible PCIe slots. Three of which are half height, the rest are full height.
Current slot usage:
- x8 Half
- Coral TPU
- Empty x8 Half
- Empty x8 Half
- x16 Full
- ASUS Hyper M.2. Quad NVMe, Requires motherboard bifurcation
- 1T Samsung 970 evo
- 1T Samsung 970 evo
- 1T Samsung 970 evo plus
- 1T Samsung 970 evo plus
- (All four of these NVMe are assigned to a striped-mirrors ZFS pool.)
- ASUS Hyper M.2. Quad NVMe, Requires motherboard bifurcation
- x8 Full
- Quad NVMe Card. Does not require bifurcation. Fits 4x NVMe.
- 500G Samsung 970 Evo (Proxmox boot drive)
- empty
- empty
- empty
- Quad NVMe Card. Does not require bifurcation. Fits 4x NVMe.
- x16 Full
Instead of putting the new NVMe into the PLX switch which had three empty slots, I felt it would be best to use the empty slots 1 and 2, and enable bifurcation.
Since, this is being used for my ceph project, enterprise NVMes were a must here. I went with 4x used enterprise Samsung PM963 1T NVMe.
For mounting the NVMe, I went with Chinese Dual M.2 PCIe Adapter. Note- requires motherboard bifurcation.
After all of the parts had arrived, the first order of business was to mount the NVMe into the adapter.
After the cards were loaded, I pulled out the right sled from the r730xd, and inserted the cards into slots 2 and 3.
Another angle.
And... after adding everything, we just need to enable bifurcation on the selected slots. This is handled through the BIOS.
Finally- I had the fun task of assigning the drives within Proxmox.
The "fun" part- Four of the NVMe are assigned for a pool hosted inside of my Unraid VM.
Can- you spot which of the drives are the new ones, and which are the old?
All done!