Skip to content

Home

Jump to bottom
Brent Roskos edited this page May 4, 2021 · 11 revisions

Using Intel N3000 FPGA with RHOSP16.1 for vRAN.

for this project we tested two versions of Intels N3000 FGPA cards.

  • N3000-2
  • N3000-N

First things first, we need to build and an image for our RHOSP16.1 compute node with the RHEL-8.2-RT Kernel.

Here is an example script that we used to create our custom compute image with the tools and configs that we needed:
3.3-create-upload-rt-image.sh

This script closely follows the process outlined in the RHOSP 16.1 docs for creating a RT compute with a few additions for our project:

  • We've customized it to use satellite as this was desired for our environment

  • lines 26-36 inserts a custom script that runs at boot time that sets up Cache Allocation Technology (CAT), and attempts to set smp_affinity. Ensure that lines 29 and 30 are customized for your processors. We used Intel Xeon Gold 6248 (20 core) processors.

  • Line 42 sets the server to use biosdevname instead of the newer RHEL8 Consistent device naming scheme. We've noted the bug in the script for reference. In summary, consistent device naming was failing to correctly identify and enumerate the FGPA NICs when using 2x1X25-2fvl firmware, resulting in inconsistent names like eth2, eth13, etc instead of the desired names begining with ens. Biosdevnames works around this by having the system use the older {pci-bus}{port} naming scheme. This will result in a system with an N3000 card in PCI slot 1 having interfaces named: p1p1, p1p2, p1p3,p1p4. In this confiuration, p1p1 and p1p3 will have the active links and are the ports we will use.

  • The script referenced above also runs rt.sh which we've added a few things too:

    • kernel-devel is added (the non-rt version) due to a dependency on this for the intel-fpga tools.
    • we install intel-cmt-cat for CAT tuning
    • we install config-fpga which is not supported by Red Hat today, it's a custom package that I built to enable and configure the FEC accellerator VFs on the N3000 so they can be properly passed to the VDU guest VMs via PCI-Passthrough.
      Note: its probably easiest to clone down the entire "dell-lab" repository and get the config-fpga rpm out of the config-fpga folder

Now deploy the compute node

Here is an example of our vdu compute node parameters

Install the Intel Accelleration stack v1.3.1

  • Follow the guide from Intel here for section 4.1.2.1
    • Be sure to disable the EPEL repository after the FPGA tools get installed.
  • Identify your card per section 5
  • I would avoid updating the xl710 driver to the Intel version in section 6. We found the one that shipped with RHEL worked fine. This is just cleaner for support. If there is a problem, end users can get support from Red Hat if we stick to the versions it ships. Then again, if something doesn't work or whatever, it may be necessary to use the Intel version for a period of time, we just avoid if where we can. The firmware can be updated as directed without support issues as RH doesn't ship firmware here.
  • Ensure we have the current fpga firmware per section 8.1
  • Ensure we have the correct bitstream. When running FEC for 4G with the sample Intel provided bitstream, the bitstream id will be:
    Bitstream ID: 0x2392950A010501
    (find the bitstream id with: fpgainfo fme).
    If using software vendor provided bitstream then check with software vendor for the proper ID.
    During our trial we used the Intel provided bistream with the file name: turbo4g_2x1x25g_2fvl_unsigned_20ww01.bin

Validation

  • Use:
    lshw -c network -businfo |grep FPGA
    To verify that the NICs are properly enumerated. If any are named "eth" then see the section above regarding biosdevnames
  • Use:
    lspci -vd1172:
    and note the following:
    • For 4G, you should expect to see a device returned with the id 1172:5052 (PF), if not you don't have the right bistream
    • You should also see 2 devices with the id 1172:5050 (VF), if not then the FEC Virtual Functions (VFs) are not getting created.
      Review the information above for the unsupported config-fpga.rpm
  • If you build VMs and put them on the Fronthaul network but can't ping their fronthaul IPs then ensure the VDU compute host has the following rpm: opae-intel-fpga-driver.rpm. This is provided as part of the Intel Accelleration stack discussed above and is a requirement during runtime. Most of the other RPMs from the acelleration stack are for programming and monitoring the card but without opae-intel-fpga-driver on the host VMs connected to FPGA provided NIC VFs will not respond to pings.

Clone this wiki locally