QSFP and SFP+ build

NXP LX2160
1

I’ve been happily playing with the pre-built images on my recently received rev2.1 clearfog, and I’m struggling a bit to build a new image with a different SERDES config that enables QSFP and SFP+

  1. I have a QSFP-SFP28 converter which puts 25Gb on channel 0. Can I use something like SD1=21 to enable two of the SFP+ ports, and still have functional QSFP at 25Gb? or does a converter only work with SD1=13,14 and fallback to 25Gb compat?
  2. I would have happily tested all this already, but it seems I cannot build bootable sd images with either uefi or build github projects. I’ve tried baremetal, baremetal on aarch64 thunderX2, and dockerized (both x86 and aarch64 native). The images that are produced look sane to me: same size and similar contents as those provided by solidrun, but when I try and boot them, the clearfog board seems to sit in an unhappy boot loop cycling power every second. Am I missing something here?

Does anyone have tips for producing functional images from github, or would be willing to post a few UEFI images with different SERDES configs for us clearfog users?

2

Just in very recent kernels has NXP started added dynamic phy reconfiguration for SFP/SFP+ modules. This is not extended out to the qsfp28 options 100/50/25 or 40. This along with the recipe NXP has documented here, https://www.nxp.com/docs/en/application-note/AN13022.pdf it should be possible to create an RCW and mc-firmware files that expose both 25Gbps as well as all 4 of the SFP+ ports on the ClearFog. This was recently brought up to me by a couple of customers and I will work on a solution.

As for not being able to build the image that boots. Are you just trying to build the default stock timings? Perhaps you can share the command you are running, and the distro you are building on. The containerized build should be very self contained and should provide consistent images to the ones we build in our CI.

3

Hi Jon,

Thanks for your reply, that’s exciting news! I look forward to playing with that once I can get up and running. I’m still suspicious about running 25Gb/s on lane0, as NXP claims in AN12950:

Only MACs 3, 4, 5, 6, 9, and 10 support 25 GbE. Only MACs 1 and 2 support 50 GbE and 100 Gb

Hopefully that does not mean using SD1=13,14 (on MAC1) will not support 25Gb lane0 (ie, 1-3 NC). To be tested alongside SD1=17,21.

Anyways, here’s my logbook on building uefi:

x86 Centos8s host tools:

sanity check:

$ DDR_SPEED=3200 SERDES=8_5_2 ./runme.sh
...
Built: images/lx2160acex7_2000_700_3200_8_5_2_sd_ee5c233.img
$ sudo dd if=images/lx2160acex7_2000_700_3200_8_5_2_sd_ee5c233.img of=/dev/rdisk2 bs=4096

Flashing this boots UEFI correctly, same as distributed builds. All good so far.

Now try SERDES listed in header of runme.sh, this should work right?

DDR_SPEED=3200 SERDES=13_5_2 ./runme.sh
...
Built: images/lx2160acex7_2000_700_3200_13_5_2_sd_ee5c233.img
$ sudo dd if=images/lx2160acex7_2000_700_3200_13_5_2_sd_ee5c233.img of=/dev/rdisk2 bs=4096

Board does not boot, power cycles repeatedly (all lights/fan pulsing).

To make sure, I repeated the above steps using the docker build environment, to the same results. 8_5_2 will boot, but not 13_5_2 (or 18_5_2, or 21_5_2, etc. ) :frowning:
For completeness:

docker run -v "$PWD":/work:Z --rm -it -e DDR_SPEED=3200 -e SERDES=13_5_2 lx2160a_uefi build

Does this work for you?

4

Oh this is for edk2. Changing SERDES for edk2 is not integrated into the build any longer since we don’t use NXP’s BSP layout. Instead the necessary files need to be integrated into edk2-non-osi and changed in the platform code. I will work on a write-up for you.

5

Hi Jon,

I see @YazanShhadySR pushed initial support for SD1=21 to the build repo, so I thought I’d give it a try. Unfortunately, it seems some customization in the FDT is missing.

  1. notice that CRC check fails on first boot(Loading Environment from MMC... *** Warning - bad CRC, using default environment ).

env print -a reveals:

serdes0=0
serdes1=0
serdes2=0

So I set these correctly & reboot to fix CRC boot errors. Not sure if this makes any difference.

=> env set serdes0 21
=> env set serdes1 5
=> env set serdes2 2
=> env save

.... reboot...

U-Boot 2021.04-00029-g5fd89bfb84 (Apr 07 2022 - 21:11:20 +0000)

SoC:  LX2160ACE Rev2.0 (0x87360020)

SERDES1 Reference: Clock1 = 161.13MHz Clock2 = 100MHz
SERDES2 Reference: Clock1 = 100MHz Clock2 = 100MHz
SERDES3 Reference: Clock1 = 100MHz Clock2 = 100Hz

Using SERDES1 Protocol: 21 (0x15)
Using SERDES2 Protocol: 5 (0x5)
Using SERDES3 Protocol: 2 (0x2)
  1. Try “simple case” firstboot fails, lacking the correct FDT:
...
mmc1(part 0) is current device
Scanning mmc 1:1...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
331 bytes read in 8 ms (40 KiB/s)
1:      primary kernel
Retrieving file: /boot/Image
41409024 bytes read in 1940 ms (20.4 MiB/s)
append: console=ttyAMA0,115200 earlycon=pl011,mmio32,0x21c0000 default_hugepagesz=1024m hugepagesz=1024m hugepages=2 pci=pcie_bus_perf root=PARTUUID=30303030-01 rw rootwait
Retrieving file: /boot/fsl-layerscape-lx2160a.dtb
Failed to load '/boot/fsl-layerscape-lx2160a.dtb'
Moving Image from 0x81100000 to 0x81200000, end=83a90000
ERROR: Did not find a cmdline Flattened Device Tree
Could not find a valid device tree
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
couldn't find /cpus
"Synchronous Abort" handler, esr 0x96000006
elr: 0000000082005010 lr : 0000000082004fd0 (reloc)
elr: 00000000fbd28010 lr : 00000000fbd27fd0
x0 : 0000000087f0037c x1 : 0000000073effc84
x2 : efbeaddeefbeadde x3 : 00000003fffffff0
x4 : 00000000efbeadde x5 : 0000000000000067
x6 : 0000000087f02dfd x7 : 0000000000000003
x8 : 0000000087f00000 x9 : 0000000000000008
x10: 0000000000000338 x11: 00000000fbb1a9cc
x12: 00000000000002fd x13: 00000000000002f0
x14: 0000000000000000 x15: 0000000087f00000
x16: 0000000000000002 x17: 0000000000000000
x18: 00000000fbb1ed80 x19: 00000000000002f0
x20: 00000000fbdb8935 x21: 0000000087f00000
x22: 00000000fbdc6a73 x23: 0000000000000000
x24: 00000000fbde6384 x25: 0000000000000000
x26: 0000000000000000 x27: 0000000000000000
x28: 00000000fbb2d6a0 x29: 00000000fbb1aa90

Code: a94153f3 a9425bf5 a8c47bfd d65f03c0 (b8616804)
Resetting CPU ...

Checking my /boot/ section indeed fsl-layerscape-lx2160a.dtb does not exist.

Will try again later with SD1=17 (which seems to have support in repo).

6

Yea, tried with SD0 13 and 17, both expect fsl-layerscape-lx2160a.dtb to exist in the ubuntu image /boot/ space. I guess nobody is using QSFP at 25/100G since you cannot build for it?

Any updates on this?

7

Hi there - is there any update on this? I tried building from the lx2160a_build in the mean time for at least some sort of baseline QSFP at any speed, but all my build combinations result in uboot expecting a dtb that doesn’t make it into the image.

I see in previous github issues that SD1=14 was tested before… If UEFI with support for QSFP isn’t in the near future, would it be possible to upload/send a previous tested image that supports 25/100G so it’s at least not a blocker?

2 Likes