Poor quality in SFP+ transmitted signal

NXP LX2160
1

We are seeing quality issues in the transmitted optical signal, in the Clearfog board. The following observations point to poor signal being the problem, in contrast to a possible CPU or IO bottleneck:

  • High packet loss even with low data rates (50Mbps has up to 40% loss on some ports), independent of data rate so long as it is not high (above 1Gbps);
  • Packet loss which increases as the MTU increases;
  • Optical receiver registers a lot of reception errors, even though the power level is adequate.

Some ports perform better than others. In one SFP+ port we got ~28% packet loss at a low data rate. I’ve tried configuring TX equalization according to the following ticket: SerDes configuration for SFP+ modules , resulting in an improvement to ~23%, still way too high.

Maybe it’s possible to improve the quality further by experimenting with other equalization values. I’d like to get some insight in case there are any better options (maybe some missing/outdated driver?) to get around this issue.

The kernel was built using lx2160a_build.

This problem seems similar to what one commenter at 10G Ethernet throughput with optical SFP+ transceivers has reported. The problem for the person who opened that ticket, however, seems to be just CPU or IO bottleneck, unlike what we are seeing. The commenter has suggested removing some patches from the build, which I didn’t yet test.

We are yet to do a test using DAC.

2

We have the same issue. Plotting the signal eye diagram on external test equipment shows a closed eye. It is worse for some SFP+ ports than others but none of them are great. Seems to be bad signal integrity either on the COM or carrier. Our custom carrier PCB has the same problem as the ClearFog carrier so more likely to be the COM.

I spent some time trying various transceiver / serdes configuration options but never got reliable and satisfactory performance. Maybe there’s some software configuration that will improve it but I haven’t found any yet.

3

We were able to reduce the loss a lot by replacing the SFP+ module by the recommended Finisar FTLX8571D3BCL.

Now the loss is about 0.16% when before it was 28%. A big improvement, but still somewhat high for an optical link. With these loss rates I’m able to reach about 9.55 Gbps in UDP, but that falls a lot when using larger datagrams (1.81 Gbps with 8000 B datagrams) – which I’m not sure if it’s completely caused by the loss rate alone --.

4

could you share your eye diagrams? We have been trying to work out a good generic solution for the signalling, but the integrity as your posts points out seems to vary a lot between SFP+ modules. Most likely this is one of the reasons that Cisco, Dell, Broadcom etc have their “certified” modules.

5

Here is an eye diagram from one of the Clearfog 10G SFP+ cages to an external device (An AMD Xilinx FPGA).

image
image1240×670 56.4 KB

Note the small size of the blue low-BER region in the centre of the plot.

And the same FPGA to another of the Clearfog 10G cages. This one is better but not great.

image
image1238×671 56 KB

For comparison here are the eye diagrams observed from other equipment:

Firstly, connecting the FPGA to another FPGA:

image
image1238×671 50.5 KB

Secondly, a FPGA connected to a third party piece of equipment (a network switch):

image
image1234×673 51.1 KB

We’ve expended a significant amount (>2 months) of professional SW and HW development effort on this and haven’t managed to significantly improve things. Here are some of our observations:

  1. Poor SI exists with a mixture of optics and DAC cables, including the recommended SFPs in the WIKI.
  2. Some SFP+ ports exhibit better BER performance than others
  3. We see similar SI issues when the Clearfog carrier is replaced with our own custom PCB.
  4. Changing SERDES SW configuration settings doesn’t help.

Our view is the SI is probably due to the CoM as we see similar results for two different carrier / motherboards. Our inability to get good SERDES performance is leading us to develop our own PCB and layout for the LX2160 rather than buying CoMs.

6

We have further improved loss rates to <0.01% by replacing the receiver with a Finisar module as well. Still not perfectly zero loss, but a great improvement overall.