ClearFog CX LX2 progress - what works for me

In case it can be helpful for others, I’m sharing some of my success here.
-Hopefully no need to share failures. :wink:

I’ve built Ubuntu (2000_700_3200_8_5_2-bc46e34) and fitted a pair of 8GB Ballistix 3200 MHz SODIMMs (CL16).
I’m able to boot successfully via SD, eMMC and SPI from …

  1. eMMC
  2. SD-card (by adding an extra partition to the card’s unused space and writing the additional / modified ubuntu-core.ext4 there).
  3. SATA harddisk (WD10JFCX);

I now have 10Gbit ports brought up by systemd and I’ve also set static IP-addresses using nmcli.
I’m using Digitus DN-81200 with OM4; my connection is to a DGS-1510-20 switch (module works with this switch as well). I will expect DN-81201 to work as well (which is for singlemode fiber such as OS2).
I’ve 3D-printed an I/O-shield for a Micro-ITX cabinet, plus a few wire-organizers.
Using a Corsair VS450 PSU (bought second-hand) for now; planning on switching to a MeanWell IRM-60-12ST later on - perhaps with a PicoPSU, but maybe I can run the board on only the 12V rail.

Details on how I made the bootable SD-card …

# 1: Set up a handy newline variable:
# 2: Duplicate the ubuntu-core.ext4 image:
cp -RPp --sparse=always images/tmp/ubuntu-core.ext4 expanded.ext4
# 3: Make the duplicated image larger:
fallocate --length 2112M expanded.ext4
# 4: Resize the image in advance:
fdisk expanded.ext4 <<<"d${lf}n${lf}p${lf}${lf}131072${lf}${lf}y${lf}w${lf}"
# 5: Allow us to modify the new image:
e2fsck -f expanded.ext4
# 6: Change the UUID and label of the image (optional; don't use this fake "UUID"):
tune2fs -L '/' -U 'beefface-feed-fade-deaf-000000000001' expanded.ext4 <<< "Y"
# 7: Finally resize the image:
resize2fs expanded.ext4
# The image is now 2112MB in size, ready to be written and can be used as is.

# 8: Set an environment variable for the device, change the X according to your needs:
# 9: Write the lx2160acex7_2000_700_xx00-xxxxxxx.img file to the SD-card (I used dd on Linux).
dd status=progress if=lx2160acex7_2000_700_3200-bc46e34.img of=${dev}
# 10: Add a new partition on the card 2GB from the beginning, size=3GB:
sudo fdisk ${dev} <<<"n${lf}p${lf}${lf}2097152${lf}+3G${lf}w${lf}"
# 11: Write the expanded image to the second partition:
sudo dd status=progress if=expanded.ext4 of=${dev}2; sync

# 12: Mount the image and make a few modifications to it.
mkdir -p sd; sudo mount ${dev} sd
# 13: Get the PARTUUID ...
partuuid=$(lsblk -no partuuid ${dev})
# 14: Modify extlinux.conf to boot from our SD-card partition:
sed -Ei.orig -e 's/APPEND /APPEND arm-smmu.disable_bypass=0 iommu.passthrough=1 /' -e 's/root=PARTUUID=[a-fA-F0-9-]+/root=PARTUUID=$partuuid/' sd/extlinux/extlinux.conf
# 15: Unmount the card:
sudo umount sd; rmdir sd

You should now be able to boot from that card.
If that works, you’ll still need to run dhclient, then preferably unminimize.
You can optionally do another resize2fs on the card if you wish, but there’s enough room for unminimizing.
Once you’ve finished dhclient and unminimizing, I recommend creating a user …

adduser admin
usermod -aG sudo admin

… after that you can log in via SSH (as you can not log in using root via SSH by default)
(feel free to replace admin by your own shortname)
The same image can be written to any partition on SATA port 0.
I prefer partition 10 (because I need some small fast partitions before the rootfs).

I made a few modifications in U-Boot (these are not necessary)

setenv bootcmd1 "$bootcmd"
setenv bootcmd_sata "scsi scan; boot_targets=SCSI0; run bootcmd1;"
setenv bootcmd "run bootcmd_sata;"
setenv bootdelay 3

-You still have enough time to stop the boot and issue a ‘run bootcmd1’ without reducing the boot_targets to SCSI0, so you can boot from USB, SD or eMMC.

Working cards and accessories:

  • Crucial Ballistix 2 x 8GB DDR4, CL16: BL2K8G32C16S4B, EAN: 649528824448
  • Multimode 10Gbit SFP+ Transceiver: Digitus DN-81200, EAN: 4016032324133
  • 1TB 2.5" WD RED Harddisk, Western Digital WD10JFCX, EAN: 718037804149
  • 64GB MicroSDXC: Kingston Canvas Select Plus SDCS2/64GB, EAN: 740617298697
  • 16GB MicroSDHC, Kingston Canvas Select SDCS/16GBSP, EAN: 740617275865
  • USB 3.1 card reader: Silicon Power SPU3AT3REDEL300W, EAN: 4713436123941

Crucial Ballistix gives me 11GB/s for reading and 9.3GB/s for writing and all memtester tests succeeded. Though the Ballistix work, they do not run on maximum speed as that would require them to run on 1.35V; only 1.2V is available. See Jon’s recommendation below; G.Skill makes some good quick RAM modules (and I plan on trying both G.Skill and Kingston later on).

Untested, but I assume these will work:

  • Singlemode 10Gbit SFP+ Transceiver: Digitus DN-81201, EAN: 4016032324140
  • Singlemode 1Gbit SFP Transceiver: Digitus DN-81001, EAN: 4016032305668
  • 40x40x11mm fan, 6.9CFM, 19dBA, 3-pin, Xilence XF031, EAN: 4044953500824
  • 2 x eSATA bracket, Logilink CS0006, EAN 4260113564141
  • M.2-to-PCIe x4 adapter, Delock DEL-62584, EAN: 4043619625840
  • M.2-to-U.2 adapter: Startech M2E4SFF8643, EAN: 065030872607
  • Crucial Ballistix 2 x 32GB SODIMM, BL2K32G32C16S4B, EAN: 649528824486
  • Multimode 40Gbit QSFP+ Transceiver: Digitus DN-81300
  • Multimode 100Gbit QSFP28 Transceiver: Digitus DN-81631, EAN:4016032483427

(The price of the DN-81631 is almost the same as the price of the DN-81300)

Note: Purchase OS2 LC-LC cables for singlemode, purchase OM3, OM4 or OM5 LC-LC for multimode.
Multimode transceivers will not work with OSx, only OMx. Singlemode transceivers will not work with OMx, only OSx.
Don’t purchase OS1, OM1 or OM2; these are too old, quite poor and usually too expensive.
DGS-1510-28X is a fine switch; it offers 4 SFP+ (10Gbit) ports and 24 RJ45 ports for a fair price, but you’ll likely not need that, as you can use the LX2 for both routing and switching.

… I’ll try and post updates as I advance. The following still needs to be solved:

  • Get docker properly installed (it installs, but still fails to start, can’t create bridge; see Jon’s reply below).
1 Like

The 10Gbit ports are not “persistent” through the kernel. These are virtual devices that need to be initialized from user-space on each boot. Generally most installations are using one-time systemd files to initialize the devices as they are detected through device-tree enumeration.

1 Like

Thank you - that was a very quick (and helpful) reply.
(I didn’t even finish editing before you replied).

Hi PacMan - Have you tried anything with your QSFP port? So far I’ve been struggling to get any builds that work for it

1 Like

Wick - I’m not there yet - but I do plan on getting a 100Gbit QSFP28 adapter from Digitus.
The DN-81631 should be able to do both 40Gbit and 100Gbit and it cost the same as DN-81300.
(The price of the DN-81631 is less than 100 Euro and less than 100USD).

Unfortunately I’ll be moving to a different town in two weeks, so I can’t do much work on the LX2 until I’m settled at the new place.

Though I do not have anything plugged into the QSFP port (yet), I’ve noticed that bc46e34 allowed me to bring up dpmac.3 … dpmac.6, however recent builds like ddab3ad does not.
-Try and see if bc46e34 will allow you to work with the QSFP port.

I did plan on updating the original post, but unfortunately it’s no longer editable.
Instead, I’ll post a few updates as replies.

I’ve now done a few simple tests on the Ballistix 2 x 8GB RAM.
memtester: Two shells running memtester 4G and one running memtester 2G simultaneously.
All reports ‘ok’.
using sysbench, I get around 9300 MB/s write, 11000MB/s read.
-So I believe Ballistix 3200MHz/CL16 can be used without problems.
If it’s possible to tune the RAM-settings, I’d be happy to see if I can get the CAS Latency closer to CL16 (I understand it’s set to about 23, but if both the LS2160 and the RAM is able to run at a lower latency, it’d be great to take advantage of it).

Otherwise - as posted elsewhere, I’m able to run ddab3da (latest build at present) with 10Gbit network and static IP.

I use nmcli for configuring the static IP addresses and a systemd “ls-addni.service” for setting up the interfaces; after powering on the CX, I can ssh into without first making any adjustments via the serial terminal.
I’m able to use the 10Gbit network via my D-Link DGS-1510-20, but I am also able to ping one port through another port (back and forth, two different static IP addresses).
-If I unplug one end of the optical cable, ping correctly reports “network is unreachable”.

Getting docker working is still pending. :wink:

The Crucial Ballistix 3200MHz/CL16 memory requires 1.35V to use the XMP2 profile that allows it to run at those speeds. Our memory controller only supports the JEDEC standard of 1.2V’s. This limits the Cruicial SODIMMs to their base SPD profile only.

1 Like

Aha, so the Kingston HyperX modules will actually be faster than the Ballistix in this case ?

maybe a bit. The HyperX modules are fastest in XMP mode 2, which is 2900/CL17. The fastest XMP2 enabled SO-DIMMs you can buy are G.Skill Ripjaws. It is tough to find their modules that support 3200/CL16, but their 3000/CL16 modules are more available and work great. That is what I run in my main workstation.

1 Like

Just took a quick look and apparently there is 2nd generation Kingston HyperX that runs 3200 CL18. I haven’t tested that yet.

1 Like

Though I found a G.Skill Ripjaws 2x16GB, 3200MHz/CL18/1.2V F4-3200C18D-32GRS, I think I’ll prefer higher speed over more memory; especially because it’ll help when using faster network interfaces.
The HyperX 3200/CL18 does sound interesting. =)
I did find the 2x8GB 3200MHz/CL16 F4-3200C16D-16GRS, but I’ll have to purchase from another country (still in EU, so no import fees). This is definitely a pair I’ll consider getting.

From what I understand you must rebuild with different SERDES to enable the QSFP on this board. You cannot use the provided images (8_5_2), as it does not have support for anything other than the 4xSFP+ ports.

That being said, none of my attempts at building have resulted in a bootable image. Let me know when you try-

1 Like

Thank you, I’ll definitely post here when I’ve gotten my QSFP transceiver and tried some builds.

Yea, Hopefully @jnettlet can spare some time to look at the ClearFog. I’ve tried SERDES 17_5_2 with partial success (100g does not seem supported yet). I get ubuntu partially booted, and it does detect my 25G transceiver in one of the SFP+ slots from, but unfortunately there’s something still missing, as it dumps me to emergency console before reaching login prompt.

As far as I understand, two of the SFP+ slots are 25G capable - or am I wrong ?
-When I look at the configuration possibilities, it looks like the QSFP is synchronized to the SFP+ block.
Eg. “dual 100Gbit” (which is confusing me a little if there are only support for 25Gbit in two SFP+ slots), “8 x 10Gbit”, “2 x 40Gbit”.

I’ve now successfully built povray on the CX LX2 for the CX LX2 and it runs well.

1 Like

Yesterday I started messing with changing the baudrate.
I added a few more baudrates:

230400, 307200, 460800, 614400 and 921600
I also set the default baudrate to 921600.

-After a couple of days of changing and rebuilding, I finally got U-Boot to switch the baudrate.
(There’s still some text being sent initially at 115200)

The result is that the CX now spend 4 second less before loading systemd, which means the CX boots in 20 seconds instead of 24 (measured from when I type “boot” and hit Return).

I attemted to use faster baudrates, but I didn’t succeed (don’t know if it’s my Mac or if the limit is somewhere else):

960000, 1000000, 1200000, 1228800, 1500000, 3000000

These - except for 1228800 - are supported by the FT230x according to the documentation.
(FT230x’s closest baudrate to 1228800 is 1263158 baud).

Today I just flashed the SPI successfully; I attempted this earlier but didn’t succeed back then.
I think U-Boot has a timing problem (not 100% sure, but it looks like it).
When I do a sf read 0xa0000000 0 4000000 some time passes and then I get a reset.
I’ve tried several lengths, and found that about 3c50000 does not reset, while 3c60000 does.

I added an extra partition to the SD-card using …

fdisk /dev/sdX
(end of last partition)

Then formatted + mounted the partition and finally copied the image:

mkfs.ext4 /dev/sdX3
mkdir sd; mount /dev/sdX3 sd
cp images/lx2160acex7_xspi*.img sd/
sync; umount sd; sync; rmdir sd

… You’ll of course need to replace /dev/sdX with the path to your SD-card.

Here’s how I successfully flashed and verified the SPI, having the above in mind:

mmc dev 0
ls mmc 0:3 /
load mmc 0:3 0xa4000000 lx2160acex7_xspi_2000_700_3200_8_5_2-ddab3ad.img
sf probe
sf update 0xa4000000 0 4000000
sf read 0xa0000000 0 2000000
sf read 0xa2000000 2000000 2000000
cmp.b 0xa0000000 0xa4000000 4000000

I had to make the workaround of reading the SPI in two halves because of that reset thing going on.

And now I’ll explain why I think U-Boot has a timing problem.
Earlier I noticed that once in a while, the board will not boot; it fails reading the ‘Image’ from the MMC.
When I cold-start, it usually succeeds, but when it fails, it’s usually after a ‘reboot’ or ‘reset’ command.
After I changed the baudrate to 921600, I noticed a lot more frequent failures of this kind.
I also tried setting down the baudrate before reading the SPI and that did help.
-So the U-Boot code might not behave correctly when talking to hardware (not verified).

In the end, I am now able to boot from my SATA harddisk (SATA0, partition 10) using the bootloader on the SPI. My boot-script is slightly altered, so I can boot a PARTUUID instead of the ‘first found /extlinux/extlinux.conf’ - because I find it completely silly that I can’t have a small partition which tells U-Boot which PARTUUID I want to load the ‘Image’ from (shortcoming in U-Boot).