I want to install GPU on LX2160A to train AI data,but I don’t know how to install a NVIDIA GPU on LX2160A.
Pls someone could tell me how to operate with it?
Currently we do not support Nvidia cards on the LX2160a, at least with Nvidia’s proprietary binary driver. I have made some slight success with the new open kernel driver and Nvidia’s latest binary driver release, but it is still very unstable and requires a Turing generation card or newer. We are hoping that soon Nvidia will be able to provide better support for their hardware on Aarch64 systems.
OK，thx！
What about AMD GPU card to install on LX2160A ? Could some operation advice give to me?
Honestly I haven’t done much regarding training on the AMD GPUs. In general you are going to need a Navi based card or newer, so you can use ROCm on the LX2160a. The amdgpu kernel module currently does not support Aarch64 for Navi and newer cars, as they require some floating point instructions in the kernel and the code needs to be re-organized so these functions are compiled with different flags and can run safely in the kernel without corrupting the userspace stack.
Most likely I will publish my patches for the nvidia binary driver first just because I already have a card that is working. The amdgpu kernel work is on my radar but GPU availability and pricing hasn’t made it a priority over the past year or so.
Have you already purchased a card, or are you looking to use hardware you already have?
Yes, I have some Nvidia GPU,such as GTX3080 Ti 、P400、A100. But no AMD GPU.
Most likely I will publish my patches for the nvidia binary driver first just because I already have a card that is working.
When will you push your patch?
Not really sure. Most likely I will get back around to testing nvidia’s drivers more this weekend.
Nvidia released a driver update. I will test it and then push a howto guide which is “use at your own risk”
Hi Jon, any update on this?
I will try a RTX A2000 on a HoneyComb LX2 running Ubuntu Server 22.04.3.
Thanks.
Hi, I’ve recently replaced my main PC GPU and now have a spare NVidia GTX 1660Ti which I just installed into my Honeycomb LX2K.
Here are the steps I used on Ubuntu 22.04 LTS:
Install Ubuntu open NVidia drivers
# Install server driver since this is a headless workstation,
# if using as workstation, install nvidia-driver-535-open (haven't tested this yet)
sudo apt install nvidia-driver-535-server-open
# Add configs
cat <<'EOF' | sudo tee /etc/modprobe.d/nvreg_fix.conf
options nvidia NVreg_OpenRmEnableUnsupportedGpus=1
EOF
cat <<'EOF' | sudo tee /etc/modprobe.d/nvidia.conf
options nvidia-drm modeset=1
EOF
# Blacklist noveau driver
cat <<'EOF' | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
options nouveau modeset=0
EOF
# Update initramfs
sudo update-initramfs -u
Reboot so new modules are correctly loaded
After reboot, run nvidia-smi to confirm your system detected the GPU.
Install Docker (if needed/used)
for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
# Add the repository to Apt sources:
echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
Install NVidia container runtime and configure Docker to use it
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \\n && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \\n sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \\n sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
Check GPU in Docker
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
# Running a MNIST trainning in Docker using GPU
docker run --runtime=nvidia --gpus all -it --rm -v $(pwd):/work -w /work --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 nvcr.io/nvidia/pytorch:23.10-py3
wget https://github.com/pytorch/examples/raw/main/mnist/main.py -o mnist_pytorch.py
time python3 mnist_pytorch.py --epochs=1
2 Likes
Apologies if there is a newer thread on this topic, I didn’t come across it. I’m wondering if there has been any progress with having NVIDIA GPUs play nice with this board. For my own usecase I’m not concerned with pixels, I’d just like to be able to dev and debug CUDA/C++ code on device..
If you are using the EFI firmware then there should be no issues using an Nvidia card with the newer open source kernel module and driver. This should also work with u-boot and device-tree but I have not specifically tested this use case.
Hi Jon, Thanks for the confirmation. I finally got around to transferring my RTX3060ti over to the Honeycomb. After multiple attempts I keep coming back to the same place which is essentially:
> nvidia-smi
No devices were found
Previously, I had the recommended AMD Navi based GPU so I have sudo apt remove --purge "*amdgpu*" the AMD drivers and removed the AMD GPU related arguments from the kernel args. I’ve done similar with "*nvidia*" to ensure that I haven’t got conflicting configurations hanging around, and I am very confident that I do not.
Currently I have installed 570 using the offline installer NVIDIA-Linux-aarch64-570.153.02.run and choosing the GPL/MIT option for the modules option. The installer completes and cheerfully reports success.
# NV Detect indicates that this is the correct driver version, as does manually selecting the driver from nv.com
> /usr/bin/nvidia-detector
nvidia-driver-570
> nvidia-smi --version
NVIDIA-SMI version : 570.153.02
NVML version : 570.153
DRIVER version : 570.153.02
CUDA Version : 12.8
> lspci
0002:01:00.0 Non-Volatile memory controller: Phison Electronics Corporation E18 PCIe4 NVMe Controller (rev 01)
0004:01:00.0 VGA compatible controller: NVIDIA Corporation Device 2803 (rev a1)
0004:01:00.1 Audio device: NVIDIA Corporation Device 22bd (rev a1)
> lsmod | grep -i nvidia
nvidia_uvm 1658880 0
nvidia_drm 122880 0
nvidia_modeset 1847296 1 nvidia_drm
nvidia 11816960 2 nvidia_uvm,nvidia_modeset
drm_kms_helper 344064 1 nvidia_drm
drm 659456 4 drm_kms_helper,nvidia,nvidia_drm
> sudo dmesg | grep -i nvidia
[ 755.047626] nvidia: loading out-of-tree module taints kernel.
[ 755.095012] nvidia-nvlink: Nvlink Core is being initialized, major device number 505
[ 755.098892] nvidia 0004:01:00.0: Adding to iommu group 9
[ 755.100471] nvidia 0004:01:00.0: enabling device (0000 -> 0003)
[ 755.100498] nvidia 0004:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 755.142981] NVRM: loading NVIDIA UNIX Open Kernel Module for aarch64 570.153.02 Release Build (dvs-builder@U22-I3-AF02-06-3) Tue May 13 16:16:49 UTC 2025
[ 755.235154] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0004:01/0004:01:00.1/sound/card0/input1
[ 755.236439] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0004:01/0004:01:00.1/sound/card0/input2
[ 755.237139] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0004:01/0004:01:00.1/sound/card0/input3
[ 755.237581] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0004:01/0004:01:00.1/sound/card0/input4
[ 755.237919] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0004:01/0004:01:00.1/sound/card0/input5
[ 755.238154] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0004:01/0004:01:00.1/sound/card0/input6
[ 755.238502] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0004:01/0004:01:00.1/sound/card0/input7
[ 755.262075] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for aarch64 570.153.02 Release Build (dvs-builder@U22-I3-AF02-06-3) Tue May 13 16:07:25 UTC 2025
[ 755.279644] [drm] [nvidia-drm] [GPU ID 0x00040100] Loading driver
[ 755.279653] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0004:01:00.0 on minor 0
[ 763.547183] audit: type=1400 audit(1749046674.715:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=1951 comm="apparmor_parser"
[ 763.547196] audit: type=1400 audit(1749046674.715:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=1951 comm="apparmor_parser"
[ 1124.959851] The NVIDIA GPU driver for AArch64 has not been qualified on this platform
So, I’m getting a bit stuck. I’ve followed most of the tips I was able to mine from other forums, in particular the NVIDIA forums and, frankly, I think my config is pretty good and consistent … apart from the bit where it doesn’t work.
Are there any special kernel parameters I need, or need to get rid of for instance? I have arm-smmu.disable_bypass=0 iommu.passthrough=1but IIRC these are needed for something networking related.
And, yes, nouveau is blacklisted. I also added the nvreg_fix.conf file as suggested by carlosedp, above.
Looking forward to any Honeycomb specific tips, or any other tips and pointer for that matter!
Have you added the module parameter for your nvidia module that allows it to initialize on unsupported platforms? If you run modinfo nvidia you should see it there.
I’m not seeing any kernel parameters related to ‘platform’, unsupported or otherwise. Did you mean unsupported GPUs, that one is set true.
More generally, the RTX3060ti is a PCIe Gen4 board. It’s backward compatible in the sense that if you plug it into a mobo with PCIe Gen3 it will work so long as the mobo accepts it. Presumably some handshake must occur for them to interoperate.
I know this to be true since the system that I took the GPU from is PCIe Gen3 only. What I don’t know is whether the LX2160A is willing or able to play nicely with a PCIe Gen4 NVIDIA GPU.. Any thoughts?
also do you have the smmu or iommu in passthrough? I had to have the iommu enabled for the nvidia module to work properly when I tested it.
I’m in a similar boat, having followed the advice in this thread, though I’m running Fedora 42.
Running this UEFI firmware from an SD Card: https://solid-run-images.sos-de-fra-1.exo.io/LX2k/lx2160a_uefi/lx2160acex7_2000_700_3200_8_5_2_sd_ee5c233.img.xz
Card is a GeForce RTX 4060.
$ lspci
0002:01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
0004:01:00.0 VGA compatible controller: NVIDIA Corporation AD107 [GeForce RTX 4060] (rev a1)
0004:01:00.1 Audio device: NVIDIA Corporation AD107 High Definition Audio Controller (rev a1)
$ lsmod | grep -i nvidia
nvidia_drm 135168 0
nvidia_modeset 1867776 1 nvidia_drm
nvidia_uvm 3489792 0
nvidia 11743232 2 nvidia_uvm,nvidia_modeset
video 77824 1 nvidia_modeset
drm_ttm_helper 16384 1 nvidia_drm
$ nvidia-smi
[ 69.408971] NVRM: objClInitPcieChipset: *** Unable to get PCI port handles
[ 70.683070] NVRM: nvAssertOkFailedNoLog: Assertion failed: Generic Error: Invalid state [NV_ERR_INVALID_STATE] (0x00000040) returned from sysmemData == vidmemData @ mem_mgr.c:424
[ 70.683175] NVRM: nvAssertOkFailedNoLog: Assertion failed: Generic Error: Invalid state [NV_ERR_INVALID_STATE] (0x00000040) returned from status @ mem_mgr.c:4050
[ 70.686174] NVRM: nvAssertOkFailedNoLog: Assertion failed: Generic Error: Invalid state [NV_ERR_INVALID_STATE] (0x00000040) returned from memmgrInitCeUtils(pMemoryManager, NV_FALSE, NV_TRUE) @ mem_mgr.c:473
[ 70.686201] NVRM: nvAssertFailedNoLog: Assertion failed: 0 @ kernel_fifo.c:3121
[ 70.686217] NVRM: RmInitNvDevice: *** Cannot load state into the device
[ 70.686221] NVRM: RmInitAdapter: RmInitNvDevice failed, bailing out of RmInitAdapter
[ 70.691138] NVRM: nvAssertFailedNoLog: Assertion failed: listCount(&pKernelBus->virtualBar2[gfid].usedMapList) == 0 @ kern_bus_vbar2.c:346
[ 70.831240] NVRM: nvAssertFailedNoLog: Assertion failed: listCount(&pKernelBus->virtualBar2[gfid].usedMapList) == 0 @ kern_bus_vbar2.c:346
[ 70.831526] NVRM: iovaspaceDestruct_IMPL: 1 left-over mappings in IOVAS 0x40100
[ 70.831544] NVRM: GPU 0004:01:00.0: RmInitAdapter failed! (0x25:0x40:1203)
[ 70.843823] NVRM: GPU 0004:01:00.0: rm_init_adapter failed, device minor number 0
No devices were found
Kernel Args:
rd.lvm.lv=fedora/root
console=ttyS0
CONFIG_AT803X_PHY=m
iommu.passthrough=1
arm-smmu.disable_bypass=0
rd.driver.blacklist=nouveau,nova_core
modprobe.blacklist=nouveau,nova_core
nvidia_drm.modeset=1
nvidia.NVreg_EnablePCIeGen3=1
pci=realloc=on
Any help would be appreciated!
Please try booting with u-boot and testing your Nvidia GPU. Nvidia drivers no longer work with the rootless PCIe mode we had to implement for ACPI to work in the mainline kernel as they wouldn’t accept an ECAM quirk for our hardware. This and having the iommu enabled and not in passthrough are generally required by their drivers to work.
1 Like
The full contents of /proc/cmdline is as follows:
BOOT_IMAGE=/vmlinuz-5.15.0-141-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro arm-smmu.disable_bypass=0 iommu.passthrough=1 usbcore.autosuspend=-1 libata.force=noncq
Regarding your new reply, to lollerbus, I don’t think I’m using U-Boot, or maybe I am. How can I check this and enable if necessary? BRs.
[Update] I’m pretty sure I’m not.
First experience with the LX2’s here for me and I did not realise the aarch64 nvidia support was so hit and miss. Have spent about half a day and have an Ubuntu 25.04 install working, but getting the nvidia driver to work on X/framebuffer is a whole nother thing.
root@solid-ds-01:~# nvidia-smi
Mon Jul 14 05:37:09 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.64.03 Driver Version: 575.64.03 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 5060 Off | 00000004:01:00.0 Off | N/A |
| 0% 47C P8 12W / 145W | 33MiB / 8151MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
Getting the driver installed seemed to be the easy bit - but getting it to work with X-org is a whole nother hell 
[ 5.252228] nvidia: loading out-of-tree module taints kernel.
[ 5.273845] nvidia-nvlink: Nvlink Core is being initialized, major device number 508
[ 5.282305] nvidia 0004:01:00.0: Adding to iommu group 9
[ 5.285787] nvidia 0004:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=io+mem
[ 5.286595] EXT4-fs (nvme0n1p2): mounted filesystem ba2d9089-06c1-4d31-a2b4-b3d5a41ec3c6 r/w with ordered data mode. Quota mode: none.
[ 5.360989] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for aarch64 575.64.03 Release Build (dvs-builder@U22-I3-B10-02-4) Wed Jun 25 18:11:26 UTC 2025
[ 5.368289] [drm] [nvidia-drm] [GPU ID 0x00040100] Loading driver
[ 5.641735] NVRM: nvAssertFailedNoLog: Assertion failed: 0 @ chipset.c:462
[ 5.658106] NVRM: nvAssertFailedNoLog: Assertion failed: 0 @ chipset.c:557
environment.
[ 8.650673] nvme nvme0: using unchecked data buffer
[ 17.176547] nvidia-modeset: ERROR: GPU:0: Failed to initialize DMA.
[ 17.228782] [drm] Initialized nvidia-drm 0.0.0 for 0004:01:00.0 on minor 1
[ 20.379883] nvidia-modeset: WARNING: GPU:0: Lost display notification (0:0x00000000); continuing.
[ 23.383440] nvidia-modeset: WARNING: GPU:0: Lost display notification (0:0x00000000); continuing.
[ 26.400083] [drm:nv_drm_atomic_commit [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00040100] Flip event timeout on head 0
[ 26.411287] nvidia 0004:01:00.0: [drm] fb0: nvidia-drmdrmfb frame buffer device
[ 29.472079] [drm:nv_drm_atomic_commit [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00040100] Flip event timeout on head 0
[ 32.544097] [drm:nv_drm_atomic_commit [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00040100] Flip event timeout on head 0
[ 130.656928] [drm:nv_drm_atomic_commit [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00040100] Flip event timeout on head 0
[ 133.729255] [drm:nv_drm_atomic_commit [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00040100] Flip event timeout on head 0
[ 136.801556] [drm:nv_drm_atomic_commit [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00040100] Flip event timeout on head 0
[ 139.873852] [drm:nv_drm_atomic_commit [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00040100] Flip event timeout on head 0
I am using custom built firmware from GitHub - evilrobot-01/HoneyComb and have X86EMU=1 enabled (this was a gotcha for me, and without it i never got HDMI output from my RTX5060).
Any tips to get further on this would be greatly appreciated. I am booting via EFI.
Hi Jon, you mean updating my current OS which is still on Ubuntu 22.04 will break the GPU support? I currently boot using EFI firmware from the SPI Flash.
Also am I supposed to keep the current drivers as is (not updating the driver too) to keep it working?