# Nvme timeout on warm boot when switching sodimm **URL:** \ **Category:** NXP LX2160\ **Created:** [January 22, 2024, 3:48pm UTC](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803 "2024-01-22T15:48:30Z")\ **Posts on this page:** 15\ **Page:** 2
**Author:** ![jnettlet](https://dub1.discourse-cdn.com/flex005/user_avatar/community.solid-run.com/jnettlet/32/62_2.png) [@jnettlet](https://community.solid-run.com/u/jnettlet)\ **Post date:** [January 31, 2024, 6:22pm UTC](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803/21 "2024-01-31T18:22:45Z")
Yes it looks like all controllers need the same fix. The 64-bit prefetchable address spaces all have the same copy paste error. Only pcie5 should be using the 0xa4 0x00000000 pcie address space. All others should be 1 to 1 mapped to the second address. 0x84 0x00000000 0x8c 0x00000000 0x9c 0x00000000 ---
**Author:** ![joema](https://avatars.discourse-cdn.com/v4/letter/j/a9adbd/32.png) [@joema](https://community.solid-run.com/u/joema)\ **Post date:** [January 31, 2024, 8:50pm UTC](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803/22 "2024-01-31T20:50:17Z")
Applied the fixes, but regrettably, it doesn’t seem to change anything wrt the issue - still getting nvme timeouts on warm boot + non-ECC memory + when loading Image+dtb from 2nd eMMC partition. Attaching logs [emmc-2nd-emmc-nvme-fail-pcie-fixes-trimmed-upload.txt](https://community.solid-run.com/uploads/short-url/5CI6WWw4XHuE7YsvRXWiNbyq4Gg.txt) (39.4 KB) ---
**Author:** ![jnettlet](https://dub1.discourse-cdn.com/flex005/user_avatar/community.solid-run.com/jnettlet/32/62_2.png) [@jnettlet](https://community.solid-run.com/u/jnettlet)\ **Post date:** [February 1, 2024, 4:34am UTC](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803/23 "2024-02-01T04:34:45Z")
> [@joema](#): > > Applied the fixes, but regrettably, it doesn’t seem to change anything wrt the issue - still getting nvme timeouts on warm boot + non-ECC memory + when loading Image+dtb from 2nd eMMC partition. Does it work properly if you boot the system with just a single DIMM of the non-ECC memory? I am starting to think that the delay is enough loading off the 2nd eMMC partition is enough that NVMe is going into a sleep state and hanging (unfortunately this is a problem even on non-lx2160a systems), or there is some erratum we are missing with the older BSP. ---
**Author:** ![joema](https://avatars.discourse-cdn.com/v4/letter/j/a9adbd/32.png) [@joema](https://community.solid-run.com/u/joema)\ **Post date:** [February 1, 2024, 8:08am UTC](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803/24 "2024-02-01T08:08:12Z")
I’m using just a single DIMM of the non-ECC memory - this is how I am reproducing everything. Maybe u-boot leaves the SSD in some automatic-suspend mode, and unless the kernel boots fast enough, the SSD drops into low power mode? ---
**Author:** ![jnettlet](https://dub1.discourse-cdn.com/flex005/user_avatar/community.solid-run.com/jnettlet/32/62_2.png) [@jnettlet](https://community.solid-run.com/u/jnettlet)\ **Post date:** [February 1, 2024, 8:12am UTC](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803/25 "2024-02-01T08:12:52Z")
That is a possibility. On warm reset if you probe the nvme before booting linux does that make a difference? So break into u-boot command line and then `nvme scan` then boot. ---
**Author:** ![joema](https://avatars.discourse-cdn.com/v4/letter/j/a9adbd/32.png) [@joema](https://community.solid-run.com/u/joema)\ **Post date:** [February 1, 2024, 8:25am UTC](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803/26 "2024-02-01T08:25:14Z")
still failing (logs attached) [emmc-emmc-nvme-fail-after-uboot-scan.txt](https://community.solid-run.com/uploads/short-url/hVHKxGYsiZKLuzTgZZyrzFaEpft.txt) (48.9 KB) ---
**Author:** ![joema](https://avatars.discourse-cdn.com/v4/letter/j/a9adbd/32.png) [@joema](https://community.solid-run.com/u/joema)\ **Post date:** [February 1, 2024, 8:30am UTC](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803/27 "2024-02-01T08:30:55Z")
I also can’t possibly think of any logical explanation which might explain how non-ECC memory can be a trigger for this thing. ---
**Author:** ![jnettlet](https://dub1.discourse-cdn.com/flex005/user_avatar/community.solid-run.com/jnettlet/32/62_2.png) [@jnettlet](https://community.solid-run.com/u/jnettlet)\ **Post date:** [February 1, 2024, 8:44am UTC](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803/28 "2024-02-01T08:44:35Z")
Could you put the ECC memory back in and provide me with the cold and warm boot logs. Just so I can do a full comparison? ---
**Author:** ![joema](https://avatars.discourse-cdn.com/v4/letter/j/a9adbd/32.png) [@joema](https://community.solid-run.com/u/joema)\ **Post date:** [February 1, 2024, 11:25am UTC](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803/29 "2024-02-01T11:25:40Z")
Please find attached. Please note, that with ECC memory, both cold and warm boot work (and kernel is loaded from 2nd eMMC partition). I’ve also attached trimmed version of the logs, without the kernel timestamps, to make it easier to compare with tools like meld. [ecc-cold-boot.txt](https://community.solid-run.com/uploads/short-url/ci6wacHJ1SX1SNkKvuK0r9mT0Kn.txt) (75.7 KB) [ecc-cold-boot-trimmed.txt](https://community.solid-run.com/uploads/short-url/thNpCNpcGljGEB7FyT9W2hkb7Ck.txt) (62.1 KB) [ecc-warm-boot.txt](https://community.solid-run.com/uploads/short-url/1em4DxF1o65yfql653F0PxBkdhU.txt) (75.0 KB) [ecc-warm-boot-trimmed.txt](https://community.solid-run.com/uploads/short-url/sN9SUF4ICAU5X5BMnuY00QFwDHx.txt) (61.5 KB) ---
**Author:** ![jnettlet](https://dub1.discourse-cdn.com/flex005/user_avatar/community.solid-run.com/jnettlet/32/62_2.png) [@jnettlet](https://community.solid-run.com/u/jnettlet)\ **Post date:** [February 1, 2024, 11:36am UTC](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803/30 "2024-02-01T11:36:08Z")
This is very suspect. Even with ECC you are getting a training error on cold boot. NOTICE: BL2: v2.4(release):opx1000-v1.0-rc2-3-g7422fe804-dirty NOTICE: BL2: Built : 15:51:00, Nov 12 2023 NOTICE: UDIMM J722GU44J2320N7 NOTICE: DDR PMU Hardware version-0x1210 NOTICE: DDR PMU Firmware vision-0x1001 (vA-2019.04) ERROR: Execution FW failed (error code -5) ERROR: Calculating DDR PHY registers failed. PHY handshake timeout, ddr\_dsr2 = 0 ERROR: Found training error(s): 0x100 ERROR: Error: Waiting for D\_INIT timeout. ERROR: Writing DDR register(s) failed ERROR: Programing DDRC error ERROR: DDR init failed. NOTICE: Incorrect DRAM0 size is defined in platform\_def.h ÿNOTICE: BL2: v2.4(release):opx1000-v1.0-rc2-3-g7422fe804-dirty NOTICE: BL2: Built : 15:51:00, Nov 12 2023 NOTICE: UDIMM J722GU44J2320N7 NOTICE: DDR PMU Hardware version-0x1210 NOTICE: DDR PMU Firmware vision-0x1001 (vA-2019.04) NOTICE: DDR4 UDIMM with 1-rank 64-bit bus (x8) What RCW changes have you made? ---
**Author:** ![joema](https://avatars.discourse-cdn.com/v4/letter/j/a9adbd/32.png) [@joema](https://community.solid-run.com/u/joema)\ **Post date:** [February 1, 2024, 12:52pm UTC](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803/31 "2024-02-01T12:52:19Z")
Sorry, please ignore those erroneous lines: I removed the SODIMM and inserted it again and they were gone (that’s where you have the bootloader logs starting again after the “ÿ” character). I meant to remove them from the log but forgot. We don’t have those errors. We haven’t done any changes to rcw. We didn’t even change u-boot or kernel - we are basically using components from tag lsdk-21.08-sr-1.1 of the lx2160a\_build git as-is. ---
**Author:** ![jnettlet](https://dub1.discourse-cdn.com/flex005/user_avatar/community.solid-run.com/jnettlet/32/62_2.png) [@jnettlet](https://community.solid-run.com/u/jnettlet)\ **Post date:** [February 1, 2024, 12:55pm UTC](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803/32 "2024-02-01T12:55:49Z")
but you are choosing a different serdes protocol or at least that is what the logs are reporting. Using SERDES1 Protocol: 14 (0xe) Using SERDES2 Protocol: 2 (0x2) Using SERDES3 Protocol: 2 (0x2) ---
**Author:** ![joema](https://avatars.discourse-cdn.com/v4/letter/j/a9adbd/32.png) [@joema](https://community.solid-run.com/u/joema)\ **Post date:** [February 1, 2024, 1:50pm UTC](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803/33 "2024-02-01T13:50:37Z")
Sorry, you are right - we are doing that change. Our serdes configuration is: 14 for 100GE.1 and PCIe.2 x4, for the SSD. 2 for PCIe.3 x8, used by FPGA via xdma driver. 2 for PCIe.5 x8, same as SolidRun. Also used by FPGA. We have been running this way for years with no issues (but we never tried no-ECC memory before. even this time was accidental, we do plan to keep using ECC memory only). ```auto diff --git a/rcw/lx2160acex7/RCW/template.rcw b/rcw/lx2160acex7/RCW/template.rcw index d0c86f370..2112f3bba 100644 --- a/rcw/lx2160acex7/RCW/template.rcw +++ b/rcw/lx2160acex7/RCW/template.rcw @@ -1,5 +1,5 @@ #include #include -#include -#include +#include +#include #include ``` ---
**Author:** ![joema](https://avatars.discourse-cdn.com/v4/letter/j/a9adbd/32.png) [@joema](https://community.solid-run.com/u/joema)\ **Post date:** [February 4, 2024, 11:29am UTC](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803/34 "2024-02-04T11:29:50Z")
Hi, I reproduced the same issue with vanilla SolidRun images. I took the latest binaries from: [https://images.solid-run.com/LX2k/lx2160a\_build/20240104-bae3e6e](https://images.solid-run.com/LX2k/lx2160a_build/20240104-bae3e6e) Specifically, I used lx2160acex7\_2000\_700\_3200\_8\_5\_2-bae3e6e.img.xz. Exactly same issue: cold boot works, but warm boot with no-ECC has nvme timeouts. Interestingly, in this case, the filesystem is in eMMC, not SSD, and still we get nvme timeouts. Using clearfrog board - so everything sw and hw is SolidRun. Attaching logs, and again, also the trimmed version for easy “meld”-ing. [vanilla-cold-boot-trimmed.txt](https://community.solid-run.com/uploads/short-url/xnAmhuwSdNoFOHqAz3foYWhm2LQ.txt) (45.8 KB) [vanilla-warm-reset-no-ecc-trimmed.txt](https://community.solid-run.com/uploads/short-url/t0bBETYXwxlkR3UPBlpFf1hha5k.txt) (37.2 KB) [vanilla-warm-reset-no-ecc.txt](https://community.solid-run.com/uploads/short-url/k4toXoq4Mx1rT2ZZsPiOpMMZ6Ta.txt) (46.5 KB) [vanilla-cold-boot.txt](https://community.solid-run.com/uploads/short-url/1pBCPXZhhT61WJRsWVMzE6hzsch.txt) (55.8 KB) ---
**Author:** ![joema](https://avatars.discourse-cdn.com/v4/letter/j/a9adbd/32.png) [@joema](https://community.solid-run.com/u/joema)\ **Post date:** [February 5, 2024, 7:51am UTC](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803/35 "2024-02-05T07:51:09Z")
It seems the issue doesn’t reproduce using the old lx2160acex7\_2000\_700\_3200\_8\_5\_2-6a1498d.img.xz image (from Dec 2020), so this might be a regression introduced sometimes recently. Attaching logs. [old-vanilla-warm-after-warm-works.txt](https://community.solid-run.com/uploads/short-url/nnHCBuDMwxEunVou1BG5k1ZpYg3.txt) (174.6 KB) [Previous page](https://community.solid-run.com/t/nvme-timeout-on-warm-boot-when-switching-sodimm/803.md?page=1)