# How to use PCIE1 **URL:** \ **Category:** OCTEON CN9130\ **Created:** [June 16, 2023, 7:59pm UTC](https://community.solid-run.com/t/how-to-use-pcie1/667 "2023-06-16T19:59:24Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![mpmurphy00](https://dub1.discourse-cdn.com/flex005/user_avatar/community.solid-run.com/mpmurphy00/32/133_2.png) [@mpmurphy00](https://community.solid-run.com/u/mpmurphy00)\ **Post date:** [June 16, 2023, 7:59pm UTC](https://community.solid-run.com/t/how-to-use-pcie1/667/1 "2023-06-16T19:59:24Z")
I’m trying to setup PCIe bus #1 for the CN9130 SOM. Background: I have ClearFog Base available to use, but not 64 bit ClearFog Pro, so I haven’t seen this bus work yet with a known software setup. The SOM is being connected to our own IO board which has two PCIe slots (as M.2 connectors). I have PCIe2, with SERDES 5 (“comphy 5”) working. PCIe1 is on SERDES 4, replicating the ClearFog Pro setup. In debug I currently have no card connected in the PCIE1 slot, as the present hurdle does not seem to about communication with a device. Software is Linux kernel 5.16.x. I’ve compared the files cn9130-cf-base.dts and cn9130-cf-pro.dts, and didn’t see anything that seems relevant that I don’t already have. Problem to solve: With my current device tree, the bootup hangs when I enable PCIe1. I’ve tracked that to this in the kernel code: \_/drivers/pci/controller/dwc/pcie\_designware.c \_/drivers/pci/controller/dwc/pcie\_armada8k.c Function armada8k\_pcie\_link\_up( ) executes, which wants to read register PCIE\_GLOBAL\_STATUS\_REG. That call goes to generic dw\_pcie\_read( ), which does a readl( ). Execution on the core in use seems to block there, something I didn’t realize was possible on that operation. Looking at readl( ), it’s an ARM64 specific macro which is wrapping a few assembly instructions. The console is not available and responsive yet, but the processor is still executing code - if one waits a while, one can occasionally see complaint messages from the kernel such as “[1376.578392] rcu: INFO: rcu\_preempt detected stalls on CPUs/tasks:” The address used for PCIE\_GLOBAL\_STATUS\_REG comes from a remap ( via devm\_pci\_remap\_cfg\_resource ), + the expected offset. The remap input came from cn9130.dtsi, f2620000 for PCIe1, or f264000 for PCIe2, and the output address is looks sane, when compared to the same code setting up PCIe2. I tried substituting a read from PCIE\_GLOBAL\_STATUS\_REG with previous register PCIE\_GLOBAL\_CONTROL\_REG. That read attempt also hangs the code. From this post, different processor: > [@Which PCIe controllers are enabled?](https://community.solid-run.com/t/which-pcie-controllers-are-enabled/374): > > The LX2160 manual and the device tree both refer to 6 controllers. However, mapping and attempting to read the configuration space for each of those causes the board to hang for most controllers, suggesting they are either not powered or simply not there. It seems that only 3 and 5 are enabled. Is that correct? Also, when I try to enumerate controller 3 I get some odd results on the first few BDF entries, which look like bridges but have a type 1 header. –Elad P.S., Keep in mind that I know … and something else I saw online, I get the impression that PCIe host controllers within the CN9130 might be disabled. I haven’t yet found in Marvell’s documentation how that might happen. Questions: - does the CN9130 have this feature, and would an IO memory read block if the host controller logic is disabled? - If so … how is the enable selected? ---
**Author:** ![jnettlet](https://dub1.discourse-cdn.com/flex005/user_avatar/community.solid-run.com/jnettlet/32/62_2.png) [@jnettlet](https://community.solid-run.com/u/jnettlet)\ **Post date:** [June 18, 2023, 5:58am UTC](https://community.solid-run.com/t/how-to-use-pcie1/667/2 "2023-06-18T05:58:13Z")
Please provide full dmesg output of the failed boot. My guess is that device-tree has mapped that slot as PCIe, but perhaps TF-A or u-boot is still setting it up as SATA causing the hang. SATA was the default setting for the other variants of the CF-PRO. ---
**Author:** ![mpmurphy00](https://dub1.discourse-cdn.com/flex005/user_avatar/community.solid-run.com/mpmurphy00/32/133_2.png) [@mpmurphy00](https://community.solid-run.com/u/mpmurphy00)\ **Post date:** [June 18, 2023, 5:57pm UTC](https://community.solid-run.com/t/how-to-use-pcie1/667/3 "2023-06-18T17:57:31Z")
Jon, Below is the boot spam from my current test build from power up, which includes U-boot’s commentary on “comphy”, and some of my debug messages. If ClearFog Pro and ClearFog Base have a different boot loader builds, then I should explore that. We are currently building the boot loader also, started I suspect from a ClearFog Base sample. Follow ups on this: looking online just now, I see a page of build instructions [https://solidrun.atlassian.net/wiki/spaces/developer/pages/200769556/CN913x+u-boot+atf+and+mv-ddr-marvell+-+Self+Build](https://solidrun.atlassian.net/wiki/spaces/developer/pages/200769556/CN913x+u-boot+atf+and+mv-ddr-marvell+-+Self+Build) showing git clone on u-boot source. The page is marked as archived, and perhaps is not specific to ClearFog Pro. Are there different source trees from Base vs Pro, or is theresome setup option to be made? I would diff the two and track down what needs to happen. Mike _BootROM - 2.03_ _Starting AP IOROM 1.02_ _Booting from eMMC 0_ _Found valid image at boot postion 0x004_ _lNOTICE: Starting binary extension_ _NOTICE: SVC: DEV ID: CN913x, FREQ Mode: 0x0_ _NOTICE: SVC: AVS work point changed from 0x2e2 to 0x2da_ _w/o ecc; strap value=0_ _4GB capacity; strap value=0_ _mv\_ddr: 14.0.0-gcfede8f8 (Mar 22 2023 - 11:37:54)_ _SSCG\_EN_ _Synopsys DDR43 PHY Firmware version: A-2017.11_ _SNPS DDR: 1D training passed_ _SNPS DDR: 2D training passed_ _SNPS DDR: training completed_ _dma memcmp pass_ _mv\_ddr: completed successfully_ _NOTICE: Cold boot_ _NOTICE: Booting Trusted Firmware_ _NOTICE: BL1: v2.3(release):R\_CLEARFOG\_IPsec-6\_0\_0\_E2\_2-2023-03-16-10-gcfede8f8 (Marvell-devel-18.12.0)_ _NOTICE: BL1: Built : 11:37:58, Mar 22 2023_ _NOTICE: BL1: Booting BL2_ _NOTICE: BL2: v2.3(release):R\_CLEARFOG\_IPsec-6\_0\_0\_E2\_2-2023-03-16-10-gcfede8f8 (Marvell-devel-18.12.0)_ _NOTICE: BL2: Built : 11:37:58, Mar 22 2023_ _NOTICE: SCP\_BL2 contains 7 concatenated images_ _NOTICE: Skipping MG CP1 related image_ _NOTICE: Load image to CP0 MG_ _NOTICE: Loading MG image from address 0x403114c Size 0xe0f0 to MG at 0xf2100000_ _NOTICE: Skipping MSS CP3 related image_ _NOTICE: Skipping MSS CP2 related image_ _NOTICE: Skipping MSS CP1 related image_ _NOTICE: Load image to CP0 MSS AP0_ _NOTICE: Loading MSS image from addr. 0x4044930 Size 0x1cfc to MSS at 0xf2280000_ _NOTICE: Done_ _NOTICE: Load image to AP0 MSS_ _NOTICE: Loading MSS image from addr. 0x404662c Size 0x5400 to MSS at 0xf0580000_ _NOTICE: Done_ _NOTICE: BL1: Booting BL31_ _lNOTICE: BL31: v2.3(release):R\_CLEARFOG\_IPsec-6\_0\_0\_E2\_2-2023-03-16-10-gcfede8f8 (Marvell-devel-18.12.0)_ _NOTICE: BL31: Built : 11:37:58, Mar 22 2023_ _U-Boot 2019.10-10.0.0 (Mar 23 2023 - 11:26:44 -0400)_ _Model: SolidRun CN9130 based SOM ClearFog Base_ _ERROR: mrvl\_sip\_smc\_handler: unhandled SMC (0x82000014)_ _ERROR: mrvl\_sip\_smc\_handler: unhandled SMC (0x82000014)_ _ERROR: mrvl\_sip\_smc\_handler: unhandled SMC (0x82000014)_ _ERROR: mrvl\_sip\_smc\_handler: unhandled SMC (0x82000014)_ _ERROR: mrvl\_sip\_smc\_handler: unhandled SMC (0x82000014)_ _ERROR: mrvl\_sip\_smc\_handler: unhandled SMC (0x82000014)_ _ERROR: mrvl\_sip\_smc\_handler: unhandled SMC (0x82000014)_ _ERROR: mrvl\_sip\_smc\_handler: unhandled SMC (0x82000014)_ _ERROR: mrvl\_sip\_smc\_handler: unhandled SMC (0x82000014)_ _ERROR: mrvl\_sip\_smc\_handler: unhandled SMC (0x82000014)_ _ERROR: mrvl\_sip\_smc\_handler: unhandled SMC (0x82000014)_ _ERROR: mrvl\_sip\_smc\_handler: unhandled SMC (0x82000014)_ _SoC: cn9130-A1; AP807-B0; CP115-A0_ _ERROR: mrvl\_sip\_smc\_handler: unhandled SMC (0x82000014)_ _Clock: CPU 2000 [MHz]_ _ERROR: mrvl\_sip\_smc\_handler: unhandled SMC (0x82000014)_ - ``` DDR 1200 [MHz]* ``` _ERROR: mrvl\_sip\_smc\_handler: unhandled SMC (0x82000014)_ - ``` FABRIC 1200 [MHz]* ``` - ``` MSS 200 [MHz]* ``` _LLC Enabled (Exclusive Mode)_ _DRAM: 4 GiB_ _Running in RAM - U-Boot at: 0x7fed4000_ - ``` Env at: 0x7ff8a117* ``` _Comphy chip #0:_ _Comphy-0: SATA1_ _Comphy-1: USB3\_HOST0_ _Comphy-2: SFI0 10.3125 Gbps_ _Comphy-3: SGMII1 1.25 Gbps_ _Comphy-4: USB3\_HOST1_ _Comphy-5: PEX2_ _UTMI PHY 0 initialized to USB Host0_ _UTMI PHY 1 initialized to USB Host1_ _Error reading output register_ _PCIE-0: Link down_ _NAND: 0 MiB_ _MMC: sdhci@6e0000: 0, sdhci@780000: 1_ _Loading Environment from EXT4… OK_ _get\_tlvinfo\_from\_eeprom: failed to read from eeprom!_ _get\_fdtfile\_from\_tlv\_eeprom: could not identify carrier, defaulting to Clearfog Pro!_ _Net:_ _Warning: mvpp2-0 (eth0) using random MAC address - 1e:5e:7e:74:4b:37_ _eth0: mvpp2-0 [PRIME]_ _Warning: mvpp2-1 (eth1) using random MAC address - ba:d0:f3:88:77:75_ _, eth1: mvpp2-1_ _Warning: mvpp2-2 (eth2) using random MAC address - 82:83:61:f4:fe:02_ _, eth2: mvpp2-2_ _Hit any key to stop autoboot: 0_ _32906833 bytes read in 2028 ms (15.5 MiB/s)_ _## Loading kernel from FIT Image at 09000000 …_ - Using ‘Edge-Crypto-Module’ configuration\* - Verifying Hash Integrity … sha256,rsa4096:firmware.sign\* - Verifying RSA4096 Signature …OK\* _+ OK_ - Trying ‘kernel-1’ kernel subimage\* - ``` Description: esm_32001092-001+ipsec+6.0.0.E3.3* ``` - ``` Created: 2023-06-18 17:30:35 UTC* ``` - ``` Type: Kernel Image* ``` - ``` Compression: gzip compressed* ``` - ``` Data Start: 0x09000140* ``` - ``` Data Size: 7695799 Bytes = 7.3 MiB* ``` - ``` Architecture: AArch64* ``` - ``` OS: Linux* ``` - ``` Load Address: 0x00200000* ``` - ``` Entry Point: 0x00200000* ``` - ``` Hash algo: sha256* ``` - ``` Hash value: fb0cc9001f76a01851535f09aeb9758e4e4a98d391312320e4c1fce98f61f3c0* ``` - ``` Sign algo: sha256,rsa4096:firmware.sign (required)* ``` - ``` Sign value: 229a0ac74754075d161ec43b477b0541c7b397a5c17f9d9590ffe3634277b9ed9322e7d7ba68a3c848af75379295445e86d4b2a86ca26542bf79653c8acf31c0d95ab8e123b065f83a8648a44ab7084efafe64866fe366ded005c9f62ed3663466d6e33d584a21c8c0fffce81587910db8166b061d1437631d96327c0aad7047fac9eaf86fcdabb6ea881e6e65cb8fe148132ba3959b78fdcfb63f34aa097e622b6f246123aa9286ffff9d6778dfe80a6f466162b3b99d7d34ce214a234e202c6aab1d66876247290c65fa1efe784714df571eabb950fffc04e79e24a7f7547e8b410a375ad15ca83b679c8325dc98d982115e82fce938101d0a116611b21f7cbebcbb1f25f766e454056c551ade4fdea045da826c762e7c28b4230cc2cb57a92349fd7f25486289857d514c3ecbf6a263989ebb5ee77376a87212a7f6f8ce27a67ed96ac8066d0099f93c9361c35cf5ea20aaa9c6f4c91013e3396398f047a66c6fdfb0ecb6cca64696b3740e6bd1bc9a651db33d76f108f5c6fe86ebf6ed7d56b69f7045647588f2380f184a703836ebaaa8130e7528c7815931fd055aecd85a3ed7b548f53b47682a43138347ac6dd7a57204ecb738d87020bc42bb6ce9abfca8da9d22be3daeee43e24b3e8f2400f04a1e3e5ae9d7fe0f82f9a0282400fbfcd5d3b576a7af833d93e4a5ed8d0d86303cc72c848df7be55820640b9957eef* ``` - ``` Timestamp: 2023-06-18 17:30:35 UTC* ``` - Verifying Hash Integrity … sha256+ sha256,rsa4096:firmware.sign\* - Verifying RSA4096 Signature …OK\* _+ OK_ _## Loading ramdisk from FIT Image at 09000000 …_ - Using ‘Edge-Crypto-Module’ configuration\* - Verifying Hash Integrity … sha256,rsa4096:firmware.sign\* - Verifying RSA4096 Signature …OK\* _+ OK_ - Trying ‘ramdisk-1’ ramdisk subimage\* - ``` Description: esm_32001092-001+ipsec+6.0.0.E3.3* ``` - ``` Created: 2023-06-18 17:30:35 UTC* ``` - ``` Type: RAMDisk Image* ``` - ``` Compression: uncompressed* ``` - ``` Data Start: 0x097572b4* ``` - ``` Data Size: 25183938 Bytes = 24 MiB* ``` - ``` Architecture: AArch64* ``` - ``` OS: Linux* ``` - ``` Load Address: unavailable* ``` - ``` Entry Point: unavailable* ``` - ``` Hash algo: sha256* ``` - ``` Hash value: d666f724a53f7bf80b5f6272032ffb82fc93e0ac25115fae1857f42f486ee8af* ``` - ``` Sign algo: sha256,rsa4096:firmware.sign (required)* ``` - ``` Sign value: 02a51af4e3938c3e09ccc010ccbeb3f0a37ebeb480c55a72bf61410bb1e9905f51c63c06aa77342a21f651bf93cb96dbedd263b5352d70216e9e0a141ff173e9f3110184361ac8bc2b15d06868dda8f45e0cb26a0874724a01f282c028192ffae6fbb36a6d11620d64d54113e6427060452e16504881b7fe98228d3f57abf7b17d01d79d95488df73a690c263083f70a30280cc02804bc7f0b87e4e24dc79b23ec2d0e6dca03a89e449feff24d5c408680510bc05cd01091620fca30950b907b799e671032780365ce7fa83dd13924f7eff68aabc297382d868e4fc4abcd4d6886e533de44d22018a7d193e2bcbc3b11a516959f164cc43b169aafb2321c9ea1d87a42df960f492621d9d42492c446688c9503e004bfb572c59daf828f0543e7d7165f39580f5a5b3f4f662baa6ba465ee30e9974363e852a73cc71ac0c6e29adeb4483b41e5e91dd4d1e85ea901e7a2dfad3a2438b31ebe6299805d2b5627feb583e86c0f82694f4053ca1c96e8bce18a45d8db82886602172adaf1f07d27e2d03df8c5e3024af1726ccff1213d70edc12a55c3ecf180308f3659d28881e3b470684722b5f93c37280c75140c0f523a54c571ec2d32a82fc9987a34c6db25576080dfc41c065a193d94aae8fd4fa511cf0487ac18e71685c71c07611d0fd513155ab4ad6f556767ed240b506ef4e3eca765fa6572086fefe2c60c43e7466738* ``` - ``` Timestamp: 2023-06-18 17:30:35 UTC* ``` - Verifying Hash Integrity … sha256+ sha256,rsa4096:firmware.sign\* - Verifying RSA4096 Signature …OK\* _+ OK_ _## Loading fdt from FIT Image at 09000000 …_ - Using ‘Edge-Crypto-Module’ configuration\* - Verifying Hash Integrity … sha256,rsa4096:firmware.sign\* - Verifying RSA4096 Signature …OK\* _+ OK_ - Trying ‘fdt-1’ fdt subimage\* - ``` Description: esm_32001092-001+ipsec+6.0.0.E3.3* ``` - ``` Created: 2023-06-18 17:30:35 UTC* ``` - ``` Type: Flat Device Tree* ``` - ``` Compression: uncompressed* ``` - ``` Data Start: 0x0af5bd10* ``` - ``` Data Size: 22427 Bytes = 21.9 KiB* ``` - ``` Architecture: AArch64* ``` - ``` Load Address: 0x02600000* ``` - ``` Hash algo: sha256* ``` - ``` Hash value: a09374e93c50a8d5c37898a58b6ac692c5cd3d24a0aaec67b72d580c9a7d53a0* ``` - ``` Sign algo: sha256,rsa4096:firmware.sign (required)* ``` - ``` Sign value: 8e5913bf4aadb4eecd9b08b758a938da9456a09671726e4170c3812bd6f458efe269506204216b63e1d070972d65673e3841de2dffd1f78dfac84ac5243656907b59518f068c13bbadac08a16bb983d8c16ebd47c284c0cf9f1fb9af1c009ad7bceb72a20ebf73812454edf12fef3b57670e6e55a0978a54b53051b6a3fad3da3ff03ad207a044b7817923856c7dc62a331778f180e363de470a41884727bf53a91d04093aeb9e1c36fc5c495e66c639af11ebad00841bcc7d5ca0093b38166c6d8ff7d6ce150563f2369ba2e38528fae48c3fd6819ef83c6570481ab73856fc3a692ea209053989a61cf8b4c77c6d9d14b89f860bb9755a7e1a5fa1bb905a61f6fd37398d96fb6e818bf6d80a428330e5285a9052ad7bfaf2c233a6b0c45e0f3f23d0da399dbfbb2ee0db9ca5da1de25bfb0cbd42a0396b7ffae72903889a8647dd2a3007eccebb443724212b4789437d430cfeb8b47b238d4eff2e3f42f5be1e34dc0e9b23a82b09d015b359b297584182df5813f37db10f4bd881cc8444c35423d01337ecf7877db8f86ba6a99f59a607c8355ee6517aad08256736b6e898f5c4931ea5df12fc1642a3f13da63d18fa4400fdd190133e29127069dd8d31f4772d9f97f247006a552e74b8da1d6390f545e475ece9e755431c4d922d1983c8591f5010e02eab354510a651de1fc7d280f2ae4fa8c491e6f9b7dbacc6dbbc7a* ``` - ``` Timestamp: 2023-06-18 17:30:35 UTC* ``` - Verifying Hash Integrity … sha256+ sha256,rsa4096:firmware.sign\* - Verifying RSA4096 Signature …OK\* _+ OK_ - Loading fdt from 0x0af5bd10 to 0x02600000\* - Booting using the fdt blob at 0x2600000\* - Uncompressing Kernel Image\* - Loading Ramdisk to 7ddb8000, end 7f5bc6c2 … OK\* - Loading Device Tree to 000000007ddaf000, end 000000007ddb779a … OK\* _Starting kernel …_ _[0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]_ _[0.000000] Linux version 5.16.13 (zeno@vm-ubuntumate22-04) (aarch64-buildroot-linux-gnu-gcc.br\_real (Buildroot -g6ac77208-dirty) 10.3.0, GNU ld (GNU Binutils) 2.36.1) #2 SMP PREEMPT Fri Jun 16 17:21:45 EDT 2023_ _[0.000000] Machine model: Ultra 3eTI FIPS carrier, SolidRun CN9130 SOM and Clearfog Base_ _[0.000000] efi: UEFI not found._ _[0.000000] NUMA: No NUMA configuration found_ _[0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x000000013fffffff]_ _[0.000000] NUMA: NODE\_DATA [mem 0x13f7d7b40-0x13f7d9fff]_ _[0.000000] Zone ranges:_ _[0.000000] DMA [mem 0x0000000000000000-0x00000000ffffffff]_ _[0.000000] DMA32 empty_ _[0.000000] Normal [mem 0x0000000100000000-0x000000013fffffff]_ _[0.000000] Movable zone start for each node_ _[0.000000] Early memory node ranges_ _[0.000000] node 0: [mem 0x0000000000000000-0x0000000003ffffff]_ _[0.000000] node 0: [mem 0x0000000004000000-0x00000000041fffff]_ _[0.000000] node 0: [mem 0x0000000004200000-0x00000000bfffffff]_ _[0.000000] node 0: [mem 0x0000000100000000-0x000000013fffffff]_ _[0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000013fffffff]_ _[0.000000] cma: Reserved 32 MiB at 0x00000000be000000_ _[0.000000] psci: probing for conduit method from DT._ _[0.000000] psci: PSCIv1.1 detected in firmware._ _[0.000000] psci: Using standard PSCI v0.2 function IDs_ _[0.000000] psci: MIGRATE\_INFO\_TYPE not supported._ _[0.000000] psci: SMC Calling Convention v1.2_ _[0.000000] percpu: Embedded 18 pages/cpu s36440 r8192 d29096 u73728_ _[0.000000] Detected PIPT I-cache on CPU0_ _[0.000000] CPU features: detected: Spectre-v2_ _[0.000000] CPU features: kernel page table isolation disabled by kernel configuration_ _[0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923_ _[0.000000] Fallback order for Node 0: 0_ _[0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1032192_ _[0.000000] Policy zone: Normal_ _[0.000000] Kernel command line:_ _[0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)_ _[0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)_ _[0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off_ _[0.000000] software IO TLB: mapped [mem 0x00000000ba000000-0x00000000be000000] (64MB)_ _[0.000000] Memory: 3968780K/4194304K available (10816K kernel code, 1778K rwdata, 4120K rodata, 2048K init, 546K bss, 192756K reserved, 32768K cma-reserved)_ _[0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1_ _[0.000000] rcu: Preemptible hierarchical RCU implementation._ _[0.000000] rcu: RCU event tracing is enabled._ _[0.000000] rcu: RCU restricting CPUs from NR\_CPUS=256 to nr\_cpu\_ids=4._ _[0.000000] Trampoline variant of Tasks RCU enabled._ _[0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies._ _[0.000000] rcu: Adjusting geometry for rcu\_fanout\_leaf=16, nr\_cpu\_ids=4_ _[0.000000] NR\_IRQS: 64, nr\_irqs: 64, preallocated irqs: 0_ _[0.000000] GIC: Adjusting CPU interface base to 0x00000000f022f000_ _[0.000000] Root IRQ handler: gic\_handle\_irq_ _[0.000000] GIC: Using split EOI/Deactivate mode_ _[0.000000] GICv2m: DT overriding V2M MSI\_TYPER (base:160, num:32)_ _[0.000000] GICv2m: range[mem 0xf0280000-0xf0280fff], SPI[160:191]_ _[0.000000] GICv2m: DT overriding V2M MSI\_TYPER (base:192, num:32)_ _[0.000000] GICv2m: range[mem 0xf0290000-0xf0290fff], SPI[192:223]_ _[0.000000] GICv2m: DT overriding V2M MSI\_TYPER (base:224, num:32)_ _[0.000000] GICv2m: range[mem 0xf02a0000-0xf02a0fff], SPI[224:255]_ _[0.000000] GICv2m: DT overriding V2M MSI\_TYPER (base:256, num:32)_ _[0.000000] GICv2m: range[mem 0xf02b0000-0xf02b0fff], SPI[256:287]_ _[0.000000] random: get\_random\_bytes called from start\_kernel+0x46c/0x658 with crng\_init=0_ _[0.000000] arch\_timer: cp15 timer(s) running at 25.00MHz (phys)._ _[0.000000] clocksource: arch\_sys\_counter: mask: 0xffffffffffffff max\_cycles: 0x5c40939b5, max\_idle\_ns: 440795202646 ns_ _[0.000000] sched\_clock: 56 bits at 25MHz, resolution 40ns, wraps every 4398046511100ns_ _[0.000100] Console: colour dummy device 80x25_ _[0.000254] printk: console [tty0] enabled_ _[0.000290] Calibrating delay loop (skipped), value calculated using timer frequency… 50.00 BogoMIPS (lpj=100000)_ _[0.000299] pid\_max: default: 32768 minimum: 301_ _[0.000325] LSM: Security Framework initializing_ _[0.000357] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)_ _[0.000371] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)_ _[0.000972] rcu: Hierarchical SRCU implementation._ _[0.001331] EFI services will not be available._ _[0.001429] smp: Bringing up secondary CPUs …_ _[0.001843] Detected PIPT I-cache on CPU1_ _[0.001878] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]_ _[0.002315] Detected PIPT I-cache on CPU2_ _[0.002339] CPU2: Booted secondary processor 0x0000000100 [0x410fd083]_ _[0.002759] Detected PIPT I-cache on CPU3_ _[0.002776] CPU3: Booted secondary processor 0x0000000101 [0x410fd083]_ _[0.002809] smp: Brought up 1 node, 4 CPUs_ _[0.002833] SMP: Total of 4 processors activated._ _[0.002838] CPU features: detected: 32-bit EL0 Support_ _[0.002842] CPU features: detected: CRC32 instructions_ _[0.009497] CPU: All CPU(s) started at EL2_ _[0.009519] alternatives: patching kernel code_ _[0.010110] devtmpfs: initialized_ _[0.011550] clocksource: jiffies: mask: 0xffffffff max\_cycles: 0xffffffff, max\_idle\_ns: 7645041785100000 ns_ _[0.011566] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)_ _[0.011859] pinctrl core: initialized pinctrl subsystem_ _[0.012081] DMI not present or invalid._ _[0.012254] NET: Registered PF\_NETLINK/PF\_ROUTE protocol family_ _[0.012791] DMA: preallocated 512 KiB GFP\_KERNEL pool for atomic allocations_ _[0.012894] DMA: preallocated 512 KiB GFP\_KERNEL|GFP\_DMA pool for atomic allocations_ _[0.013015] DMA: preallocated 512 KiB GFP\_KERNEL|GFP\_DMA32 pool for atomic allocations_ _[0.013057] audit: initializing netlink subsys (disabled)_ _[0.013126] audit: type=2000 audit(0.012:1): state=initialized audit\_enabled=0 res=1_ _[0.013283] thermal\_sys: Registered thermal governor ‘step\_wise’_ _[0.013287] thermal\_sys: Registered thermal governor ‘power\_allocator’_ _[0.013546] cpuidle: using governor menu_ _[0.013584] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers._ _[0.013628] ASID allocator initialised with 65536 entries_ _[0.013810] Serial: AMBA PL011 UART driver_ _[0.019990] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages_ _[0.020004] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages_ _[0.020009] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages_ _[0.020014] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages_ _[0.020219] cryptd: max\_cpu\_qlen set to 1000_ _[0.020446] ACPI: Interpreter disabled._ _[0.020855] ap0\_sd\_vccq: Bringing 3300000uV into 1800000-1800000uV_ _[0.021046] iommu: Default domain type: Translated_ _[0.021054] iommu: DMA domain TLB invalidation policy: strict mode_ _[0.021120] vgaarb: loaded_ _[0.021225] SCSI subsystem initialized_ _[0.021287] usbcore: registered new interface driver usbfs_ _[0.021305] usbcore: registered new interface driver hub_ _[0.021319] usbcore: registered new device driver usb_ _[0.021421] pps\_core: LinuxPPS API ver. 1 registered_ _[0.021426] pps\_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [giometti@linux.it](mailto:giometti@linux.it)_ _[0.021436] PTP clock support registered_ _[0.021482] EDAC MC: Ver: 3.0.0_ _[0.022034] clocksource: Switched to clocksource arch\_sys\_counter_ _[0.022099] VFS: Disk quotas dquot\_6.6.0_ _[0.022121] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)_ _[0.022203] pnp: PnP ACPI: disabled_ _[0.024369] NET: Registered PF\_INET protocol family_ _[0.024459] IP idents hash table entries: 65536 (order: 7, 524288 bytes, linear)_ _[0.025357] tcp\_listen\_portaddr\_hash hash table entries: 2048 (order: 3, 32768 bytes, linear)_ _[0.025390] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)_ _[0.025500] TCP bind hash table entries: 32768 (order: 7, 524288 bytes, linear)_ _[0.025708] TCP: Hash tables configured (established 32768 bind 32768)_ _[0.025758] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear)_ _[0.025806] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear)_ _[0.025898] NET: Registered PF\_UNIX/PF\_LOCAL protocol family_ _[0.026055] RPC: Registered named UNIX socket transport module._ _[0.026063] RPC: Registered udp transport module._ _[0.026067] RPC: Registered tcp transport module._ _[0.026071] RPC: Registered tcp NFSv4.1 backchannel transport module._ _[0.026078] PCI: CLS 0 bytes, default 64_ _[0.026168] Unpacking initramfs…_ _[0.038552] Initialise system trusted keyrings_ _[0.038659] workingset: timestamp\_bits=42 max\_order=20 bucket\_order=0_ _[0.040946] squashfs: version 4.0 (2009/01/31) Phillip Lougher_ _[0.041200] NFS: Registering the id\_resolver key type_ _[0.041216] Key type id\_resolver registered_ _[0.041221] Key type id\_legacy registered_ _[0.041252] nfs4filelayout\_init: NFSv4 File Layout Driver Registering…_ _[0.041258] nfs4flexfilelayout\_init: NFSv4 Flexfile Layout Driver Registering…_ _[0.060723] Key type asymmetric registered_ _[0.060738] Asymmetric key parser ‘x509’ registered_ _[0.060798] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)_ _[0.060805] io scheduler mq-deadline registered_ _[0.060810] io scheduler kyber registered_ _[0.062471] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver_ _[0.062845] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver_ _[0.063389] EINJ: ACPI disabled._ _[0.064431] mv\_xor\_v2 f0400000.xor: Marvell Version 2 XOR driver_ _[0.064674] mv\_xor\_v2 f0420000.xor: Marvell Version 2 XOR driver_ _[0.064926] mv\_xor\_v2 f0440000.xor: Marvell Version 2 XOR driver_ _[0.065165] mv\_xor\_v2 f0460000.xor: Marvell Version 2 XOR driver_ _[0.065561] mv\_xor\_v2 f26a0000.xor: Marvell Version 2 XOR driver_ _[0.065818] mv\_xor\_v2 f26c0000.xor: Marvell Version 2 XOR driver_ _[0.067260] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled_ _[0.087967] f0512000.serial: ttyS0 at MMIO 0xf0512000 (irq = 16, base\_baud = 12500000) is a 16550A_ _[0.448106] Freeing initrd memory: 24592K_ _[0.448135] printk: console [ttyS0] enabled_ _[1.048107] loop: module loaded_ _[1.053035] spi-nor spi2.0: s25fl064k (8192 Kbytes)_ _[1.061541] spi-nor spi2.1: unrecognized JEDEC id bytes: 00 00 00 01 81 ff_ _[1.068464] spi-nor: probe of spi2.1 failed with error -2_ _[1.074488] tun: Universal TUN/TAP device driver, 1.6_ _[1.081868] hwmon hwmon0: temp1\_input not attached to any thermal zone_ _[1.090424] hwmon hwmon1: temp1\_input not attached to any thermal zone_ _[1.097620] mvpp2 f2000000.ethernet: using 8 per-cpu buffers_ _[1.109130] sky2: driver version 1.30_ _[1.112845] rvu\_af: Marvell OcteonTX2 RVU Admin Function Driver_ _[1.118830] rvu\_nicpf: Marvell RVU NIC Physical Function Driver_ _[1.124788] rvu\_nicvf: Marvell RVU NIC Virtual Function Driver_ _[1.130750] VFIO - User Level meta-driver version: 0.3_ _[1.136330] armada38x-rtc f2284000.rtc: registered as rtc0_ _[1.141860] armada38x-rtc f2284000.rtc: setting system clock to 2019-02-07T16:27:08 UTC (1549556828)_ _[1.151082] i2c\_dev: i2c /dev entries driver_ _[1.155710] at24 0-0053: supply vcc not found, using dummy regulator_ _[1.162631] at24 0-0053: 256 byte 24c02 EEPROM, writable, 16 bytes/write_ _[1.169767] pca953x 1-0070: supply vcc not found, using dummy regulator_ _[1.176461] pca953x 1-0070: using no AI_ _[1.180439] pca953x 1-0070: failed writing register_ _[1.185408] pca953x: probe of 1-0070 failed with error -5_ _[1.204291] sbsa-gwdt f0610000.watchdog: Initialized with 10s timeout @ 25000000 Hz, action=0._ _[1.213273] sdhci: Secure Digital Host Controller Interface driver_ _[1.219490] sdhci: Copyright(c) Pierre Ossman_ _[1.223998] Synopsys Designware Multimedia Card Interface Driver_ _[1.230173] sdhci-pltfm: SDHCI platform and OF driver helper_ _[1.236224] ledtrig-cpu: registered to indicate activity on CPUs_ _[1.242421] SMCCC: SOC\_ID: ARCH\_SOC\_ID not implemented, skipping …_ _[1.249056] usbcore: registered new interface driver usbhid_ _[1.254657] usbhid: USB HID core driver_ _[1.259040] NET: Registered PF\_PACKET protocol family_ _[1.264198] 9pnet: Installing 9P2000 support_ _[1.267005] mmc0: SDHCI controller on f06e0000.sdhci [f06e0000.sdhci] using ADMA 64-bit_ _[1.268517] Key type dns\_resolver registered_ _[1.280894] Loading compiled-in X.509 certificates_ _[1.288736] hw perfevents: enabled with armv8\_cortex\_a72 PMU driver, 7 counters available_ _[1.297940] dw-pcie f2620000.pcie: IRQ index 1 not found_ _[1.303308] armada8k\_pcie\_probe 297_ _[1.306830] armada8k\_pcie\_probe 338 base: ctrl_ _[1.311308] devm\_pci\_remap\_cfg\_resource 4329 name:f2620000.pcie ctrl start:0xf2620000 size:65536_ _[1.320185] devm\_pci\_remap\_cfg\_resource 4342 dest=0x99e0000_ _[1.325798] armada8k\_pcie\_setup\_phys 123_ _[1.329800] armada8k\_pcie\_setup\_phys 138 phy\_count=1_ _[1.334800] armada8k\_pcie\_enable\_phys 89_ _[1.338748] armada8k\_add\_pcie\_port 257_ _[1.342546] armada8k\_add\_pcie\_port 262_ _[1.346334] devm\_pci\_remap\_cfg\_resource 4329 name:f2620000.pcie config start:0xe0f00000 size:524288_ _[1.355442] devm\_pci\_remap\_cfg\_resource 4342 dest=0x9a00000_ _[1.361051] armada8k-pcie f2620000.pcie: host bridge /cp0/pcie@f2620000 ranges:_ _[1.368410] armada8k-pcie f2620000.pcie: Parsing ranges property…_ _[1.374716] armada8k-pcie f2620000.pcie: MEM 0x00e0000000…0x00e0efffff → 0x00e0000000_ _[1.383207] armada8k\_pcie\_host\_init 185_ _[1.387080] dw\_pcie\_link\_up 537_ _[1.390245] dw\_pcie\_link\_up 541_ _[1.393404] armada8k\_pcie\_link\_up 157 read PCIE\_GLOBAL\_STATUS\_REG_ _[1.399532] dw\_pcie\_read\_dbi 148 base+0x99e0000 reg=0x8008_ _[1.405048] dw\_pcie\_read 101 address=0x99e8008, size=4_ ---
**Author:** ![jnettlet](https://dub1.discourse-cdn.com/flex005/user_avatar/community.solid-run.com/jnettlet/32/62_2.png) [@jnettlet](https://community.solid-run.com/u/jnettlet)\ **Post date:** [June 19, 2023, 3:19am UTC](https://community.solid-run.com/t/how-to-use-pcie1/667/4 "2023-06-19T03:19:39Z")
Looks like u-boot is setting up Comphy4 as USB-3 which is usually the default, which is strange because our patches for u-boot definitely have the device-tree set to PEX1. Are you using the ClearFog Pro device-tree as the basis for your custom dts? I ask because your Machine Model is reporting, SolidRun CN9130 SOM and Clearfog Base. ```auto Comphy-4: USB3_HOST1 Comphy-5: PEX2 ``` ---
**Author:** ![mpmurphy00](https://dub1.discourse-cdn.com/flex005/user_avatar/community.solid-run.com/mpmurphy00/32/133_2.png) [@mpmurphy00](https://community.solid-run.com/u/mpmurphy00)\ **Post date:** [June 19, 2023, 2:11pm UTC](https://community.solid-run.com/t/how-to-use-pcie1/667/5 "2023-06-19T14:11:52Z")
Jon, The DTS for Linux started from ClearFog Base, which is in the out of date model string. In the ClearFog Pro DTS I didn’t see what I might need to bring in on PCIe setup. I’m selecting SERDES4 there. The Marvell documentation has a diagram of 2 host controllers and 1 device controller and options to connect them to SERDES lanes. Perhaps I need to be saying something about the host controller for PCIe1, beyond status=“okay”. The history of our u-boot code I need to research. Could you point me to the patches or what the pertinent patch does? I would prefer to setup the bus in Linux, but the current task is simply to prove the bus in our IO board layout. Mike ---
**Author:** ![mpmurphy00](https://dub1.discourse-cdn.com/flex005/user_avatar/community.solid-run.com/mpmurphy00/32/133_2.png) [@mpmurphy00](https://community.solid-run.com/u/mpmurphy00)\ **Post date:** [June 21, 2023, 12:01am UTC](https://community.solid-run.com/t/how-to-use-pcie1/667/6 "2023-06-21T00:01:53Z")
Resolved. The intended SERDES must be selected in u-boot’s device tree. * * * Some details for future explorers: 1. yes, a Linux readl( ) in ARM64 can hang if it’s addressing some IO memory which is disabled or absent. This is entirely a processor matter, demonstrable from u-boot’s command line, with “md” and “mm” Checking PCIe host #2 link status, succeeds: > Marvell\>\> md.l 0xf2648008 1 > f2648008: 0000007c |… > Marvell\>\> Turn off clock to PCIe host #2: > Marvell\>\> mm.l 0xf2440220 > f2440220: 3feffbff ? 3f3fc3f7 > f2440224: 00000000 ? Marvell\>\> Now try reading link status again … processor stuck: > Marvell\>\> md.l 0xf2648008 1 > f2648008: Note on that example: clock to PCIe host is one reason the registers might be unreadable, but not the only one … 1. If, in U-boot, the PCIe bus is associated with any SERDES, it seems that the host controller logic also gets enabled. It’s flipping the on switch to make that portion of the address space accessible. U-boot goes through some ARM feature called “Secure Monitor Calls”. Unclear to me is if that could be implemented on the Linux side; perhaps the ability to do this gets disabled in the boot process as a security concept? ---
**Author:** ![jnettlet](https://dub1.discourse-cdn.com/flex005/user_avatar/community.solid-run.com/jnettlet/32/62_2.png) [@jnettlet](https://community.solid-run.com/u/jnettlet)\ **Post date:** [June 21, 2023, 5:45am UTC](https://community.solid-run.com/t/how-to-use-pcie1/667/7 "2023-06-21T05:45:12Z")
The SSMC calls could be implemented in Linux, but generally everything is designed for the bootloader to setup the configuration of the SERDES. Glad you got everything sorted out.