How to use PCIE1

OCTEON CN9130
1

I’m trying to setup PCIe bus #1 for the CN9130 SOM.

Background:
I have ClearFog Base available to use, but not 64 bit ClearFog Pro, so I haven’t seen this bus work yet with a known software setup. The SOM is being connected to our own IO board which has two PCIe slots (as M.2 connectors). I have PCIe2, with SERDES 5 (“comphy 5”) working. PCIe1 is on SERDES 4, replicating the ClearFog Pro setup. In debug I currently have no card connected in the PCIE1 slot, as the present hurdle does not seem to about communication with a device. Software is Linux kernel 5.16.x. I’ve compared the files cn9130-cf-base.dts and cn9130-cf-pro.dts, and didn’t see anything that seems relevant that I don’t already have.

Problem to solve:
With my current device tree, the bootup hangs when I enable PCIe1. I’ve tracked that to this in the kernel code:
_/drivers/pci/controller/dwc/pcie_designware.c
_/drivers/pci/controller/dwc/pcie_armada8k.c

Function armada8k_pcie_link_up( ) executes, which wants to read register PCIE_GLOBAL_STATUS_REG. That call goes to generic dw_pcie_read( ), which does a readl( ). Execution on the core in use seems to block there, something I didn’t realize was possible on that operation. Looking at readl( ), it’s an ARM64 specific macro which is wrapping a few assembly instructions. The console is not available and responsive yet, but the processor is still executing code - if one waits a while, one can occasionally see complaint messages from the kernel such as “[ 1376.578392] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:”

The address used for PCIE_GLOBAL_STATUS_REG comes from a remap ( via devm_pci_remap_cfg_resource ), + the expected offset. The remap input came from cn9130.dtsi, f2620000 for PCIe1, or f264000 for PCIe2, and the output address is looks sane, when compared to the same code setting up PCIe2.

I tried substituting a read from PCIE_GLOBAL_STATUS_REG with previous register PCIE_GLOBAL_CONTROL_REG. That read attempt also hangs the code.

From this post, different processor:

and something else I saw online, I get the impression that PCIe host controllers within the CN9130 might be disabled. I haven’t yet found in Marvell’s documentation how that might happen.

Questions:

  • does the CN9130 have this feature, and would an IO memory read block if the host controller logic is disabled?

  • If so … how is the enable selected?

2

Please provide full dmesg output of the failed boot. My guess is that device-tree has mapped that slot as PCIe, but perhaps TF-A or u-boot is still setting it up as SATA causing the hang. SATA was the default setting for the other variants of the CF-PRO.

3

Jon,

Below is the boot spam from my current test build from power up, which includes U-boot’s commentary on “comphy”, and some of my debug messages.

If ClearFog Pro and ClearFog Base have a different boot loader builds, then I should explore that. We are currently building the boot loader also, started I suspect from a ClearFog Base sample.

Follow ups on this: looking online just now, I see a page of build instructions
https://solidrun.atlassian.net/wiki/spaces/developer/pages/200769556/CN913x+u-boot+atf+and+mv-ddr-marvell+-+Self+Build
showing git clone on u-boot source. The page is marked as archived, and perhaps is not specific to ClearFog Pro.
Are there different source trees from Base vs Pro, or is theresome setup option to be made? I would diff the two and track down what needs to happen.

Mike

BootROM - 2.03
Starting AP IOROM 1.02
Booting from eMMC 0
Found valid image at boot postion 0x004
lNOTICE: Starting binary extension
NOTICE: SVC: DEV ID: CN913x, FREQ Mode: 0x0
NOTICE: SVC: AVS work point changed from 0x2e2 to 0x2da
w/o ecc; strap value=0
4GB capacity; strap value=0
mv_ddr: 14.0.0-gcfede8f8 (Mar 22 2023 - 11:37:54)
SSCG_EN
Synopsys DDR43 PHY Firmware version: A-2017.11
SNPS DDR: 1D training passed
SNPS DDR: 2D training passed
SNPS DDR: training completed
dma memcmp pass
mv_ddr: completed successfully
NOTICE: Cold boot
NOTICE: Booting Trusted Firmware
NOTICE: BL1: v2.3(release):R_CLEARFOG_IPsec-6_0_0_E2_2-2023-03-16-10-gcfede8f8 (Marvell-devel-18.12.0)
NOTICE: BL1: Built : 11:37:58, Mar 22 2023
NOTICE: BL1: Booting BL2
NOTICE: BL2: v2.3(release):R_CLEARFOG_IPsec-6_0_0_E2_2-2023-03-16-10-gcfede8f8 (Marvell-devel-18.12.0)
NOTICE: BL2: Built : 11:37:58, Mar 22 2023
NOTICE: SCP_BL2 contains 7 concatenated images
NOTICE: Skipping MG CP1 related image
NOTICE: Load image to CP0 MG
NOTICE: Loading MG image from address 0x403114c Size 0xe0f0 to MG at 0xf2100000
NOTICE: Skipping MSS CP3 related image
NOTICE: Skipping MSS CP2 related image
NOTICE: Skipping MSS CP1 related image
NOTICE: Load image to CP0 MSS AP0
NOTICE: Loading MSS image from addr. 0x4044930 Size 0x1cfc to MSS at 0xf2280000
NOTICE: Done
NOTICE: Load image to AP0 MSS
NOTICE: Loading MSS image from addr. 0x404662c Size 0x5400 to MSS at 0xf0580000
NOTICE: Done
NOTICE: BL1: Booting BL31
lNOTICE: BL31: v2.3(release):R_CLEARFOG_IPsec-6_0_0_E2_2-2023-03-16-10-gcfede8f8 (Marvell-devel-18.12.0)
NOTICE: BL31: Built : 11:37:58, Mar 22 2023

U-Boot 2019.10-10.0.0 (Mar 23 2023 - 11:26:44 -0400)

Model: SolidRun CN9130 based SOM ClearFog Base
ERROR: mrvl_sip_smc_handler: unhandled SMC (0x82000014)
ERROR: mrvl_sip_smc_handler: unhandled SMC (0x82000014)
ERROR: mrvl_sip_smc_handler: unhandled SMC (0x82000014)
ERROR: mrvl_sip_smc_handler: unhandled SMC (0x82000014)
ERROR: mrvl_sip_smc_handler: unhandled SMC (0x82000014)
ERROR: mrvl_sip_smc_handler: unhandled SMC (0x82000014)
ERROR: mrvl_sip_smc_handler: unhandled SMC (0x82000014)
ERROR: mrvl_sip_smc_handler: unhandled SMC (0x82000014)
ERROR: mrvl_sip_smc_handler: unhandled SMC (0x82000014)
ERROR: mrvl_sip_smc_handler: unhandled SMC (0x82000014)
ERROR: mrvl_sip_smc_handler: unhandled SMC (0x82000014)
ERROR: mrvl_sip_smc_handler: unhandled SMC (0x82000014)
SoC: cn9130-A1; AP807-B0; CP115-A0
ERROR: mrvl_sip_smc_handler: unhandled SMC (0x82000014)
Clock: CPU 2000 [MHz]
ERROR: mrvl_sip_smc_handler: unhandled SMC (0x82000014)

  •    DDR     1200 [MHz]*

ERROR: mrvl_sip_smc_handler: unhandled SMC (0x82000014)

  •    FABRIC  1200 [MHz]*

  •    MSS     200  [MHz]*

LLC Enabled (Exclusive Mode)
DRAM: 4 GiB
Running in RAM - U-Boot at: 0x7fed4000

  •             Env at:    0x7ff8a117*

Comphy chip #0:
Comphy-0: SATA1
Comphy-1: USB3_HOST0
Comphy-2: SFI0 10.3125 Gbps
Comphy-3: SGMII1 1.25 Gbps
Comphy-4: USB3_HOST1
Comphy-5: PEX2
UTMI PHY 0 initialized to USB Host0
UTMI PHY 1 initialized to USB Host1
Error reading output register
PCIE-0: Link down
NAND: 0 MiB
MMC: sdhci@6e0000: 0, sdhci@780000: 1
Loading Environment from EXT4… OK
get_tlvinfo_from_eeprom: failed to read from eeprom!
get_fdtfile_from_tlv_eeprom: could not identify carrier, defaulting to Clearfog Pro!
Net:
Warning: mvpp2-0 (eth0) using random MAC address - 1e:5e:7e:74:4b:37
eth0: mvpp2-0 [PRIME]
Warning: mvpp2-1 (eth1) using random MAC address - ba:d0:f3:88:77:75
, eth1: mvpp2-1
Warning: mvpp2-2 (eth2) using random MAC address - 82:83:61:f4:fe:02
, eth2: mvpp2-2
Hit any key to stop autoboot: 0
32906833 bytes read in 2028 ms (15.5 MiB/s)
## Loading kernel from FIT Image at 09000000 …

  • Using ‘Edge-Crypto-Module’ configuration*
  • Verifying Hash Integrity … sha256,rsa4096:firmware.sign*
  • Verifying RSA4096 Signature …OK*
    + OK
  • Trying ‘kernel-1’ kernel subimage*
  • Description:  esm_32001092-001+ipsec+6.0.0.E3.3*

  • Created:      2023-06-18  17:30:35 UTC*

  • Type:         Kernel Image*

  • Compression:  gzip compressed*

  • Data Start:   0x09000140*

  • Data Size:    7695799 Bytes = 7.3 MiB*

  • Architecture: AArch64*

  • OS:           Linux*

  • Load Address: 0x00200000*

  • Entry Point:  0x00200000*

  • Hash algo:    sha256*

  • Hash value:   fb0cc9001f76a01851535f09aeb9758e4e4a98d391312320e4c1fce98f61f3c0*

  • Sign algo:    sha256,rsa4096:firmware.sign (required)*

  • Sign value:   229a0ac74754075d161ec43b477b0541c7b397a5c17f9d9590ffe3634277b9ed9322e7d7ba68a3c848af75379295445e86d4b2a86ca26542bf79653c8acf31c0d95ab8e123b065f83a8648a44ab7084efafe64866fe366ded005c9f62ed3663466d6e33d584a21c8c0fffce81587910db8166b061d1437631d96327c0aad7047fac9eaf86fcdabb6ea881e6e65cb8fe148132ba3959b78fdcfb63f34aa097e622b6f246123aa9286ffff9d6778dfe80a6f466162b3b99d7d34ce214a234e202c6aab1d66876247290c65fa1efe784714df571eabb950fffc04e79e24a7f7547e8b410a375ad15ca83b679c8325dc98d982115e82fce938101d0a116611b21f7cbebcbb1f25f766e454056c551ade4fdea045da826c762e7c28b4230cc2cb57a92349fd7f25486289857d514c3ecbf6a263989ebb5ee77376a87212a7f6f8ce27a67ed96ac8066d0099f93c9361c35cf5ea20aaa9c6f4c91013e3396398f047a66c6fdfb0ecb6cca64696b3740e6bd1bc9a651db33d76f108f5c6fe86ebf6ed7d56b69f7045647588f2380f184a703836ebaaa8130e7528c7815931fd055aecd85a3ed7b548f53b47682a43138347ac6dd7a57204ecb738d87020bc42bb6ce9abfca8da9d22be3daeee43e24b3e8f2400f04a1e3e5ae9d7fe0f82f9a0282400fbfcd5d3b576a7af833d93e4a5ed8d0d86303cc72c848df7be55820640b9957eef*

  • Timestamp:    2023-06-18  17:30:35 UTC*
  • Verifying Hash Integrity … sha256+ sha256,rsa4096:firmware.sign*
  • Verifying RSA4096 Signature …OK*
    + OK
    ## Loading ramdisk from FIT Image at 09000000 …
  • Using ‘Edge-Crypto-Module’ configuration*
  • Verifying Hash Integrity … sha256,rsa4096:firmware.sign*
  • Verifying RSA4096 Signature …OK*
    + OK
  • Trying ‘ramdisk-1’ ramdisk subimage*
  • Description:  esm_32001092-001+ipsec+6.0.0.E3.3*

  • Created:      2023-06-18  17:30:35 UTC*

  • Type:         RAMDisk Image*

  • Compression:  uncompressed*

  • Data Start:   0x097572b4*

  • Data Size:    25183938 Bytes = 24 MiB*

  • Architecture: AArch64*

  • OS:           Linux*

  • Load Address: unavailable*

  • Entry Point:  unavailable*

  • Hash algo:    sha256*

  • Hash value:   d666f724a53f7bf80b5f6272032ffb82fc93e0ac25115fae1857f42f486ee8af*

  • Sign algo:    sha256,rsa4096:firmware.sign (required)*

  • Sign value:   02a51af4e3938c3e09ccc010ccbeb3f0a37ebeb480c55a72bf61410bb1e9905f51c63c06aa77342a21f651bf93cb96dbedd263b5352d70216e9e0a141ff173e9f3110184361ac8bc2b15d06868dda8f45e0cb26a0874724a01f282c028192ffae6fbb36a6d11620d64d54113e6427060452e16504881b7fe98228d3f57abf7b17d01d79d95488df73a690c263083f70a30280cc02804bc7f0b87e4e24dc79b23ec2d0e6dca03a89e449feff24d5c408680510bc05cd01091620fca30950b907b799e671032780365ce7fa83dd13924f7eff68aabc297382d868e4fc4abcd4d6886e533de44d22018a7d193e2bcbc3b11a516959f164cc43b169aafb2321c9ea1d87a42df960f492621d9d42492c446688c9503e004bfb572c59daf828f0543e7d7165f39580f5a5b3f4f662baa6ba465ee30e9974363e852a73cc71ac0c6e29adeb4483b41e5e91dd4d1e85ea901e7a2dfad3a2438b31ebe6299805d2b5627feb583e86c0f82694f4053ca1c96e8bce18a45d8db82886602172adaf1f07d27e2d03df8c5e3024af1726ccff1213d70edc12a55c3ecf180308f3659d28881e3b470684722b5f93c37280c75140c0f523a54c571ec2d32a82fc9987a34c6db25576080dfc41c065a193d94aae8fd4fa511cf0487ac18e71685c71c07611d0fd513155ab4ad6f556767ed240b506ef4e3eca765fa6572086fefe2c60c43e7466738*

  • Timestamp:    2023-06-18  17:30:35 UTC*
  • Verifying Hash Integrity … sha256+ sha256,rsa4096:firmware.sign*
  • Verifying RSA4096 Signature …OK*
    + OK
    ## Loading fdt from FIT Image at 09000000 …
  • Using ‘Edge-Crypto-Module’ configuration*
  • Verifying Hash Integrity … sha256,rsa4096:firmware.sign*
  • Verifying RSA4096 Signature …OK*
    + OK
  • Trying ‘fdt-1’ fdt subimage*
  • Description:  esm_32001092-001+ipsec+6.0.0.E3.3*

  • Created:      2023-06-18  17:30:35 UTC*

  • Type:         Flat Device Tree*

  • Compression:  uncompressed*

  • Data Start:   0x0af5bd10*

  • Data Size:    22427 Bytes = 21.9 KiB*

  • Architecture: AArch64*

  • Load Address: 0x02600000*

  • Hash algo:    sha256*

  • Hash value:   a09374e93c50a8d5c37898a58b6ac692c5cd3d24a0aaec67b72d580c9a7d53a0*

  • Sign algo:    sha256,rsa4096:firmware.sign (required)*

  • Sign value:   8e5913bf4aadb4eecd9b08b758a938da9456a09671726e4170c3812bd6f458efe269506204216b63e1d070972d65673e3841de2dffd1f78dfac84ac5243656907b59518f068c13bbadac08a16bb983d8c16ebd47c284c0cf9f1fb9af1c009ad7bceb72a20ebf73812454edf12fef3b57670e6e55a0978a54b53051b6a3fad3da3ff03ad207a044b7817923856c7dc62a331778f180e363de470a41884727bf53a91d04093aeb9e1c36fc5c495e66c639af11ebad00841bcc7d5ca0093b38166c6d8ff7d6ce150563f2369ba2e38528fae48c3fd6819ef83c6570481ab73856fc3a692ea209053989a61cf8b4c77c6d9d14b89f860bb9755a7e1a5fa1bb905a61f6fd37398d96fb6e818bf6d80a428330e5285a9052ad7bfaf2c233a6b0c45e0f3f23d0da399dbfbb2ee0db9ca5da1de25bfb0cbd42a0396b7ffae72903889a8647dd2a3007eccebb443724212b4789437d430cfeb8b47b238d4eff2e3f42f5be1e34dc0e9b23a82b09d015b359b297584182df5813f37db10f4bd881cc8444c35423d01337ecf7877db8f86ba6a99f59a607c8355ee6517aad08256736b6e898f5c4931ea5df12fc1642a3f13da63d18fa4400fdd190133e29127069dd8d31f4772d9f97f247006a552e74b8da1d6390f545e475ece9e755431c4d922d1983c8591f5010e02eab354510a651de1fc7d280f2ae4fa8c491e6f9b7dbacc6dbbc7a*

  • Timestamp:    2023-06-18  17:30:35 UTC*
  • Verifying Hash Integrity … sha256+ sha256,rsa4096:firmware.sign*
  • Verifying RSA4096 Signature …OK*
    + OK
  • Loading fdt from 0x0af5bd10 to 0x02600000*
  • Booting using the fdt blob at 0x2600000*
  • Uncompressing Kernel Image*
  • Loading Ramdisk to 7ddb8000, end 7f5bc6c2 … OK*
  • Loading Device Tree to 000000007ddaf000, end 000000007ddb779a … OK*

Starting kernel …

[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[ 0.000000] Linux version 5.16.13 (zeno@vm-ubuntumate22-04) (aarch64-buildroot-linux-gnu-gcc.br_real (Buildroot -g6ac77208-dirty) 10.3.0, GNU ld (GNU Binutils) 2.36.1) #2 SMP PREEMPT Fri Jun 16 17:21:45 EDT 2023
[ 0.000000] Machine model: Ultra 3eTI FIPS carrier, SolidRun CN9130 SOM and Clearfog Base
[ 0.000000] efi: UEFI not found.
[ 0.000000] NUMA: No NUMA configuration found
[ 0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x000000013fffffff]
[ 0.000000] NUMA: NODE_DATA [mem 0x13f7d7b40-0x13f7d9fff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000000000-0x00000000ffffffff]
[ 0.000000] DMA32 empty
[ 0.000000] Normal [mem 0x0000000100000000-0x000000013fffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x0000000003ffffff]
[ 0.000000] node 0: [mem 0x0000000004000000-0x00000000041fffff]
[ 0.000000] node 0: [mem 0x0000000004200000-0x00000000bfffffff]
[ 0.000000] node 0: [mem 0x0000000100000000-0x000000013fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000013fffffff]
[ 0.000000] cma: Reserved 32 MiB at 0x00000000be000000
[ 0.000000] psci: probing for conduit method from DT.
[ 0.000000] psci: PSCIv1.1 detected in firmware.
[ 0.000000] psci: Using standard PSCI v0.2 function IDs
[ 0.000000] psci: MIGRATE_INFO_TYPE not supported.
[ 0.000000] psci: SMC Calling Convention v1.2
[ 0.000000] percpu: Embedded 18 pages/cpu s36440 r8192 d29096 u73728
[ 0.000000] Detected PIPT I-cache on CPU0
[ 0.000000] CPU features: detected: Spectre-v2
[ 0.000000] CPU features: kernel page table isolation disabled by kernel configuration
[ 0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[ 0.000000] Fallback order for Node 0: 0
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1032192
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line:
[ 0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] software IO TLB: mapped [mem 0x00000000ba000000-0x00000000be000000] (64MB)
[ 0.000000] Memory: 3968780K/4194304K available (10816K kernel code, 1778K rwdata, 4120K rodata, 2048K init, 546K bss, 192756K reserved, 32768K cma-reserved)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[ 0.000000] rcu: Preemptible hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
[ 0.000000] Trampoline variant of Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[ 0.000000] GIC: Adjusting CPU interface base to 0x00000000f022f000
[ 0.000000] Root IRQ handler: gic_handle_irq
[ 0.000000] GIC: Using split EOI/Deactivate mode
[ 0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:160, num:32)
[ 0.000000] GICv2m: range[mem 0xf0280000-0xf0280fff], SPI[160:191]
[ 0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:192, num:32)
[ 0.000000] GICv2m: range[mem 0xf0290000-0xf0290fff], SPI[192:223]
[ 0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:224, num:32)
[ 0.000000] GICv2m: range[mem 0xf02a0000-0xf02a0fff], SPI[224:255]
[ 0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:256, num:32)
[ 0.000000] GICv2m: range[mem 0xf02b0000-0xf02b0fff], SPI[256:287]
[ 0.000000] random: get_random_bytes called from start_kernel+0x46c/0x658 with crng_init=0
[ 0.000000] arch_timer: cp15 timer(s) running at 25.00MHz (phys).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5c40939b5, max_idle_ns: 440795202646 ns
[ 0.000000] sched_clock: 56 bits at 25MHz, resolution 40ns, wraps every 4398046511100ns
[ 0.000100] Console: colour dummy device 80x25
[ 0.000254] printk: console [tty0] enabled
[ 0.000290] Calibrating delay loop (skipped), value calculated using timer frequency… 50.00 BogoMIPS (lpj=100000)
[ 0.000299] pid_max: default: 32768 minimum: 301
[ 0.000325] LSM: Security Framework initializing
[ 0.000357] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[ 0.000371] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[ 0.000972] rcu: Hierarchical SRCU implementation.
[ 0.001331] EFI services will not be available.
[ 0.001429] smp: Bringing up secondary CPUs …
[ 0.001843] Detected PIPT I-cache on CPU1
[ 0.001878] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]
[ 0.002315] Detected PIPT I-cache on CPU2
[ 0.002339] CPU2: Booted secondary processor 0x0000000100 [0x410fd083]
[ 0.002759] Detected PIPT I-cache on CPU3
[ 0.002776] CPU3: Booted secondary processor 0x0000000101 [0x410fd083]
[ 0.002809] smp: Brought up 1 node, 4 CPUs
[ 0.002833] SMP: Total of 4 processors activated.
[ 0.002838] CPU features: detected: 32-bit EL0 Support
[ 0.002842] CPU features: detected: CRC32 instructions
[ 0.009497] CPU: All CPU(s) started at EL2
[ 0.009519] alternatives: patching kernel code
[ 0.010110] devtmpfs: initialized
[ 0.011550] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.011566] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[ 0.011859] pinctrl core: initialized pinctrl subsystem
[ 0.012081] DMI not present or invalid.
[ 0.012254] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[ 0.012791] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations
[ 0.012894] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[ 0.013015] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[ 0.013057] audit: initializing netlink subsys (disabled)
[ 0.013126] audit: type=2000 audit(0.012:1): state=initialized audit_enabled=0 res=1
[ 0.013283] thermal_sys: Registered thermal governor ‘step_wise’
[ 0.013287] thermal_sys: Registered thermal governor ‘power_allocator’
[ 0.013546] cpuidle: using governor menu
[ 0.013584] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[ 0.013628] ASID allocator initialised with 65536 entries
[ 0.013810] Serial: AMBA PL011 UART driver
[ 0.019990] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.020004] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[ 0.020009] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 0.020014] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[ 0.020219] cryptd: max_cpu_qlen set to 1000
[ 0.020446] ACPI: Interpreter disabled.
[ 0.020855] ap0_sd_vccq: Bringing 3300000uV into 1800000-1800000uV
[ 0.021046] iommu: Default domain type: Translated
[ 0.021054] iommu: DMA domain TLB invalidation policy: strict mode
[ 0.021120] vgaarb: loaded
[ 0.021225] SCSI subsystem initialized
[ 0.021287] usbcore: registered new interface driver usbfs
[ 0.021305] usbcore: registered new interface driver hub
[ 0.021319] usbcore: registered new device driver usb
[ 0.021421] pps_core: LinuxPPS API ver. 1 registered
[ 0.021426] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti giometti@linux.it
[ 0.021436] PTP clock support registered
[ 0.021482] EDAC MC: Ver: 3.0.0
[ 0.022034] clocksource: Switched to clocksource arch_sys_counter
[ 0.022099] VFS: Disk quotas dquot_6.6.0
[ 0.022121] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.022203] pnp: PnP ACPI: disabled
[ 0.024369] NET: Registered PF_INET protocol family
[ 0.024459] IP idents hash table entries: 65536 (order: 7, 524288 bytes, linear)
[ 0.025357] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, linear)
[ 0.025390] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[ 0.025500] TCP bind hash table entries: 32768 (order: 7, 524288 bytes, linear)
[ 0.025708] TCP: Hash tables configured (established 32768 bind 32768)
[ 0.025758] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear)
[ 0.025806] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear)
[ 0.025898] NET: Registered PF_UNIX/PF_LOCAL protocol family
[ 0.026055] RPC: Registered named UNIX socket transport module.
[ 0.026063] RPC: Registered udp transport module.
[ 0.026067] RPC: Registered tcp transport module.
[ 0.026071] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 0.026078] PCI: CLS 0 bytes, default 64
[ 0.026168] Unpacking initramfs…
[ 0.038552] Initialise system trusted keyrings
[ 0.038659] workingset: timestamp_bits=42 max_order=20 bucket_order=0
[ 0.040946] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 0.041200] NFS: Registering the id_resolver key type
[ 0.041216] Key type id_resolver registered
[ 0.041221] Key type id_legacy registered
[ 0.041252] nfs4filelayout_init: NFSv4 File Layout Driver Registering…
[ 0.041258] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering…
[ 0.060723] Key type asymmetric registered
[ 0.060738] Asymmetric key parser ‘x509’ registered
[ 0.060798] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[ 0.060805] io scheduler mq-deadline registered
[ 0.060810] io scheduler kyber registered
[ 0.062471] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
[ 0.062845] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
[ 0.063389] EINJ: ACPI disabled.
[ 0.064431] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
[ 0.064674] mv_xor_v2 f0420000.xor: Marvell Version 2 XOR driver
[ 0.064926] mv_xor_v2 f0440000.xor: Marvell Version 2 XOR driver
[ 0.065165] mv_xor_v2 f0460000.xor: Marvell Version 2 XOR driver
[ 0.065561] mv_xor_v2 f26a0000.xor: Marvell Version 2 XOR driver
[ 0.065818] mv_xor_v2 f26c0000.xor: Marvell Version 2 XOR driver
[ 0.067260] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 0.087967] f0512000.serial: ttyS0 at MMIO 0xf0512000 (irq = 16, base_baud = 12500000) is a 16550A
[ 0.448106] Freeing initrd memory: 24592K
[ 0.448135] printk: console [ttyS0] enabled
[ 1.048107] loop: module loaded
[ 1.053035] spi-nor spi2.0: s25fl064k (8192 Kbytes)
[ 1.061541] spi-nor spi2.1: unrecognized JEDEC id bytes: 00 00 00 01 81 ff
[ 1.068464] spi-nor: probe of spi2.1 failed with error -2
[ 1.074488] tun: Universal TUN/TAP device driver, 1.6
[ 1.081868] hwmon hwmon0: temp1_input not attached to any thermal zone
[ 1.090424] hwmon hwmon1: temp1_input not attached to any thermal zone
[ 1.097620] mvpp2 f2000000.ethernet: using 8 per-cpu buffers
[ 1.109130] sky2: driver version 1.30
[ 1.112845] rvu_af: Marvell OcteonTX2 RVU Admin Function Driver
[ 1.118830] rvu_nicpf: Marvell RVU NIC Physical Function Driver
[ 1.124788] rvu_nicvf: Marvell RVU NIC Virtual Function Driver
[ 1.130750] VFIO - User Level meta-driver version: 0.3
[ 1.136330] armada38x-rtc f2284000.rtc: registered as rtc0
[ 1.141860] armada38x-rtc f2284000.rtc: setting system clock to 2019-02-07T16:27:08 UTC (1549556828)
[ 1.151082] i2c_dev: i2c /dev entries driver
[ 1.155710] at24 0-0053: supply vcc not found, using dummy regulator
[ 1.162631] at24 0-0053: 256 byte 24c02 EEPROM, writable, 16 bytes/write
[ 1.169767] pca953x 1-0070: supply vcc not found, using dummy regulator
[ 1.176461] pca953x 1-0070: using no AI
[ 1.180439] pca953x 1-0070: failed writing register
[ 1.185408] pca953x: probe of 1-0070 failed with error -5
[ 1.204291] sbsa-gwdt f0610000.watchdog: Initialized with 10s timeout @ 25000000 Hz, action=0.
[ 1.213273] sdhci: Secure Digital Host Controller Interface driver
[ 1.219490] sdhci: Copyright(c) Pierre Ossman
[ 1.223998] Synopsys Designware Multimedia Card Interface Driver
[ 1.230173] sdhci-pltfm: SDHCI platform and OF driver helper
[ 1.236224] ledtrig-cpu: registered to indicate activity on CPUs
[ 1.242421] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping …
[ 1.249056] usbcore: registered new interface driver usbhid
[ 1.254657] usbhid: USB HID core driver
[ 1.259040] NET: Registered PF_PACKET protocol family
[ 1.264198] 9pnet: Installing 9P2000 support
[ 1.267005] mmc0: SDHCI controller on f06e0000.sdhci [f06e0000.sdhci] using ADMA 64-bit
[ 1.268517] Key type dns_resolver registered
[ 1.280894] Loading compiled-in X.509 certificates
[ 1.288736] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[ 1.297940] dw-pcie f2620000.pcie: IRQ index 1 not found
[ 1.303308] armada8k_pcie_probe 297
[ 1.306830] armada8k_pcie_probe 338 base: ctrl
[ 1.311308] devm_pci_remap_cfg_resource 4329 name:f2620000.pcie ctrl start:0xf2620000 size:65536
[ 1.320185] devm_pci_remap_cfg_resource 4342 dest=0x99e0000
[ 1.325798] armada8k_pcie_setup_phys 123
[ 1.329800] armada8k_pcie_setup_phys 138 phy_count=1
[ 1.334800] armada8k_pcie_enable_phys 89
[ 1.338748] armada8k_add_pcie_port 257
[ 1.342546] armada8k_add_pcie_port 262
[ 1.346334] devm_pci_remap_cfg_resource 4329 name:f2620000.pcie config start:0xe0f00000 size:524288
[ 1.355442] devm_pci_remap_cfg_resource 4342 dest=0x9a00000
[ 1.361051] armada8k-pcie f2620000.pcie: host bridge /cp0/pcie@f2620000 ranges:
[ 1.368410] armada8k-pcie f2620000.pcie: Parsing ranges property…
[ 1.374716] armada8k-pcie f2620000.pcie: MEM 0x00e0000000…0x00e0efffff → 0x00e0000000
[ 1.383207] armada8k_pcie_host_init 185
[ 1.387080] dw_pcie_link_up 537
[ 1.390245] dw_pcie_link_up 541
[ 1.393404] armada8k_pcie_link_up 157 read PCIE_GLOBAL_STATUS_REG
[ 1.399532] dw_pcie_read_dbi 148 base+0x99e0000 reg=0x8008
[ 1.405048] dw_pcie_read 101 address=0x99e8008, size=4

4

Looks like u-boot is setting up Comphy4 as USB-3 which is usually the default, which is strange because our patches for u-boot definitely have the device-tree set to PEX1. Are you using the ClearFog Pro device-tree as the basis for your custom dts? I ask because your Machine Model is reporting, SolidRun CN9130 SOM and Clearfog Base.

Comphy-4: USB3_HOST1
Comphy-5: PEX2
5

Jon,

The DTS for Linux started from ClearFog Base, which is in the out of date model string. In the ClearFog Pro DTS I didn’t see what I might need to bring in on PCIe setup. I’m selecting SERDES4 there. The Marvell documentation has a diagram of 2 host controllers and 1 device controller and options to connect them to SERDES lanes. Perhaps I need to be saying something about the host controller for PCIe1, beyond status=“okay”.

The history of our u-boot code I need to research. Could you point me to the patches or what the pertinent patch does? I would prefer to setup the bus in Linux, but the current task is simply to prove the bus in our IO board layout.

Mike

6

Resolved. The intended SERDES must be selected in u-boot’s device tree.

Some details for future explorers:

  1. yes, a Linux readl( ) in ARM64 can hang if it’s addressing some IO memory which is disabled or absent. This is entirely a processor matter, demonstrable from u-boot’s command line, with “md” and “mm”
    Checking PCIe host #2 link status, succeeds:

Marvell>> md.l 0xf2648008 1
f2648008: 0000007c |…
Marvell>>

Turn off clock to PCIe host #2:

Marvell>> mm.l 0xf2440220
f2440220: 3feffbff ? 3f3fc3f7
f2440224: 00000000 ? Marvell>>

Now try reading link status again … processor stuck:

Marvell>> md.l 0xf2648008 1
f2648008:

Note on that example: clock to PCIe host is one reason the registers might be unreadable, but not the only one …

  1. If, in U-boot, the PCIe bus is associated with any SERDES, it seems that the host controller logic also gets enabled. It’s flipping the on switch to make that portion of the address space accessible. U-boot goes through some ARM feature called “Secure Monitor Calls”. Unclear to me is if that could be implemented on the Linux side; perhaps the ability to do this gets disabled in the boot process as a security concept?
7

The SSMC calls could be implemented in Linux, but generally everything is designed for the bootloader to setup the configuration of the SERDES. Glad you got everything sorted out.