Nvme on NEO 5 M.2 MVE RPI5_8GB not working --> brcm-pcie 1000110000.pcie: link down

[cg200]

Hi,
my setup:

• RPI5_8GB with 128GB SD
• Kingston SNV2S NV2 Gen4 PCIe NVMe M.2 2280 Solid-State Drive
• Argon NEO 5 M.2 NVME PCIE Case for Raspberry Pi 5
• installed with the two Argon scripts

chris@RPI:~ $ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
mmcblk0 179:0 0 119,3G 0 disk
├─mmcblk0p1 179:1 0 512M 0 part /boot/firmware
└─mmcblk0p2 179:2 0 118,7G 0 part /

chris@RPI:~ $ sudo dmesg | grep -i pcie
[ 0.000000] Kernel command line: reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe smsc95xx.macaddr=2C:CF:67:74:BC:27 vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000 console=ttyAMA10,115200 console=tty1 root=PARTUUID=8eab817b-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=DE
[ 0.387543] brcm-pcie 1000110000.pcie: host bridge /axi/pcie@110000 ranges:
[ 0.387549] brcm-pcie 1000110000.pcie: No bus range found for /axi/pcie@110000, using [bus 00-ff]
[ 0.387560] brcm-pcie 1000110000.pcie: MEM 0x1b00000000…0x1bfffffffb → 0x0000000000
[ 0.387567] brcm-pcie 1000110000.pcie: MEM 0x1800000000…0x1affffffff → 0x0400000000
[ 0.387573] brcm-pcie 1000110000.pcie: IB MEM 0x0000000000…0x0fffffffff → 0x1000000000
[ 0.388933] brcm-pcie 1000110000.pcie: Forcing gen 2
[ 0.389131] brcm-pcie 1000110000.pcie: PCI host bridge to bus 0000:00
[ 0.815759] brcm-pcie 1000110000.pcie: link down
[ 0.820589] pcieport 0000:00:00.0: PME: Signaling with IRQ 38
[ 0.820659] pcieport 0000:00:00.0: AER: enabled with IRQ 38
[ 0.820941] brcm-pcie 1000120000.pcie: host bridge /axi/pcie@120000 ranges:
[ 0.820946] brcm-pcie 1000120000.pcie: No bus range found for /axi/pcie@120000, using [bus 00-ff]
[ 0.820955] brcm-pcie 1000120000.pcie: MEM 0x1f00000000…0x1ffffffffb → 0x0000000000
[ 0.820960] brcm-pcie 1000120000.pcie: MEM 0x1c00000000…0x1effffffff → 0x0400000000
[ 0.820969] brcm-pcie 1000120000.pcie: IB MEM 0x1f00000000…0x1f003fffff → 0x0000000000
[ 0.820973] brcm-pcie 1000120000.pcie: IB MEM 0x0000000000…0x0fffffffff → 0x1000000000
[ 0.822147] brcm-pcie 1000120000.pcie: Forcing gen 2
[ 0.822183] brcm-pcie 1000120000.pcie: PCI host bridge to bus 0000:00
[ 0.927763] brcm-pcie 1000120000.pcie: link up, 5.0 GT/s PCIe x4 (!SSC)
[ 0.939877] pcieport 0000:00:00.0: enabling device (0000 → 0002)
[ 0.939907] pcieport 0000:00:00.0: PME: Signaling with IRQ 39
[ 0.939962] pcieport 0000:00:00.0: AER: enabled with IRQ 39

[cg200]

I installed the hardware twice with the second flex cabel to exclude any hardware error.

any suggestions?

[HarryH]

Try another NVMe model, maybe from another vendor. The Kingston SNV2S V2 is one of the worst choices:

reason:

• no specification about power consumption available
• variyng controllers in use
• reports that these NVMe get very hot indicate high power consumption

limits of the RPi5 to be in specification:

• 5W max. via flexible pcb (PCIe cable)
• flexible pcb may not longer than 5cm
• PCIe Gen2 (not Gen4)

Not all NVMe controllers are supported, because of least one of these limits is mostly hurt, the NVMe has firmware issues or one of the components has been bad manufactured. In my opinion so it’s only a lottery, unfortunately.

[wojak]

The same!

I bought two cases:

• Argon Neo 5 M.2 NVMe PCIe
• Argon One V3 M.2 NVMe
  And two Raspberry Pi 5 with 8GB RAM.

No matter what I do or how I try to troubleshoot (and I’ve been at it for two weeks now), I always get a “PCIe Link Down” error.
I’ve tested a variety of drives: 980 Pro, 990 Pro, PineDrive, and everything else I had on hand. I swapped the Raspberry Pi between cases, tried new and different PCIe cables, but nothing changes. I just can’t get it to work.

[HarryH]

Have you already made sure that the flexible PCB is not inserted upside down? And also checked that the NVMe is properly inserted into the socket as described in this and the following article?:

Argon NEO 5 M.2 - yet another SSD not detected

The fragility of the nvme cable makes it highly likely to be a common point of failure, but I feel like I have tried every possible which way (both cables in the pack, followed the instructions to the letter) and still no success. That fact that it’s just ‘dead’ - I can’t see anything about nvme running ‘dmesg’ still makes the cable a prime candidate for problems. I’m attaching a close-up picture of my latest (non-working) setup. If the cable is upside down or anything obvious I’ll wear it if …

PS: If the problem persists, please be more specific. You have not mentioned about:

• Current installed bootloader version, for example: 2024-09-10
• Model number of the power supply you are using
• Is the NVMe not working in both cases, or only in one of them
• Operating system/distribution
• Capacity, firmware, model number of the NVMe
• Content of bootloader config/EEPROM
• Photo of the boot screen
• …

[Wils]

run sudo rpi-eeprom-config check if you got the same setting

[Wils]

run sudo nano /boot/firmware/config.txt and see if the argonneo5 script is installed properly

[wojak]

BOOTLOADER: up to date
   CURRENT: Tue 30 Jul 14:25:46 UTC 2024 (1722349546)
    LATEST: Tue 30 Jul 14:25:46 UTC 2024 (1722349546)
   RELEASE: latest (/lib/firmware/raspberrypi/bootloader-2712/latest)
            Use raspi-config to change the release.

SSDs:
variety of samsungs:
970 evo 1TB
980 pro 256
980 pro 512
990 pro 1TB

currently:

Pinedrive - SSD NVMe M.2 2280 - 256GB - HatDrive! Pineboards

NVMe is not working in both cases, on both RPI’s

operating system:

Linux raspberrypi 6.6.47+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.47-1+rpt1 (2024-09-02) aarch64 GNU/Linux

Can’t do photo now but it is “nvme failed” or something and then info that pcie link down.

[all]
WAKE_ON_GPIO=0
PSU_MAX_CURRENT=5000
BOOT_UART=1
POWER_OFF_ON_HALT=1
USB_MSD_PWR_OFF_TIME=1000
USB_MSD_STARTUP_DELAY=3000
BOOT_ORDER=0xf41
PCIE_PROBE=1

after update config.txt is currently empty, yesterday it was set according to all guides and by argon scripts.

I also updated Argon One V3 firmware.

Maybe it is related to RPI 8GB? Also I noticed that ribbon likes to move a little, so yesterday I even tried to insert some paper on back to make it more rigid, no changes but at least I am sure they are secured correctly.

Only yesterday I tried with multiple clear rpi systems (rpi imager), using USB and SD card. No luck, no change.

Power supply: official 27W 5A

[    0.000000] Kernel command line: reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe bcm2708_fb.fbwidth=640 bcm2708_fb.fbheight=480 bcm2708_fb.fbdepth=16 bcm2708_fb.fbswap=1 smsc95xx.macaddr=2C:CF:67:00:A0:A1 vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000  console=ttyAMA10,115200 console=tty1 root=PARTUUID=de0956b8-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=PL nvme_core.default_ps_max_latency_us=2500 pcie_aspm=off
[    0.396448] brcm-pcie 1000110000.pcie: host bridge /axi/pcie@110000 ranges:
[    0.396454] brcm-pcie 1000110000.pcie:   No bus range found for /axi/pcie@110000, using [bus 00-ff]
[    0.396466] brcm-pcie 1000110000.pcie:      MEM 0x1b00000000..0x1bfffffffb -> 0x0000000000
[    0.396472] brcm-pcie 1000110000.pcie:      MEM 0x1800000000..0x1affffffff -> 0x0400000000
[    0.396478] brcm-pcie 1000110000.pcie:   IB MEM 0x0000000000..0x0fffffffff -> 0x1000000000
[    0.397822] brcm-pcie 1000110000.pcie: Forcing gen 2
[    0.398037] brcm-pcie 1000110000.pcie: PCI host bridge to bus 0000:00
[    0.823638] brcm-pcie 1000110000.pcie: link down

as you see I even tried some hacks from internets with cmdline.txt:

Kernel command line: reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe bcm2708_fb.fbwidth=640 bcm2708_fb.fbheight=480 bcm2708_fb.fbdepth=16 bcm2708_fb.fbswap=1 smsc95xx.macaddr=2C:CF:67:00:A0:A1 vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000  console=ttyAMA10,115200 console=tty1 root=PARTUUID=de0956b8-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=PL nvme_core.default_ps_max_latency_us=2500 pcie_aspm=off

[wojak]

I bought third Raspberry Pi 3s (8GB), and everything works fine…
It looks like the two I bought earlier together have some kind of connector damage.
Same ribbons are working correctly, even original ribbons from argon cases.

[HarryH]

Thank you for the feedback. Nice to hear/read, that is working now.

After you delivered the details of your odyssey, there was not many room for another root cause than an hardware issue. In the past some users had the ribbon cable (flexible pcb) connected up side down. But 2 times a defective RPi5 sounds hard …

Perhaps you should have played the lottery in the last few days?

[wojak]

This must be some production batch issue, cause 2 not working are from same vendor, third one is from different vendor. It may also explain why some 8gb versions are problematic with PCIE.

But yeah - I am playing lottery this week