Does not start up reliably

Okay, I’m new to the RP5 and have recently purchased an Argon ONE V3 NVMe. I have the Pi 5 installed and booting from the nVME drive and all is well and good when it is running - the problem occurs at startup. If I press the power button sometimes the LED will flash red once and nothing happens, sometimes nothing at all happens. I can press and hold the power button for >5 seconds, wait a few seconds after releasing it, then press it again and it will start. If I shut it down after it has been running I will have to repeat the procedure to get it to start again.
I am using the Argon power supply (I had a problem with other power supplies not reaching the connections in the USB-C port, so it would not power up at all).
All updates have been applied.
I have tried a different RP5, but the problem still occurs, so I have concluded that the problem is in the case. Not sure what to do at this point and would appreciate any help.

Depending on the operating system you are using, you should install the script (described in the manual) or an add-on (LibreELEC) to install a background service that monitors the power button signal, handles fan control, and supports graceful shutdown.

Only the graceful shutdown will properly power off the RPi5 in the case. The MCU (RP2040) inside of the case will be in a defined state afterwards, and you will not need to force a hard shutdown by holding the power button for more than 5 seconds.

Thanks. Currently running Raspberry PI OS (64-bit). I have previously installed the script, but can try it again.
Another, perhaps related, question: should the power LED remain on (red) when shut down?

I can only speak for the Argon ONE V2 with a RPi4. The LED should be off after a graceful shutdown. It should be in same state that you have as you unplug/plug the power supply again, before you press the power button the first time.

You can check the service state:
sudo systemctl status argononed.service
sudo journalctl -u argononed