To take the second part first: >You need a backup password or similar which kinda defeats the point of having the key.
So the main threat models HSMs address are 1) using keys with online systems without remote attackers being able to compromise those keys (and also potentially increasing the difficulty of performing remote hot attacks too), and 2) making it much harder to attackers in unsecure physical locations to get at original keys as well purely from theft.
Having a backup password that is kept in a safe or the like, or an airgapped system(s) in a secure room/building that all HSMs are loaded from, in no way defeats the point. The point of the token is to be able to then go out into the world and make use of those keys in places which aren't secure and on systems that are online and multiple use and thus vastly easier to compromise. The Yubikey (or any of a range of smartcards or heavier duty HSMs) ideally should mean that obtaining the original private keys at least requires physically finding and breaching the generation location (assuming the keys aren't generated on device and simply manually switched upon device breakage), and that even blackbox usage requires both physically obtaining the token and the PIN or other second factor (more sophisticated HSMs may require multiple person involvement as well). This radically shifts the economic costs for attackers.
>Yubikeys make me nervous, what happens when it breaks? or your house burns down.
If using it for on-key generation, presumably with systems that you have at least intermittent physical access to, then breakage merely means doing a manual shuffle of going around and updating certs with a new key. If that's a fairly infrequent and low probability event, there may be no further need to think about it than that. You had to setup the systems in the first place after all. Alternatively if you have keys stored offline in some manner, it's trivial to setup a new token, or to buy multiple tokens and have them all be the same (with a few kept around in a safe maybe) so that having one get destroyed involves no downtime at all, just scheduling to bring it back up to n+whatever at a future time.
