We rely on the tamper protection of the embedded smart card. Smart cards are especially designed to protect the stored keys. They embed technologies like special metal coating and covers on the silicon so that attacks using an electron microscope are not possible (with regular chips you can actually see the stored bits). They also implement protection circuits to protect against current analysis attacks and more. So we can assume that the smart card, even if it gets intercepted, can not be tampered with, especially it is not possible to extract the key(s) from it.
A different thing is the micro-controller on the Librem Key. This is not especially protected but you also have to ask what the possible attack vector could be. Since the private key(s) never leave the smart card the micro-controller thus never sees them or could expose them. So even it being unprotected does not impose any risk for exposing the private key(s).