In truth, there were several reasons that one could decline a duel without loss of honor. For example if the duel challenge was issued with obvious quarrelous intent.
Eg:
"You're a liar"
"No I'm not. What are you talking about?"
"Ah, so you deny being a liar?"
"Yes, wtf are you getting at?"
"Then by your denial, you accuse me of being a liar! This insult shall not stand. I demand satisfaction."
"Lol, fuck off"
Another case would be if one duelist was not of sufficient station to match the honor of their opponent. A freshly-minted bourgeoisie vs a nobleman, for example.
Lastly, duels might be turned down if it were obvious to all that that a significant skill mismatch were at play. For example, a military officer might not be allowed to duel a civilian with sabres. Guns, however, were generally considered more egalitarian.