It's tricky business and most often applies to women, but with more and more of us deciding to take the initiative, even men must develop that diplomatic skill of the "Subtle Turn-Down".
No one wants to hurt another's feelings, and in the case of women, we have been conditioned not to hurt. We are healers and the carriers of life, our job being to nurture, to protect, to build up and not to tear down. We learn early to accommodate men, making it all the more difficult for us to say no. Being a woman, i will address it from that point of view, but any suggestion can be applied to the guy finding himself in that delicate position of having to say "No thank you".
We've all been there and likely it began very early. You were 13 and that weird guy with the glasses who sat behind you in English asked you to go the the "Spring Fling". You were so hoping to get that same invitation from the guy in your homeroom with the wild green eyes, but you were so polite, so shy and so afraid you would hurt his feelings that you went anyway... and you were miserable.
If you are even old enough to say yes, you must be mature enough to say no and to do it in a way that's polite but firm, honest and direct. Sometimes we just can't use the real reason for having to decline, but there are a few little white lies, or even ways to make the truth go down easier and face it - it's far more polite to be truthful than to give a guy hope where there isn't any.
Try to avoid the trite standards... "I just don't want a relationship now". He didn't ask for a relationship but one little dinner out. If "I'm seeing someone" is true, then by all means use it, but if not, a lie will come back to bite you... promise.
The truth in one form or another, spoken in such a way that he can save face (and so can you) is the best way to go. The usual reason for turning a man down is just not "feeling it". Maybe he's a friend, maybe just not your type. Both of those things can be conveyed without stepping on toes or wounding that fragile male ego.
If you give him a positive first (but not a hope in hell, you don't want him to keep asking), then you can blunt that barb you are about to stick him with. A few of my past turn downs include...
(To a guy I knew well and was close to)
"John, I like you very much, but we've been friends a long time and you really are like a brother. Not only do I not want to ruin that, but I just don't feel that way about you. It isn't that you won't be great for someone else, but it just isn't there for me. Now get your coat and let's find a beer."
(To that guy from the office that I had to see every day)
"Wow, I'm really flattered, but I'm sorry, I have to say no. I have a policy of not dating men I work with".
And if he knows you have dated other men from your place of employment...
"Wow, I'm really flattered but I have to say no. I am certain of the kind of men that I get along with best. You're a great guy, but I'm sure we are less compatible than you might think. But thanks, really. Being asked by a guy like you is... well, great, but it just won't work."
Then excuse yourself and make sure no one is in the ladies room when you let loose laughing for 15 minutes.
(How about your best friend's brother?)
"Paul, I'm sorry but I know that would be a mistake. Phoebe and I are such close friends. It would be too much of a risk to our friendship, not that I wouldn't jump at the chance otherwise, I just don't want to risk anything coming between us".
(The married man, the arrogant creep or the man-whore from the gym)
"NO thank you".
And no more than that. You can even skip the "thank you". If you apologize or qualify your answer, these guys will only twist whatever you say and make you find further excuses. In fact, the more simple ANY turn down, the better and for the same reason.
Saying that you aren't dating right now is a trap and usually a lie. It will likely only embarrass you later, so learning to tell a man he isn't what you are looking for is actually kinder in the long run. Making excuses or putting them off so that they keep thinking that they may have a chance later when they don't is just as impolite, but far more selfish, being used only to make it easier on YOU, not him.
In the end, you may hurt his feelings a bit, but don't be so into yourself as to think you have shattered his world. It is just an invitation for a date, after all, and not a marriage proposal. So you may need to simply get over the idea that you can assert yourself without anyone suffering. Some people are going to feel wounded no matter what you do and you can't sacrifice yourself to anyone's feelings. Being disappointed and occasionally disappointing others is just part of life. The mature person does their best and moves on.
Should you find yourself in the position of having to turn down the man who has been less than polite himself, who has pursued you with gusto, behaved like a stalker, turning you into a nervous and hyper-vigilant reject from the witness protection program, peeking round every corner and wearing your dark glasses indoors... Well, if that's the case, forget everything I've said and learn to be rude. This type deserves no more of your time than you've already lost to him and some simply can't hear anything but the most blatant and harsh "NEVER".