Anything that puts noise into your ears can be harmful, so of course headphones can cause damage!
Sure earphones may cause more damage by actually going inside your ears, but let's face it anything you put up to your ears and increase the volume can cause serious or potentially serious damage to your ears. If you went to a loud concert you can hurt your ears and there is nothing even touching your ears, so lets face it folks by putting something that close to your ears probably will hurt your hearing down the road. Even if you're listening to quiet music, there is still something up against the side of your head making noise that isn't natural. If someone went around talking to you an inch from your ears, one hour or so each day, by the time you reach your 30's or 40's that could mean some serious problems.
Sure this topic is still in debate because headphones haven't been around for more than 30 years, so where can they get the scientific evidence. All you have to do is wait a few years down the road and see how many people have hearing problems when they're older. Sure they had headphones in the 90's listening to tapes and Walkmans, but now with Ipod's and mp3 players being so easily accessible and convenient in size people are going around three, four hours a day listening to music by using headphones. It has become the norm in society to walk around with your headphones on your head.
Even though ear infections caused by headphones might be on the rare side, they still do happen. With kids going to school on the bus listening to music, then their friend wants to listen to one of the songs playing, so they pass along the headphones. Little do they know that the other child has an ear infection that he is going to pass on to his friend at that moment. The thing is your ears are very sensitive, as a child you probably have little regard for what you're putting in or around your ear.
As I said in the earlier paragraph, your ears are very sensitive. People everywhere do not appreciate loud noises, however just because the volume is turned down, doesn't mean that your ear drum can't still feel the constant pressure it's under when it's covered by headphones. Headphones may not cause extreme damage to your ears, but damage to your ears is damage to your ears, and when you're in your elderly years, you will probably appreciate things more if you have better hearing to help you out.