It's definitely a concern. I know my hearing's not what it should be, especially in things like crowded rooms. (there's nothing more embarrasing than having a girl say something to you at a party and having to go "wha??" all the time)
If I use headphones with lots of bass, my eardrums itch like crazy for days so that's telling me something. It's a constant battle though, deciding whether to protect your hearing or turn it up loud so you can get into the music.