It took me a while before I started focusing on them because they just seemed so difficult. It's probably more your hand strength and flexibility that's causing you problems, particularly if you're fairly new at it.

I'd say you'll definitely want to play them at some point, so don't give up entirely. You'll need to reach over to the E and A strings anyway for power chords and open chords, so you won't ever be able to escape the stretch completely.