In a perfect world, we could all fret the low E string with index on the first fret, middle on the second, ring on the third and pinky on the fourth, without any buzzes or touching of adjacent strings. In reality, this can be hard to do.

With guitar, there's perfect form, and there's what you need to do to make the music happen. It technically doesn't matter if your pinky is touching an adjacent string, if you're not playing that string at that moment. And you'd never really need to have frets 1-3 fretted with your other fingers during a song, in addition to having the pinky on fret 4. So you could remove your other fingers and only have the pinky down, or maybe just remove the index and middle.

I would suggest the following:

*Try this exercise on higher frets, like frets 7 or 8, where the frets are closer together, making the stretch less.

*Try to put the pinky down cleanly without touching another string, while lifting up some of your other fingers if you need to.

Eventually this will get easier. The pinky is the most difficult of fingers, because it shares a tendon with your ring finger. Teaching it to be strong and have a life of its own can be hard. Don't give up on it.