It's not necessary to hear the E anyways. If you want the E note to ring through, maybe try playing the A chord at the 5th fret (Bar chord version).

Or drop your thumb further down so your wrist is angled more. Should be coming straight down onto the frets. Harder to grip your guitar this way though.