Which fingers are you using? For instance, I use my middle & ring fingers just as one would to form the Emaj, with the index finger up and off of the 'g' string to get the E minor. Using my index and middle finger to form an E minor is a rare occurence. Another possible factor; is your palm resting on the back of the neck? For most open chords, I only use my thumb- anywhere from the tip to the joint depending on the wrist angle- which acts as a very low-friction pivot point for switching between chords. I also tend to use only my thumb on the back of the neck on barre chords and chord fragments, unless the manner in which they're being used warrants that good ol' baseball bat grip.

I don't know if any of that helps, but no two hands are exactly alike so you may have to experiment with your grip and which fingers to arrive at a comfortable solution.