Same for me as gogachachi.
For example, I typically play Amin with 1,2,3. (Same shape as your E, just rooted on the A string instead of the E string). Similarly, Bmin has the same shape as your F, also just rooted on A instead of E string, so the barre and 2,3,4. When playing Lyin' Eyes, the chorus has a spot that goes from Bmin to Amin. So in that case I just slide down the neck two frets to make the transition keeping 2,3,4 as the active fingers. It's all about the most efficient and best sounding transition.
I don't use 2,3,4 for the E or Amin in most circumstances because it is easier to transition quickly with 1,2,3. Others may find differently.