I do the same thing. They're not really a miiiiiiiile but they aren't as tight to the fretboard as they technically should be. But you know what... I don't really like to play as fast as I can all the time anyway so it's no big deal to me. Just keep practicing it. Try to do some chromatic scales and keep your fingers close.