I don't know how long you have been playing, but like William MG said it's just consistency, and practice.  It will all come together. Don't worry so much about not looking at the fret board, it's ok to look at it. One thing that might help, is while you're playing a chord for example, is to look at the fret board to where you're going next this sometimes helps with smoother transitions, especially in the beginning stages of playing.