I just stopped looking down. There were many misplaced chords, some of which actually sounded interesting, but persistence paid off. I still look at the fretboard occasionally, though, if learning something new.

Chord changes are, as snojones said, just a matter of locking them into muscle memory. Like learning to ride a bike: you have to really concentrate when first learning, then do it without thinking (but you will often have to anticipate the next chord change but, when the time comes, your fingers do it without much conscious thought).