The reason the frets are marked so that you can see at a quick glance where you are on the neck. The markings for the frets have no relevance to scales or playing scales. They are just like little signposts to tell you what fret you are at.
It may seem like it, but it doesn't take a genius to memorize the scale patterns all over the neck. it just takes a lot of practice and time.
It's also worth getting to know where all the notes are on the guitar and also knowing which notes are in which major scales.
It just takes time man.