I notice that when I play something in E, A or B Minor (or their relative majors, G, C, and D), I can play all over the fretboard never worrying about playing anything out of scale (on accident). However, with other scales I have to consciously think about where the patterns are. The more you play songs in a key (and most of the songs I played before this year have been in G, C, and D), the better I get at not having to think about it. My fingers "know" what's in the scale and what isn't, but I can always just think for a second to play in any key. People who don't know music theory well (like the other guitarist in my band, who doesn't know about moveable scale patterns, etc.) can only solo in certain keys. This kind of proves what you were saying about people specializing in certain keys.