In my opinion being able to playing something by ear is more valuable than theory, but theory definitely helps. Figuring out songs you can easily do with just ear, but when you want to write your own material. Theory is very valuable because it gives you ideas you may not have conceived before. An example is you probably wouldn't play a Neopolitan chord (bII7) if you don't know how it works. People without the theory knowledge may stick with this chord progression, ii V7 I. Theory (tri-tone sub) introduces you to the idea you can substitute the V7 chord with bII7, and actually produce a new sound you probably wouldn't otherwise have come up with.

As for you last question. It's not so much a note that is out of key that sticks out like a sore thumb. But a tone in the melody you want to play has a bad melodic quality. Usually referred to as bad phrasing. Also playing notes that may seem out of key is not a bad thing. Your right it sticks out like a sore thumb, but that can be good. Why? Because it immediately attracts the ear, which combats monotony. Notes out of key provoke interest. I'm not saying do this, because that is not the point. The point is having good melodic phrasing regardless of the notes. There are a ton of scales out there that have notes out of key, yet they sound good.