Notes out of the key are all up to the ear of the beholder. If your playing by yourself, and it sounds good to you let her rip. The only place you have to pay more attention is when your playing along with other bandmates. Then you all have to be on the same page when it comes to key, and note selections that are out of the key. By letting them know about your chromatic travels in certain areas of the piece of music your working on they can make adjustments if necessary.