Noticing is right, that's what I'll say first of all.

There's a slight flaw in most books that teach jazz improvisation. They usually set you to some standard on how you should play, so you end up getting something that sounds good, but you never really think outside the box.

Usually in those type of books they say to make big use of chord tones, especially the 3rd and 7th. If you are playing D Dorian, you may be playing over a D minor 7th chord. The chord tones are -
D F A C, where you emphasize F (3rd) and the C (7th).

Now as for notes you shouldn't play, let your ears decide that. You may decide that something said in your book is total crap, and you can use any note you want over any chord. It's all up to you. In fact, try this -

Play a progression and record yourself humming over it (this can be done over a small tape recorder, provided you don't use the tape recorder to play the progression), then listen back. Find out what you liked about the melodies you created, and the bginning of the phrases and ends. Then transfer this stuff to guitar. This can be tedious work, but the results are worthwhile. Every one of us has the capacity to come up with melodies, and it'll really shine out in your guitar playing. Oh, one more thing, if you found out you really don't like something you did in the humming exercise, remember not to do it in improvising. Simple.

Hope that helps ya!