It's tough man, I know. Been there myself. If your situation allows it, you need to separate yourself from the temptation. Easier said than done I know but why put yourself in a situation that brings it to your attention all the time. If you can't leave then you gotta talk to the dude. Tell him the same thing you wrote here. If he's halfway cool, he'll understand.

Hope it goes well for you. Like Superhuman said, 'stay strong' and remember why you gave the stuff up to begin with.