This is a serious infringement of your rights. First things first, call an attorney. Now. They may be able to force you out since you're not on a lease but they cannot make boundless accusations and slanderous comments like that. Do your neighbors know what you're being accused of? If so, that's slander. I might mention that to the landlord. Ask him for his proof. Where's his evidence? Have the cops actually been watching you? Do you know any cops? If you do, ask them. If you're clean, you wouldn't have anything to worry about. One question though; is everyone who lives there clean? (No offense but that's just one of those things you have to think about) You don't want to call attention to the situation if there's a possible problem.

Hang tough. Don't be afraid to fight. If you've done nothing wrong, stand tall.