If ya wanna use drums as a way to improve your rhythm and timing, ya gotta make sure your drumming is even to. Practice with a metronome. You can get some metronomes without a click, just a light, so the noise it would otherwise make doesn't get confused with the sound of your stick on a drum.
I dont play drums, but I feel that rhythm and timing is very important, so I've always practiced with a metronome. Its just the way I was taught.