What program are you using to record and what operating system and sound card. If your using win 98 make sure to have your DMA turned on for your hard drives, this wont help latency but this will improve overall performance greatly. Also if you are using Pro Tools there is a "low latency" monitoring option.

Also are your drum parts midi? You could be having midi drift, which is a pain.