I've been down this road; its easier to record everything at once, since like taht way you said it'd be very hard to get the drums to sync up perfectly with the music.

However, it could be much easier to record the drums first THEN record the rest of the music around it.