You don't have to go for 4/4...you can pick any time signature. I understand what you're talking about, but the average listener may think that going out of time is a mistake by the musician. Rather than having people think you're a hack, I'd say it's probably safer keeping in the pocket. Slowing down and speeding up within a song requires great precision between all band members or it can come out sloppy. Unless you're going for the 'aleotoric' thing...