Are you angling the pick? When you sweep downward it's pretty convenient to angle the top of the pick (the part you're holding) towards the ground and when going upward I find it best to angle back up at me. I'm still learning about sweeping though (getting there) and this is just my observations when I practice. It gets funner (obviously) when you get the hang of it. My fretting fingers are the main issue as they don't move fast enough. Time to hit the metronome up some more. Just take your time and watch your hands, in this case particularly your picking hand. Just watch what it does that causes it to not flow well and try things to get it to be more manueverable.