I think your diagnosis is probably correct. I suggest starting with the switch. There's a good chance you're tapping it just enough to make it cut out.

If you have an Ohm meter, you can check the pot by measuring from one of the end terminals to the centre terminal while slowly turning the shaft. If the reading suddenly jumps around a lot, there's some damage to the track.

A shot of contact cleaner may solve that (if dirt on the track is the only problem), but it's usually a more reliable repair if you replace the pot.