I only use my fingers when I vibrato as well, Jeff, and people have told me that my vibrato sounds fine. I think it's just a matter of building up your finger strength. You could get used to doing it with your entire wrist, or you could just keep doing it with your fingers and practicing it consciously - whatever you prefer.