If yes, you might need some set up adjustment. If no, practice, practice practice! :) If it is purely a strength issue, it will come over time. You could possible get a hand exerciser for supplemental development.
...so ever since then, I always hang on to the buckle.