If you have too much string wrapped around the peg, it will tend to 'creep', giving the same stored-tension kind of hassle as a 'pinching' nut. The highest string should have 4-6 wraps around the peg. Less than 4 risks breakage due to tension being concentrated on the sharp bend through the peg hole or slot. More than 6 gets into 'creepy' territory. As the strings get bigger, you can reduce the number of turns. The low E is usually good with 2 turns.

Be sure to 'lock' the strings at the tuner post. I don't feel that I can describe this well in text. This is one of those things you need to get someone to show you.

If the strings are heavier guage than what the nut was fitted for, its gonna pinch. Try that press-and-release thing before you do anything else. Don't waste a new set of strings if the nut needs to be opened up a bit. You can take the strings with you when you bring the guitar in for a setup.