Next year I'm graduating, and I sort of don't want to go for the same reason: it'll make me sad I'm leaving. However, grad will probably be the last time I ever see most of these people, so I'm gonna go for that reason alone.

If you already had grad and you didn't go, then there's really nothing you can do about it. It's your own choice to make, and if people are mad at you for making a choice that isn't theirs, then people are dumb.