can modes be built off of both the major and the minor scales or is it just off the major?

Well, the natural minor scale modes are the same as the major scale modes since the natural minor scale is already a mode of the major scale.
On the other hand, the harmonic minor, or melodic minor scales are variations of the natural minor and have their very own interval system, therefore their inversions ( or modes ) will give a whole new set of possibilities, and they both have their own modal system. You probably heard of the phrygian dominant mode before ( especially if you're an Yngwie fan ). Well the phrygian dominant mode is simply the 5th mode of the harmonic minor scale. So if the harmonic minor scale is this :

A B C D E F G# played over an Am7 chord

Then the phrygian dominant would be this

E F G# A B C D played over an E7 ( E dominant 7 ) chord.

Same notes, different root and underlying chord.

And that's all there is to it really. Well not really but that's the basics...