That's a little oversimplificated, some chords fit in several scales, and chords inversions can be considered as other chords with notes added, which broadens the possibilities. If you consider modes, then it's totally crazy.
Luckily most music we hear is written in major or minor, so what you say is right in 90% of the cases