Arr, that is because once you have hammered out audio latency when you initially set the program up, unless you suddenly start recording like 8 inputs simultaneously, you will not have to ever look at it again.

Unlike recording audio VSTs require varying amounts of system resources hence the more you use the greater the drain on the system which as you rightly point out causes the VST latency thing. The problems are both similar and different !!