Gain, volume and dynamics
Gain and volume are independent concepts as they're commonly applied to guitar amps.
How much gain you need depends upon what kind of sound you're going for, how loud you play, how you attack the strings, how you set your tone controls, what kind of pickups you have in your guitar, and on and on and on...
Don't confuse gain with volume. Although they're related at at technical level, guitarists perceive gain as being independent of volume. Gain is usually associated with distortion. And distortion compresses your sound which reduces your dynamics.
High volume opens up a different world of possibilities when the amp starts interacting with the guitar. Increased sustain, controlled feedback, notes that shift into higher harmonics as you hold them, etc.
You can get some but not all of the effects of high volume by using high gain. However, high gain brings increased noise which interferes with your notes and causes them to eventually die out. You can get "infinite" sustain using high volume, but not so with high gain and very low volume. You can make tradeoffs in between the two extremes; the trick (unless you're playing metal, and even then the goal is dependent upon the subgenre) is to use just enough extra gain to get that guitar/amp interaction going.
You also have to think about volume and how it affects the way your amp and guitar interact with each other. The only good way to learn about this is to experiment; it varies with the guitar, the amount of gain, the EQ settings, the room and the player's position and orientation in the room. There's a certain volume where the guitar starts to behave differently; learning to find that spot and not only control but also take advantage of the guitar's interaction with the amp is a big part of learning to use your amp.
You need enough volume to make the guitar come alive. How much volume you need depends upon how acoustically lively the guitar is (semi-hollows are usually better in this respect) and upon the overall (pickups, tone controls, speakers, amps) EQ. You can't make up for lack of volume by using more gain; that flattens out the note attack and hides your playing dynamics.
Finally, an amp-related technique that can be important for some musical styles is to vary your attack on the strings to control how much your amp distorts. Pick lighter to clean up the sound and dig in to add some "bite". You can supplement your touch by adjusting the guitar's volume control. Different kinds of amps respond differently, and that difference has a lot to do with where your distortion is coming from. Master-volume amps tend to give you less control in this regard; they get a lot -- if not all -- of their distortion from the preamp section which has a far less complex behavior than a non-MV amp in which every part of the amp contributes to distortion in a manner that depends upon volume. The downside is that non-MV amps have to have their volume cranked up to behave this way. In many cases a cranked amp will be too loud and you'll have to find another means of getting that kind of response from the amp; usually by adding an attenuator or switching to a smaller amp, but sometimes a pedal can help.
A Hot Plate is a good tool for controlling the stage volume of a cranked amp. If you're going into a situation where you don't know how much amp you'll need for the room, it's a lot easier to bring a larger amp and a Hot Plate than it is to carry an assortment of amps. But I don't like to use a Hot Plate to get my basic overdrive tone; the controls are too coarse.