I never turn the volume up very high. Will the most powerful Anthem amp still sound better?

It can, because requirements are ultimately dictated by the signal peaks, not the average levels. When playing recordings with peak-to-average ratio of, for example, 20dB (i.e. not pop recordings) and playback level is set for average output of three watts, the peaks still need 300W to play cleanly. Each 10dB increase requires 10 times the power output to prevent clipping.