In most cases, you're right. Not, however, if you're a gamer.
The reason I replaced my card (GeForce 2 GTS Pro, 64 MB) was that it was occassionally hiccuping on Neverwinter nights. That game eats graphics power. Unc's 32 MB Matrox G400 never stood a chance, nor did the 32 MB ATI I just replaced in his gaming rig over here. The game was damn near unplayable.
If all you're doing is browsing and office apps, a 32 MB card might even be a little overkill.