Really shouldn't be a problem. I don't think I'd remove it completely though. If you can get to it, just poke a few holes in it, then it will still buzz, but not the annoying BEEP
...or you can clip a wire heading to the speaker. If you poked holes in the speaker, it would still make noise because the signal is still going to the voice coil. The buzzing is the battery noise.