Mic spazzes for some, fine for others

I have a pretty weird problem with my microphone.
90% of the people can hear me fine, but the other 10% can’t.
The 10% that can’t tells me that it cuts out all sound in game while I hold down my microphone button.
I find it weird as its only 5-10% of the server that cannot hear me, and its starting to become a problem. Has anyone else had this/know how to fix it?
Oh, and it happens in ALL source games, not just Garry’s Mod.

It will be a problem with the internet connection on the other end; nothing you can do about it.

[editline]09:33PM[/editline]

why does opera’s spellchecker not think “internet” is a word