Hobo - I can't be 100% certain ... but I believe the amount of rep a person gives or takes is related to the amount of rep they have on the forum already. So for instance, if someone like myself or Murph + or - reps you, you'll lose or receive a greater amount of rep than, say, if someone with the default amount (I think it's 10 points) adds or takes rep. So I'm guessing someone with a fair amount of rep struck at you there.
If you see a gray bubble - instead of red or green - that's probably a case of someone with negative or very little rep using the button.
This explanation crossed my mind. Its a system that makes sense. Just like in the real world, if someone popular says someone else is cool, it has a lot more pull than if someone who is invisible to everyone else says someone is cool :P