I think a lot of "down" votes can have a poster banned/unable to post. IOW, three red squares and you're out.

It doesn't do anything on the "plus" side per se but, when one sees posts from those with a lot of yellow squares like Barry Green (shouldn't Barry have green squares?), Mitch Gross, PureDrifting, et.al, it implies a very credible poster with a solid take on whatever topic is being brought up.