This idea leads back to ancient Athens. If the Athenians wanted to banish somebody for not being classical enough, they would vote on the question by putting little black or white fragments of pottery in a box. White meant yes: black no. These tiles were called ostrakons. Hence ostracism. Ostracism has nothing to do with ostriches but is distantly related to oysters (both words relate to bone).
The first verbing of ostracism was in this apt couplet by Andrew Marvell:
Therefore the democratic stars did rise,
And all that worth from hence did ostracize.
The method and term survives to this day in blackballing. In the gentlemen's clubs of London an application for membership may be refused on the basis of a single black ball in the ballot box.
In ancient Syracuse votes for banishment did not use shards of pottery. They used olive leaves and so ostracism was called petalismos, which is far more beautiful.
A brown ballot box containing balls, darling.