Phil, sorry but there are specific rules in place about not wearing any political affiliation things within a polling station or near it. It is to allow a truly neutral site for all to vote. It is the same to setting up tables, signs, giving away water. Handing out political information.
It dates back to when candidates would use actual bribery and bully boy gangs to entice and force people to vote a certain way.
It's also why you can't film in polling stations.
So this person is one hundred percent in the wrong, period.
This guy was clearly spoiling for a fight, and was a complete asshole. And said inflammatory things to escalate it to this. Then tried to escalate a physical confrontation. No doubt it's a two way street. But the poll worker did not start this. That is clear.