Then inform the caller of the loss of your vote, then inform the party of said transaction.
Are you aware of the voter suppression tactics that are employed by parties these days?
The Blue party will identify people they think are supporters of the Red party. They will then call the Red supporters at all kinds of inconvenient times, 4am in the morning, in the middle of the superbowl, at dinner time, and repeatedly. They will adopt a rude tone of voice. They will say, "Hello, I am calling to see if you will be supporting the Red party this election, and whether you have any questions about the Red party platform?"
Their goal is to piss off all the Red party voters so that they will become frustrated with the party and subsequently not show up at the polls to vote.
There were many cases of these sorts of dirty tricks in the most recent election. Some of it amounted to actual criminal fraud (people pretending to be elections canada and confusing people as to where their poll was). A lot of it is much more ambiguous--there is nothing illegal about a Conservative asking you if you will support the Liberals or if you have questions about the Liberal platform, but the person being called will automatically assume that is a call from the Liberal party. One that is designed to piss them off at the Liberal party.
Anyway... before you change your vote based on what you perceive is annoying/irritating behavior by one party, make sure it's not actually a dirty trick by the other party. It's a surprisingly common tactic.