PCS members will strike for 24 hours next Thursday – when many thousands of visitors are due to arrive in the UK.
Home Secretary Theresa May said the action was “shameful” as it threatens disruption to people travelling to London for the Games.
Immigration minister Damian Green said contingency plans were in place.
East Midlands Trains staff have also voted to strike during the Olympics.
PCS union members will take other forms of action from July 27 to August 20, including working-to-rule and an overtime ban.
The PCS said 57.2% of those who voted backed strike action – the turnout was 20%.
The action will involve staff across the Home Office, including the UK Border Agency, the Identity and Passport Service and Criminal Records Bureau.
PCS general secretary Mark Serwotka told the BBC: “I think the government is whipping up hysteria about the Olympics, there’ll be no disruption to the Olympics, this is a 24-hour strike before the Olympics actually takes place.”
He said he was prepared to meet the culture secretary and home secretary any time in the next week to avert a strike but if they kept their “heads in the sand” the strike would continue.
Mr Hunt said the union’s behaviour was “totally irresponsible… To threaten us in this way is totally inappropriate.
“To suggest that it won’t cause disruption is so extraordinary that it completely beggars belief.”
London Mayor Boris Johnson has said he does not think the union will succeed in disrupting the Olympics and the majority of PCS members want to put on a great Games.