Zimbabwe's foreign minister said it did not recognise the rebel National Transitional Council.
President Robert Mugabe is a close ally of Col Gaddafi, who bankrolled the African Union. Only a few African countries have recognised the NTC.
Last week South Africa blocked moves at the UN to give the NTC access to Libyan government funds.
Zimbabwe's Foreign Minister Simbarashe Mumbengegwi, from Mr Mugabe's Zanu-PF party, said Mr Elmagrahi and all his staff now had 72 hours to leave the country.
"Once you renounce the authority that gave you the letter of credence and then proceed to pledge allegiance to another authority... it means that act deprives you of your diplomatic standing," he said.
When he switched sides last week, Mr Elmagrahi said: "I am not Gaddafi's ambassador. I represent the Libyan people."
Correspondents say Mr Mugabe and his allies are wary of the revolutions which have toppled three long-serving North African leaders this year.
More than 40 activists were arrested in February after watching videos about the Egypt uprising.
Mr Mugabe has condemned Nato's intervention in Libya and says the conflict is really about oil.