Algeria's UN ambassador said they were received on humanitarian grounds.
Libya's rebels said sheltering Gaddafi family members was an act of aggression, and called for their extradition.
Col Gaddafi's whereabouts are unknown.
Meanwhile rebels said that another of his sons, Khamis, may have been killed in a clash near Bani Walid.
The reports came as Libyan rebels were trying to overcome the pockets of resistance by Gaddafi loyalists, and preparing to advance on Col Gaddafi's birthplace, Sirte.
Obvious refuge The arrival of Col Gaddafi's wife and three children in Algeria had been reported to the UN and to Libyan rebel authorities, the Algerian foreign ministry said in a statement carried by Algeria's APS news agency.
It said they had crossed the border between Libya and Algeria at 08:45 local time (07:45 GMT) on Monday.
The BBC's Jon Leyne in the eastern Libyan city of Benghazi said first word of such a move had already come from Libyan rebel headquarters two days ago, and that at the time, Algerian authorities denied that a convoy of six heavily armoured vehicles had crossed the border.
Algeria is an obvious refuge for the Gaddafi family as the two countries have a long border and the Algerian government has still not recognised the rebel National Transitional Council (NTC), our correspondent says.
"They have been greeted on humanitarian grounds," Algeria's ambassador to the UN, Mourad Benmehidi, told the BBC World Service.
"The holy rule of hospitality in our culture is something that one has to keep in mind, especially in the desert region where it's a duty to provide assistance to anyone."
But NTC spokesman Mahmoud Shamman said: "This would be an act of aggression against the Libyan people and against the wishes of the Libyan people.
"We will use all legal means to seek the return of these criminals and to bring them to justice in Libya."
Muhammad and Hannibal are two of the sons with the least involvement in politics.
Rebels had previously suggested that other sons may be in or close to the Gaddafi stronghold of Sirte.
On Monday, rebel Col Al-Mahdi Al-Haragi was quoted as saying Khamis, who has led a feared army unit blamed for a massacre last week, had died after being badly wounded in a clash.
Rebel military spokesman Ahmed Bani said bodies in a convoy destroyed during the clash were burnt beyond recognition, but captured soldiers said they were Khamis' bodyguards.
He also said Col Gaddafi's brother-in-law and the head of his intelligence services, Abdullah al-Sanussi, was "almost certainly killed" in the same confrontation.
Leaked UN document There were reports earlier in the conflict that Khamis had been killed in an air strike, though those reports were later questioned.
Claims earlier this month that rebels had detained Col Gaddafi's most prominent son, Saif al-Islam, turned out not to be true.
Rebel fighters have been pushing towards Sirte, and earlier took the small town of Nofilia on their way to the city.
The rebels say they think Col Gaddafi himself may still be in the Tripoli area, says our correspondent.
Meanwhile, a leaked document that appears to outline UN proposals for post-conflict Libya calls for up to 200 military observers and 190 UN police to help stabilise the country.
The deployment would follow a UN mission with a core staff of 61 civilians for an initial three month period, according to the report on the website Inner City Press.
Any such plan would be implemented only if requested by the Libyan transitional authorities and approved by the Security Council, it said.