Al-Shabab has long restricted the work of international aid groups but on Monday banned 16 groups outright.
Years of conflict mean Somalia is worst hit by the East African drought.
The lack of rain is said to be the worst in 60 years.
The list of groups banned outright included the United Nations children's agency, Unicef, and other UN bodies, the British charity Concern and groups from Denmark, France, Germany, Italy, Norway, and Sweden.
Unicef spokesman Jaya Murthy told the BBC a group of men, suspected to belong to al-Shabab, occupied their offices in Baidoa and ordered staff to leave.
"They just said they [Unicef staff] should go home immediately and our office is now their office," Mr Murthy told the BBC's Focus on Africa programme.
The al-Shabab statement accused the groups of exaggerating the scale of the problems in Somalia for political reasons and to raise money.
'Risk of death' It also alleges that the agencies are working with church groups trying to convert vulnerable Muslim children and opposing al-Shabab's attempts to impose Sharia law.
Aid groups and al-ShababBanned:
- Un agencies including UNHCR, Unicef and WHO
- Norwegian Refugee Council (NRC)
- Danish Refugee Council (DRC)
- Italy's Cooperazione Internazionale (COOPI)
- Swedish African Welfare Alliance (SAWA)
- German Agency For Technical Cooperation (GTZ)
- France's Action Contre la Faim (ACF)
- COSV (Italy)
A senior al-Shabab official told the BBC that those groups which had been closed down had not carrying out many activities, and that the measures would not increase the suffering of ordinary people.
He also pointed out that three groups - the International Committee of the Red Cross, medical aid charity MSF and Italy's Copi - would still be allowed to operate.
Mr Murthy said Unicef was involved only in humanitarian work and al-Shabab's decision would threaten the lives of children.
"About 160,000 severely malnourished children are at imminent risk of death if assistance does not continue," he said.
The UN says the areas worst effected by famine are in the southern and central areas, which are under the control of the al-Qaeda linked group.
The UN-backed government only runs a few areas, including the capital, Mogadishu, which al-Shabab forces withdrew from in August.
Earlier this month, the UN said that famine conditions no longer existed in three of the areas previously worst affected - Bay, Bakool, and Lower Shabelle.
However, a quarter of a million people still face imminent starvation in the country, the UN says.
UN humanitarian affairs co-ordinator Mark Bowden told the BBC: "Somalia still remains the world's most critical situation."
Three other areas, including the squalid camps in the capital, Mogadishu, remain in a state of famine.
However, a senior aid worker familiar with the situation in Somalia who did not wish to be named told the BBC that the situation was still getting worse.
He said the UN could not admit this because it had to show the aid money was being well spent and having an impact.
Other aid workers have also warned that the situation could be worsened through conflict.
Kenya has sent troops into southern parts of Somalia, accusing al-Shabab of abducting Westerners from border areas - charges denied by the militants.
Tens of thousands of Somalis have fled rural areas - many over the borders to Ethiopia and Kenya - in search of food.
Somalia has not had a functioning central government for more than 20 years and has been wracked by fighting between various militias.