His son Saif al-Arab was killed, as well as three of his grandsons.
Journalists say the building was extensively damaged and one unexploded bomb remains at the site.
Saif al-Arab was the youngest of Col Gaddafi's six sons, with a lower profile than his brother Saif al-Islam.
He had been studying in Germany and returned to Libya recently.
Government spokesman Moussa Ibrahim said the villa was attacked "with full power."
"The attack resulted in the martyrdom of brother Saif al-Arab Muammar Gaddafi, 29 years old, and three of the leader's grandchildren," he said.
"The leader with his wife was there in the house with other friends and relatives, the leader himself is in good health, he wasn't harmed." Col Gaddafi's wife was also unharmed, he said.
"This was a direct operation to assassinate the leader of this country," the spokesman added.
'Law of the jungle' Libyan rebels began a campaign in mid-February to end more than four decades of rule by Col Gaddafi.
Since last month they have been aided by an international coalition acting on a UN mandate to protect Libyan civilians.
Mr Ibrahim said the attack late on Saturday was against international law.
"How is this helping in the protection of civilians? Mr Saif al-Arab was a civilian, a student... He was playing and talking to his father and mother and his nieces and nephews and other visitors when he was attacked and killed for no crime he committed."
He accused the international coalition conducting air strikes over Libya of not wanting peace.
"We have again and again declared that we are ready for negotiation, ready for road maps for peace, ready for political transitional periods, ready for elections, ready for a referendum.
"Nato does not care to test our promises. The West does not care to test our statements. They only care to rob us of our freedom, our wealth, which is oil, and our right to decide out future as Libyans."
Gunfire rang out in celebration in the eastern rebel stronghold of Benghazi following the news that Saif al-Arab Gaddafi had been killed