The talks offer was agreed at an AU summit in Malabo, Equatorial Guinea.
The AU also told members not to execute an arrest warrant for Col Gaddafi from the International Criminal Court (ICC).
The warrant "seriously complicates the efforts aimed at finding a negotiated political settlement to the crisis in Libya, which will also address, in a mutually reinforcing way, issues related to impunity and reconciliation," delegates said in a statement.
The chairman of the AU Commission, Jean Ping, said they were not against the ICC, but felt that the court was "discriminatory" and targeted only officials from the African continent.
A total of 31 states in Africa are signatories to the ICC, representing nearly a third of the nations where the mandate applies.
Libyan government spokesman Moussa Ibrahim welcomed the decision.
"The ICC is a European Guantanamo Bay. It's only against the African leaders. It never deals with the crimes committed by the United States of America... and by the European powers," he told reporters in Tripoli.
Col Gaddafi, along with his son Saif al-Islam and intelligence chief Abdullah al-Sanussi, has been accused of crimes against humanity. The ICC said it had grounds to believe they ordered attacks on civilians.
Ceasefire call The offer of talks without Col Gaddafi's involvement followed intense debates between African leaders at the summit over two days.
The AU also called for an immediate ceasefire and the lifting of the UN no-fly zone which paved the way for Nato's military intervention.
It said both parties should formally request the deployment of a UN peacekeeping mission in Libya to monitor the implementation of a cessation of hostilities.
Rebel representatives at the summit said they would need a number of guarantees from the AU before they could agree to a ceasefire.
TNC representative Mansour Saif al-Nasr told reporters that the rebels were prepared to end hostilities if Col Gaddafi stepped down.
"If we see that Gaddafi withdraws, we are ready to stop and negotiate with our brothers who are around Gaddafi," he said.