A series of meetings between Gen Kosaner and PM Recep Tayyip Erdogan failed to resolve their differences.
Turkish President Abdullah Gul moved quickly to appoint General Necdet Ozel as the new army chief.
Gen Ozel is widely expected to be swiftly elevated to chief of the general staff in place of General Kosaner. Tradition dictates that only the head of the army can take over the top job.
There has been a history of tension between the secularist military and the governing AK party, with the two sides engaged in a war of words for the past two years over allegations that parts of the military had been plotting a coup.
Investigations into those allegations, known as the "Sledgehammer" conspiracy - appear to be the root cause of today's resignations, says the BBC's correspondent in Istanbul, with the senior military wanting to go ahead with scheduled annual promotions for some of the officers implicated - and the government refusing.
The Supreme Military Council, which will decide on promotions, is scheduled for next week.
The government says the top brass asked to retire.
But Gen Kosaner portrayed his resignation as a protest at the jailing of military officers in a variety of court cases.
"It has become impossible for me to continue in this high office, because I am unable to fulfil my responsibility to protect the rights of my personnel as the chief of general staff," Gen Kosaner told the Hurriyet news group.
'Sledgehammer' Gen Kosaner and his senior commanders quit just hours after a court charged 22 suspects, including several generals and officers, with carrying out an internet campaign to undermine the government.
This case is the latest element of the protracted 'Sledgehammer' controversy - a coup plan allegedly presented at an army seminar in 2003.
Seventeen generals and admirals currently in line for promotion were among those jailed in the Sledgehammer prosecutions. Altogether nearly 200 officers were charged with conspiracy.
Twenty-eight servicemen will go on trial next month.
Gen Kosaner was appointed overall head of the Turkish armed forces just a year ago.
His appointment followed a period of intense friction between the government and the military over the Sledgehammer controversy.
At that point, the politicians vetoed the army's original choice for joint chief, Gen Hasan Igsiz, because he was implicated in the alleged plot.
Nato no comment It reportedly involved plans to bomb mosques and provoke tensions with Greece, in order to spark political chaos and justify a military takeover.
The defendants have argued that the plot was a just theoretical scenario to help them plan for potential political unrest.
The dramatic mass resignation has particular resonance in Turkey, which endured a series of military coups from 1960 to 1980.
In 1997, an army-led campaign forced the resignation of the country's first Islamist-led government.
The joint resignation of military chiefs is thought to be unprecedented in Turkey, which is a Nato member.
In Brussels, a Nato spokeswoman declined to comment on the resignations.