The Turkish Foreign Minister, Ahmet Davutoglu, said it was "time Israel pays a price for its illegal actions".
Israel has refused to apologise and said its troops acted in self-defence.
"Israel, like any other country, has a legitimate right to protect its citizens and soldiers," an Israeli government official told the BBC.
"Israel has made many attempts in the recent months to settle the dispute between our countries. Unfortunately these attempts were unsuccessful."
The report was delivered to UN Secretary General Ban Ki-moon's office on Friday and would soon be published officially, a UN spokesman said.
'Null and void'
The nine pro-Palestinian activists who died were on board the Turkish-flagged ship, Mavi Marmara, when it was intercepted by the Israeli navy in international waters as sailed towards Gaza's coast on 31 May 2010.
AnalysisThe Palmer report was seen by the media in Israel as a rare vindication by the UN.
Crucially it accepted that its naval blockade of Gaza is both legal and "a legitimate security measure" to stop weapons reaching militants by sea.
An Israeli inquiry reached the same conclusion, while a Turkish one found it to be unlawful and a collective punishment of the people of Gaza.
The dilemma that remains for Israeli officials is how to handle the deepening of the rift with their long-time regional ally, Turkey, with which it has trade, military and strategic ties.
This report was meant to mend relations but has achieved just the opposite.
Turkey wants an apology and compensation for the families of the victims. Israel has expressed only regret but may consider payouts. It believes a full apology would demoralise its citizens and project weakness.
But it said Israel's decision to board the vessels "with such substantial force at a great distance from the blockade zone and with no final warning immediately prior to the boarding was excessive and unreasonable".
The report noted "forensic evidence showing that most of the deceased were shot multiple times, including in the back, or at close range".
The inquiry also found Israel's naval blockade "was imposed as a legitimate security measure in order to prevent weapons from entering Gaza by sea and its implementation complied with the requirements of international law".
The report was completed months ago, but its publication was delayed several times as Turkey and Israel attempted to repair their diplomatic relations, which have been frozen since the flotilla incident.
But on Friday, hours before the report was expected to be released, Turkey' announced that diplomatic relations had been reduced to the level of second secretary and all military agreements had been suspended.
"The time has come for Israel to pay for its stance that sees it above international laws and disregards human conscience," Mr Davutoglu said. "The first and foremost result is that Israel is going to be devoid of Turkey's friendship."
"As long as the Israeli government does not take the necessary steps, there will be no turning back," he warned.
Mr Davutoglu said the report "displayed the violence committed by the Israeli soldiers", but added that some of its findings were questionable.
"Turkey does not recognise Israel's blockade of Gaza. It will secure the study of this blockade at the International Court of Justice."
He added that Turkey would take all necessary measures to protect its shipping in the eastern Mediterranean.
Asked to comment on the UN panel's decision not to recommend that Israel apologise, Turkish President Abdullah Gul said: "To be frank, the report is null and void for us."
Turkey's embassy in Washington later said in a statement that it was "determined to defend the rights of its citizens and will take legal actions against all those responsible for the crimes committed".
Hamas, whose decision to seize control of the Gaza Strip in 2007 led to Israel imposing the blockade, applauded the Turkish move.
The Israeli official said the government accepted with "reservations" the UN report, which it considered "professional, profound and serious".
"During the events of the Mavi Marmara flotilla, Israeli soldiers boarded the boat with non-lethal means. They had no intention to hurt anyone," the official said.
"Once the Israeli soldiers were viciously attacked by dozens of violent IHH activists, armed with batons, knives and steel pipes, the soldiers had to defend themselves," the official added, referring to the IHH Humanitarian Relief Foundation, which Israel has banned for supporting Hamas.
"After many of the soldiers were wounded during the operation, nine of the IHH rioters who had endangered their lives were killed."
"As recommended in the report, Israel again expresses regret about the loss of life but will not apologise for acts of self-defence by its soldiers."
The official noted that the Israeli ambassador had finished his posting in Ankara a few days ago, and said Israel assumed that Turkey would "respect the rules of international maritime law in the Mediterranean".