He gave no details of the agreement, but said initial talks would be held in Washington next week.
The last round of direct talks broke down nearly three years ago over the issue of Israeli settlements in the West Bank and East Jerusalem.
Mr Kerry told reporters in Amman that the parties had "reached an agreement that establishes a basis for resuming direct final status negotiations between the Palestinians and the Israelis."
"This is a significant and welcome step forward," he said, but stressed that it was still "in the process of being formalised".
'Difficult road ahead' Chief Palestinian negotiator Saeb Erekat and his Israeli counterpart Tzipi Livni will travel to Washington "in the next week or so" to begin initial talks, Mr Kerry said.
AnalysisIn his brief statement, Mr Kerry was able to announce the breakthrough that he had worked towards over six trips to this region: a resumption in direct peace talks between Israel and the Palestinians. However, he gave few clues about how they might succeed.
During the day, Palestinian officials told me that they were seeking "written clear commitments" that negotiations on borders would be based on pre-1967 ceasefire lines. They also sought reassurances about what would happen if they dropped a demand for a freeze on Jewish settlement building in the occupied West Bank and East Jerusalem.
Meanwhile, Israeli journalists speculated that talks would restart with no preconditions - as their government has demanded. But some also anticipated the release of hundreds of Palestinian prisoners from Israeli jails.
It could take time before the facts become clear. Mr Kerry is now flying back to the US; in Israel, the Jewish Sabbath has started, and Palestinians are also on their weekend.
Earlier on Friday Mr Kerry made an unscheduled visit to Ramallah in the West Bank, to meet Palestinian President Mahmoud Abbas for the third time this week.
The secretary of state had also spoken by phone to Israeli Prime Minister Benjamin Netanyahu, and praised both leaders' courage.
"The representatives of two proud people today have decided that the difficult road ahead is worth travelling," he said.
The BBC's State Department correspondent Kim Ghattas says the agreement sounds fragile, and could yet be derailed before talks begin.
But these will be the first direct talks since the negotiations broke off in 2010, she adds.
Mr Kerry has used the Jordanian capital as a base for this whirlwind four-day trip to the region, twice postponing his departure.
On Thursday, US President Barack Obama spoke to Mr Netanyahu and encouraged him to continue to work with Mr Kerry, the White House said.
Stumbling blocks Since taking office in February, Mr Kerry has paid six visits to the Middle East in an effort to restart talks between Israel and the Palestinians.
He has said that time is running out for a two-state solution to their decades-old conflict.
The issue of Jewish settlements in the West Bank remains one of the biggest stumbling blocks between the two sides.
The Palestinian president has said that Israel must freeze settlement-building before stalled peace talks can resume, while Mr Netanyahu has urged Mr Abbas to return to talks without preconditions.
There was an angry reaction in Israel earlier this week when the European Union approved new guidelines which prevent Israeli projects in occupied territories from receiving European grants or prize money.
Israel gave its final approval on Wednesday for the construction of more than 700 new settlement homes in Modiin Ilit, halfway between Jerusalem and Tel Aviv.
Earlier this year, Mr Netanyahu ruled out removing any Jewish settlements in the West Bank and East Jerusalem ahead of elections in Israel earlier this year. Following the vote, Mr Netanyahu remained prime minister, leading a new coalition.
Tzipi Livni, Israeli foreign minister under centrist former Prime Minister Ehud Olmert, returned to cabinet and will act as chief negotiator in any talks with the Palestinians.
Ms Livni responded optimistically to Friday's developments on her Facebook page, saying "four years of diplomatic stagnation" were about to end, after "months of scepticism and cynicism".
British Foreign Secretary William Hague, welcoming Friday's announcement, paid tribute to Mr Kerry's efforts and commended "the leadership shown by both Prime Minister Netanyahu and President Abbas".
The settlement issue is just one of several thorny problems which have stymied previous attempts to get the two sides back to the negotiating table.
The Palestinians have also demanded previously that any talks be on the basis of the borders in place from 1949 to 1967, when Israel seized the West Bank, Gaza and East Jerusalem.
The future status of Jerusalem and any "right of return" for Palestinian refugees and their descendents are also core issues in any future peace talks.
The Palestinian position is further complicated by the the fact that Gaza is ruled by the Islamist group Hamas, whereas Mr Abbas's Fatah movement is in control of the West Bank.
Hamas has rejected the announcement of a return to talks, according to AFP, saying Mr Abbas had no right to negotiate on behalf of the Palestinian people.