'Lone wolf' terror threat warning
Returnees from "wars in Somalia, Yemen or Nigeria" could use their experience on UK streets, Rusi said.
The Home Office said its security arrangements would reflect "the nature of the terrorist threat we face".
It is also feared that the return of Britons from overseas could coincide with the release of people convicted of terrorist charges over the last decade.
BBC security correspondent Frank Gardner says the think tank is warning of "a perfect storm of combined threats coming together in the near future".
The UK terror threat level was reduced from "severe" to "substantial" in July last year, but Rusi warned that the threat from jihadist terrorism had not diminished.
Tobias Feakin Royal United Services InstituteThere is a sense in Whitehall that major decisions are being postponed until the [Olympics have] ended in August, with an overriding priority to complete the Games without major incident”
"Although actual capabilities may have deteriorated, the intention to conduct large-scale attacks on British soil remains."
The report also warned that UK counter-terrorism spending and staffing levels could face significant cuts after the end of the London Olympics this summer.
The focus on averting a terrorist threat during the Games had postponed much-needed reform until afterwards.
"The Games are both a help and a hindrance to UK counter-terrorism," said Rusi senior fellow Tobias Feakin. "A help because they have stimulated intense co-operation between the security agencies, but a hindrance because the shadow of the Olympics disguises the landscape for the years beyond.
"As budgetary restrictions are increasingly applied across the public sector, it is almost certain that the security agencies will also have to tighten their belts."
He added: "There is a sense in Whitehall that major decisions are being postponed until the event has ended in August, with an overriding priority to complete the Games without major incident.
"After this, the changes for the various security organisations involved will be inevitable."
Long-term strategy On Monday, a report by the Home Affairs Committee of MPs warned the government not to neglect the threat to the UK from extreme far-right terrorism.
The committee cited the growth of far-right groups with links to similar organisations in Europe.
A Home Office spokeswoman said: "National security is the first duty of any government.
"The UK's counter-terrorism strategy (Contest) sets out our long-term plans to deal with the threat from terrorism. It covers the build-up to the Olympics and the following three years.
"Over that same period we are allocating £2bn a year to the security and intelligence agencies budget.
"The [Contest] strategy is designed to be flexible and we will continue to ensure that the UK's response reflects the nature of the terrorist threat we face."