The European Commission describes it as modern slavery and says this is the tip of the iceberg.
There have been confirmed cases in six European countries, including Sweden, Norway and Belgium.
The gangs pick vulnerable men off the streets in the UK, who are often homeless and many have drink or drugs problems.
They are promised well-paid work, but are then transported abroad where they are forced into long, hard days tarmacking or paving driveways for little or no money.
One man the BBC spoke to had arrived in the Swedish port of Malmo with two other Britons who all had been homeless when they were picked up. He has asked not to be named, because he fears for his safety.
The men worked 14-hour days for little or no pay and lived in appalling, cramped conditions. They were too frightened to escape, until the Swedish police offered them help. He says there was a culture of violence.
"I've seen people threatened with pickaxes. I've seen people kicked, punched. I've nearly been pushed off a moving vehicle. It's very tense. You're waiting for the next thing to happen, " he says.
'Targeting most vulnerable' The European Commissioner for Home Affairs, Cecilia Malmstrom, says she fears this is only the tip of the iceberg.
Aidan McQuade Anti-Slavery InternationalThat physically fit British men can be threatened or coerced into working without pay and living in fear for their safety reflects the brutal reality of modern slavery”
The project manager of human trafficking at the European law enforcement agency, Europol, believes there have been dozens of British victims. David Ellero says traveller gangs have been doing this for a long time.
"[They are] targeting the most vulnerable in society and forcing them to work, but the cases are not categorised as trafficking. The work is normally carried out in northern Europe, where they work in rural areas and focus on elderly victims.
"These people are intimidated into paying for substantial work, so it is a double crime, exploitation of the victims and fraud of the person paying."
A report into human trafficking in Sweden, published in 2010, found 26 reports of human trafficking for non-sexual purposes. "In particular, these concern British and Irish tarmac and paving layers in Sweden," it says.
"The victims do not usually report personally that they have been the subject of human trafficking because they often have no confidence in the authorities that administer justice and are afraid of acts of reprisal."
'It cost him his life'Oliver Hayre, 22, from Lincolnshire, died in a caravan fire in Sweden in 2005 after working in appalling conditions for a traveller gang for more than three months.
The British inquest into his death heard how Oliver had been afraid of his bosses.
A police report prepared for the coroner said several of his friends had claimed that his savings and passport had been confiscated, and that Oliver had been beaten up over trivial matters and was only staying to keep his parents safe.
Detective Superintendent Guy Collings, who investigated his death, said: "It is my view that Oliver was most definitely the victim of trafficking by a gang of individuals who in effect kept him hostage by removing his passport and threatening violence if he did not comply."
He said Oliver was taken to hospital after an assault: "The gangs create a culture of fear, and Oliver and others like him are placed in fear for their own safety and that of their families back home."
Oliver's parents want the government to do more to stop such exploitation.
His father, Martin Hayre, said: "It's the 21st Century, we abolished slavery, but yet we haven't... my perception of the authorities is that they turn a blind eye to it and the intimidation to Oliver was real. It cost him his life."
In Belgium, the ministry of justice has said it is currently investigating a case involving British nationals being trafficked into the country for forced labour.
The BBC has also heard anecdotal accounts from soup kitchens, shelters, church groups, homeless charities, anti-trafficking organisations and trade unions which suggest that traveller gangs are also operating in Germany, Holland and Denmark.
Dr Aidan McQuade, director of Anti-Slavery International, says the BBC's investigation shows how vulnerable, often homeless, people were being targeted for forced labour.
"That physically fit British men can be threatened or coerced into working without pay and living in fear for their safety reflects the brutal reality of modern slavery," he says.
"The widespread nature of the problem means that it is essential that local police officers consider new approaches to investigating this crime, such as regular meetings with homeless charities, soup kitchens and migrant drop-in centres to identify risks and potential victims of trafficking, and gather intelligence on gangs seeking to exploit vulnerable people for forced labour."
He believes that the British and other governments should be doing much more to combat the problem. That is certainly on Commissioner Malmstrom's agenda. "This is not worthy of Europe today," she says, "and we should do everything to prevent it."
A Home Office spokesman said: "The government is committed to tackling human trafficking and preventing the harm it causes to vulnerable members of our society.
"The National Crime Agency on establishment in 2013 will have a key role in building on the existing arrangements for tackling human trafficking. Its enhanced intelligence capabilities and co-ordination functions will target the organised criminal gangs involved in human trafficking, wherever they are."
Yvonne MacNamara, director of the Irish Traveller Movement in Britain, said her organisation condemned "absolutely slavery and forced labour, not least because travelling people have been subjected to slavery and forced labour throughout their history, including recent history.
"If individuals are suspected of criminality, they should be subjected to the full force of due process and the law."