They published the findings in the journal Behavioural Processes.
Mandrills are the fifth species of Old World monkey seen deliberately modifying tools.
Non-human apes, including chimpanzees and orangutans, can adapt basic tools for specific jobs.
One well-known example of this behaviour is termite fishing in chimpanzees, where the animals strip down grasses to make fishing rods that they then poke into termite mounds to snag the nutritious insects.
End Quote Riccardo Pansini Durham UniversityThe gap between monkeys and great apes is not as large as we thought it was”
"The gap between monkeys and great apes is not as large as we thought it was in terms of tool use and modification," he told BBC Nature.
Dr Pansini captured the footage while studying stress-related behaviour in the zoo's mandrills.
His research during that time helped inform the design of a specially landscaped enclosure, which contained shrubs to give the animals hiding places. The design won an animal welfare award in 2007.
In the footage that Dr Pansini captured, a large male mandrill strips down a twig, apparently to make it narrower. The animal then uses the modified stick to scrape dirt from underneath its toenails.
Though the scientist was excited to witness this deliberate tool modification, he said it was not entirely surprising.
"Mandrills have been seen to clean their ears with modified tools in the wild," he told BBC Nature. "This was thought to help prevent ear infections and therefore might be an important behaviour in terms of hygiene."
He thinks the captive setting may have helped bring out this behaviour.
"Animals have more time in captivity to carry out tasks that are not focused on looking for food or mating," he said. "So in zoos, you can occasionally pick up behaviours that are a little bit strange.
"In the wild this 'pedicuring' would be considered trivial," he explained. "But cleaning their ears with the same modified tool probably gave the animals some relief from the pain in their ears.
"So we're witnessing the same behaviour that's used in quite important tasks being adapted for a less important task," Dr Pansini said.
Dr Amanda Seed, an expert in primate tool-use from the University of St Andrews, UK, praised the researchers for capturing such interesting footage.
She added, though, that it was not entirely clear that the mandrill was deliberately modifying the stick for the specific goal of producing a "sharpened toenail-cleaning tool".
She told BBC Nature: "For me, the behaviour is closer to what we already know from other species, using a stick for self-cleaning purposes, than the tool modification of say chimpanzees - which rake their stick tools through their teeth to produce a brush for gathering termites.
"But these definitions are always tricky. You could say that as soon as an animal pulls a branch from a tree, they're modifying that branch."
Dr Sonya Hill a research officer at Chester Zoo, added that research findings from zoos could have a "direct impact on evidence-based conservation and husbandry practices".
"They can also contribute to a wider body of scientific knowledge, as this mandrill study has shown."