"Sure, I was paid a lot of money, but I'm not in a regular job," he said on Wednesday. "I'm in show business and I was at the top of my game."
Ross's last contract with the BBC was reportedly worth £6m a year.
The 50-year-old left the BBC in July 2010 after 13 years. The same month he signed a new deal to present a chat show for ITV.
His departure followed the "Sachsgate" affair, in which Ross and comedian Russell Brand left lewd messages on the answer machine of actor Andrew Sachs.
Ross said the pressure he felt after the incident, together with media scrutiny over his salary, contributed towards his decision to leave.
"When the money became such a big story in the papers, I didn't enjoy the scrutiny.
"I didn't think it was fair because I turned down a much bigger offer to go outside than stay at the BBC. So I felt odd I was painted as this greedy guy when, in actual fact, I turned down millions to stay there."
However the presenter said he had enjoyed working for the corporation, asserting that the BBC "still makes some of the best TV in the world".
"Its funding is a great thing and I'm glad I'm not there to be used as a stick to beat them with any more. So that's the good that came out of it," he said.
"It wouldn't have got any easier for me or [the BBC] if I stayed there. So by mutual decision we said 'let's take a break.'"
BBC 'Too defensive'
However, Ross criticised the BBC for becoming over-cautious in the wake of the Sachs affair, echoing comments made by comedy producer John Lloyd earlier this week.
Lloyd, who helmed the likes of Black Adder, Not The Nine O'Clock News and QI, accused the corporation of making programmes "by committee" to avoid causing offence.
"Sauciness is no longer allowed before 9pm anywhere on the BBC - particularly not on BBC One," he wrote in the Radio Times.
Ross argued the introduction of compliance forms to the BBC production process in the wake of "Sachsgate" was not his fault.
"Maybe I lit the fuse paper, but I didn't stack the bonfire - I think they did that to themselves," he said.
"They were far too defensive and still continue to be defensive."
Elsewhere in the hour-long interview, Ross claimed his phone had been hacked by a newspaper.
"The police contacted us and told us my number and my wife's number was on the list," he said, "and various friends, and my agents, and people we'd been calling."
"The thing is... because they hadn't got any stories out of them, there's nothing much you can do, because you have to prove some damage."
Ross added that he believed newspapers other than the News Of The World were involved, and said he would take legal action if he could prove the law had been broken.
"I would pursue that as strongly as I could."
The Jonathan Ross Show begins on ITV on Saturday.