Sir Desmond Swayne, who was Parliamentary Private Secretary to Mr Cameron for 7 years, has been speaking as the Conservative leader hands his resignation to the Queen.
Mr Cameron exited the famous black door of 10 Downing Street with his family for the final time on Wednesday telling the gathered media that he was leaving the country "much stronger" and the economy "immeasurably stronger" after his six years in office.
He paid tribute to Samantha, who he described as "the love of my life" who had "kept me vaguely sane".
Mr Cameron concluded: "It has been the greatest honour of my life to serve our country as Prime Minister over these last six years and to serve as leader of my party over 11 years.
Mr Swayne, was at Downing Street for Mr Cameron's last evening there.
He said: "He certainly didn't want to go but he's been remarkably cheerful about it in the last few weeks.
"I went to dinner at Downing Street last night and it was a tremendously cheerful and optimistic affair, there was no tears or regrets or remorse at all."
Earlier on Wednesday, David Cameron hosted his last ever Prime Minister's Questions in the House of Commons.
He called on his successor to stay as close to the European Union as possible.
Opposition leader Jeremy Corbyn praised him for his role in changing the law on gay marriage and securing the release of Shaker Aamer who was held by the US at the Guantanamo Bay detention camp in Cuba.
On answering his final question, Mr Cameron was greeted with a standing ovation from much of the chamber.
Speaking about how he'll be remembered, Sir Desmond Swayne said: "It's almost too soon to start talking of legacy because you don't know what's going to happen next.
"One of the things that might an enormous legacy is going to be the fact he gave us the referendum on the EU.
"But we won't know the nature of that legacy for 10 years or so."
David Cameron will continue in his role for MP for Witney. There has been suggestions he may even stand again at the next election.
Listen to Sir Desmond Swayne speaking to Premier's Antony Bushfield here: