Faith leaders, residents, tourists, police and politicians gathered at Trafalgar Square for the vigil, which was led by London Mayor Sadiq Khan.
He told the mourners: "Those evil and twisted individuals who tried to destroy our shared way of life will never succeed and we condemn them.
"The victims were people who came from all corners of our world. This is a time to express our gratitude to the heroism of our police officers and emergency services, who ran towards danger to help, and at the same time they encouraged others to run for safety.
"London is a great city, full of amazing people from all backgrounds. When Londoners face adversity we always pull together."
Christian MP and Home Secretary Amber Rudd paid tribute to PC Keith Palmer who was stabbed to death by the attacker: "He was courageous, he was brave, and he was doing his duty.
"And he was not alone in doing that. I know that all officers of the Met are like that and in my experience, so are all policemen.
"I want us to say thank you to them all for the great sacrifice and risks they take to keep us safe."
Candles were lit and a minute's silence was observed for the victims who died in the attack.
This morning police named the fourth victim of the attack as Leslie Rhodes, 75, from Streatham. He died overnight in hospital.
Assistant Deputy Commissioner Mark Rowley also said that 50 people were injured in Wednesday's attack with two people remaining in hospital in critical condition.