Most of the victims were schoolgirls who were lined up at a bus stop near Noborito Park in the city of Kawasaki when the man in his 40s or 50s, carrying a knife in each hand, began slashing them.
An official at the Kawasaki city office told the AP that 16 people, most of them schoolgirls, were wounded and three others, including the attacker, were believed to have died.
The official said three of the injuries were serious and 13 others were not life-threatening.
NHK national television, quoting police, said that the suspect died after cutting himself in the neck.
Police would not immediately confirm the specifics of that report.
Most of the victims attended a private school founded by Soeurs de la Charite de Quebec, an organisation of Catholic nuns in Quebec City in Canada.
A witness told the Mainichi newspaper that he heard children shrieking after walking past a bus, and when he turned around, he saw a man wielding a knife in each hand, screaming "I will kill you" and that several children were on the ground.
NHK, citing police, said that a bus driver told officials that a man holding a knife in each hand walked toward the bus and started slashing children.
NHK also interviewed a witness who said he saw the suspect trying to force his way onto a bus.
The attacker's identity and motives were not immediately known.
Television footage showed emergency workers giving first aid to people inside an orange tent set up on the street, and police and other officials carrying the injured to ambulances.
Japanese Prime Minister Shinzo Abe described the stabbing spree as harrowing, adding that he is outraged and will take all necessary measures for children's safety.
Mr Abe said: "It was an extremely harrowing incident in which many small children were victimised, and I feel strong resentment... I will take all possible measures to protect the safety of children."
Donald Trump, who is on a visit to Japan with first lady Melania, offered his condolences to the victims.
"On behalf of the first lady and myself I want to take a moment to send our prayers and sympathy to the victims of the stabbing attack this morning in Tokyo," he said.
"All Americans stand with the people of Japan and grieve for the victims and their families."
Mr Trump made the comments whilst on a tour of a Japanese navy ship at the Yokosuka naval base located south of Tokyo.
Although Japan has one of the lowest crime rates in the world, it has had a series of high-profile killings, including in 2016 when a former employee at a home for the disabled allegedly killed 19 and injured more than 20 others.
In 2008, seven people were killed by a man who slammed a truck into a crowd of people in central Tokyo's Akihabara electronics district and then stabbed passers-by.
Also in 2016, a man stabbed four people at a library in northeastern Japan, allegedly over their mishandling of his questions. No one was killed.
Stay up to date with the latest news stories from a Christian perspective. Sign up to our daily newsletter and receive more stories like this straight to your inbox every morning.
