According to AP, Shinzo Abe has also demanded the release of a second Japanese hostage, journalist Kenji Goto, a Christian, who is also being held by the extremists.
"I am left speechless," he said, stressing he wants Goto released unharmed. "We strongly and totally criticize such acts."
Abe declined to comment on the message in the latest video, which demanded a prisoner exchange for 47-year-old Goto, with a woman held in Jordan.
Abe said after a Cabinet meeting late Saturday that the government of Japan will not succumb to terrorism and will continue to cooperate with the international community in the fight against terrorism.
Japan has joined other major industrial nations in opposing ransom payments. U.S. and British officials said they advised against paying.
However, it's understood he spoke by phone with Jordanian King Abdullah II on Saturday, the state-run Petra news agency reported, without elaborating on what they discussed. He also called the two hostages' families.
Goto's mother, Junko Ishido, told NHK that in the purported message her son "seemed to be taking seriously what may be happening to him as well."
"I'm petrified," Ishido said. "He has children. I'm praying he will return soon, and that's all I want."
But Ishido also was sceptical about the voice claiming to be Goto. "Kenji's English is very good. He should sound more fluent," she said.
The Islamic State group had threatened on Tuesday to behead the men within 72 hours unless it received a $200 million ransom. Kyodo News agency reported that Saturday's video was emailed to Goto's wife.
David Cameron has pledged solidarity with the Japanese people after a video was temporarily posted online purporting to announce the murder of hostage Haruna Yukawa.
The Prime Minister said the reported execution of Mr Yukawa by Islamic State extremists is "another reminder of the murderous barbarity of these terrorists".
Yukawa was captured last summer, and Goto is thought to have been seized in late October after going to Syria to try to rescue him.
