A cross-party group of senior political and religious figures has urged Boris Johnson and Secretary of State Brandon Lewis to "urgently create the space for local politics to regain the initiative" in Northern Ireland.
Police came under sustained and violent attack on both sides of a Belfast peace wall earlier this month following successive nights of violence.
Loyalists staged protests at the so-called border in the Irish Sea following Brexit, claiming the Northern Ireland Protocol has undermined the region's place within the Union.
There is also anger at a decision not to prosecute Sinn Fein members for alleged coronavirus regulation breaches at a funeral last year.
Protests - many of which passed off peacefully - were paused following the death of the Duke of Edinburgh, but resumed last week.
Police came under attack in the Sandy Row area of south Belfast on Friday; however, the disorder was described by officers as "minor".
A joint letter signed by Lord Patten, four former secretaries of state, a former chief constable and former Archbishop Robin Eames calls for urgent action, including engaging directly with local parties and communities.
They warned that "nothing is more dangerous than a political vacuum" and they need to "listen and be seen to be listening, not intermittently, as has been the case, but continuously".
The signatories warned that without that, there is a very real risk things could "fall over" in Northern Ireland.
"With our long collective experience we are extremely worried that violent unrest on the streets of loyalist areas and at interfaces is a consequence of politics, both in Stormont and in Whitehall, failing the people of Northern Ireland," they wrote.
"We stress that the peace process did not end with the Belfast (Good Friday) Agreement.
"Tony Blair and Gordon Brown recognised that, as did John Major before them.
"They each took personal charge of the peace process, convening regular summits and being in constant touch with all the parties."
Mr Johnson is warned in the letter that recent unrest "will not literally burn itself out over time".
"While the violence is unacceptable, the fact is that there are grievances, real and perceived, within the broader unionist community and that cannot be ignored by the UK Government," they wrote.
"Politics must be made to work again and, crucially, politics must be seen to work again.
"There is a strong sense within loyalist and unionist communities that no-one is listening to them, and that nobody in authority in Whitehall has been honest with them about the consequences of Brexit.
"The most immediate step is therefore for the Government, at the highest level, to be seen to take an interest.
"As a matter of urgency, the Government must address the outworking of the protocol with Brussels."
The letter said Mr Johnson and Mr Lewis "also need to rebuild trust with Dublin, which has been badly damaged, and to breathe fresh life into the British Irish intergovernmental process that is a key part of the Belfast (Good Friday) Agreement".
"The lesson of the past 50 years or more in Northern Ireland is that, if there is no forward movement, things do not stand still: they fall over," they wrote.
"It is the responsibility of the UK Government to ensure that does not happen because there is nothing more dangerous than a political vacuum."
The list of signatories includes former secretaries of state Lord Hain, Lord Mandelson, Lord Murphy of Torfaen and Shaun Woodward, as well as former chief constable Sir Hugh Orde.
It was also signed by Lord Patten of Barnes, architect of the reform of policing in Northern Ireland in 1998.