Over the past week a number of Christian villages have been recaptured from the militants by Iraqi forces as they attempt to take back the city of Mosul.
But Hani El Mahdi from Catholic Relief Services, which works with CAFOD, told Premier the mess left by the jihadis meant the towns would not be inhabitable.
"It will definitely not be a matter of just weeks or months before these people can go back," he warned.
Iraqi forces backed by US-led coalition air strikes have recaptured a number of small Christian towns and villages in the past week as it battles to retake the northern city of Mosul.
IS fighters have abandoned most without putting up a challenge but the army has been faced with a trail of booby traps and roadside bombs.
Most churches have been destroyed or converted to Islamic State facilities and almost all crosses have been torn down.
El Mahdi said the militants had also damaged most infrastructure like water and electricity: "It will take time. Those refugees will unfortunately have to live this situation for some time to come until their villages and towns are reconstructed and more importantly, seen as a safe haven.
"People will not go back until they are sure their areas are secured. Even in the 'liberated areas' people were not able to return in large numbers basically because of security and lack of infrastructure."
Hundreds of thousands of Christians were forced to flee Islamic State militants as they started to take large parts of Iraq in 2014.
The terrorists warned them to pay a tax, convert to Islam or face death.
Listen to Premier's Antony Bushfield speaking to Hani El Mahdi:
