President Temer, 76 and his wife, Marcela, 33, had only stayed in the Alvorada Palace for two weeks before they decided to vacate the premises.
Temer told Veja magazine that "the energy wasn't good" and that they even tried to get a priest to drive out the "evil spirits".
The couple will now move back into the residence of the vice-president, where they had been living since 2011.
President Temer had been the country's vice-president until Dilma Rouseff was impeached last August.
The presidential palace was designed by Oscar Niemeyer and built in 1958. There are 160 employees working in the residency.