The paintings - including the 1884 work Congregation leaving the Reformed Church in Nuenen and the 1882 View of the Sea at Scheveningen were discovered in Italy last year.
Congregation Leaving the Reformed Church in Nuenen, Vincent van Gogh, 1884 - 1885
Italian police found the priceless paintings wrapped in cotton sheets, stuffed in a box and hidden behind a wall in a toilet during an investigation at a farmhouse belonging to suspected mobsters.
The police were searching the property in Naples for cocaine.
Investigators say they seized £20 million in assets linked to two Camorra drug kingpins, Mario Cerrone and Raffaele Imperiale.
"After years shrouded in darkness, they can now shine again," Dutch minister for education, culture and science Jet Bussemaker said as the paintings were revealed at the museum.
Both paintings received remarkably little damage despite being ripped from the wall by thieves in 2002.
View of the Sea at Scheveningen by Vincent van Gogh 1882
"It is not only a miracle that the works have been recovered but it's even more miraculous almost that they are in relatively unharmed condition," Rueger said.
Van Gogh painted the church in Nuenen to cheer up his mother who had broken her leg and had to stay in bed.
The church was where Van Gogh's father served as a minister.
"He had painted as a gift to his mother, so it's a very personal and emotional connection," Rueger said.
The museum is home to dozens of works by Van Gogh, whose paintings fetch millions of pounds on the rare occasions they come up for auction.
"The security, I can assure you, is of triple-A quality now, so I'm very confident that everything is safe in the museum," Rueger added.