The selection of Archbishop John marks the first time a bishop from his diocese has been chosen as the Archbishop of Wales.
Dr John said: "I am overwhelmed and humbled. I would like to thank members of the College and especially my fellow bishops for the confidence and trust they have shown in me.
"We will work together as a team to grow and strengthen the Church as it serves the communities of Wales and helps build the kingdom of God."
John Davies will juggle the new role with his current position, which he has held for nine years. Details are expected in due course of his enthronement service at Brecon Cathedral.
Dean of Brecon, Dr Paul Shackerley said: "I am delighted with the news that Bishop John has been called to be our next Archbishop.
"He has proven gifts and experience to lead the Church into the future and will receive our full support and prayers he prepares to exercise his weighty, yet joyful, archiepiscopal ministry.
"I feel the future of the Church in Wales is in good hands with all our faithful Bishops, to lead us with hope into the future that we may flourish and serve the communities in which we are called."
During a secretive meeting at Holy Trinity Church in Llandrindod Wells, Archbishop John secured the necessary two-thirds majority of votes from members of the Electoral College.
Born in Newport and educated as Bassaleg Grammar School, Archbishop John read law at the University of Southampton, before being admitted as a solicitor in 1977.
In a career change, he became ordained in 1984 before being going on to be appointed Dean of Brecon in 2000. Archbishop John was named Bishop of Swansea and Brecon in 2008.