The accusation came after Kimmel sent a comedian to one of Moore's rallies at a church in Alabama to be a "supportive" heckler.
The comedian, Tony Barbieri, also known as Jake Byrd, sarcastically celebrated Moore as "a man's man".
When he was being kicked out the rally, he yelled that he was Moore's number one fan and continued: "Does that look like the face of someone who hits on teenage girls?"
Moore, who is a Christian, has been accused of sexual misconduct with young girls but denied the claims at the rally and deemed them a conspiracy from "lesbians, gays, bisexuals and socialists".
In response to the disruption at the rally, Moore criticised the Jimmy Kimmel Live host in a tweet and invited him to fight it out.
Kimmel accepted the invitation for "a fight" and said: "There is no one I would love to fight more than you.
"I will put my Christian values aside, just for you and for that fight."
Kimmel added: "It doesn't fit your stereotype, but I happen to be a Christian too.
"I made my first holy communion, I was confirmed, I pray, I support my church, one of my closest friends is a priest, I baptized my children.
"Christian is actually my middle name. I know that's shocking, but it's true.
"So if you're open to, when we sit down, I will share with you what I learned at my church."