Police were responding to a call that 73-year-old Francisco Serna was carrying a revolver and acting strangely.
Bakersfield Police Department said that the pensioner had been shot after he refused to remove his hands from his coat pocket. Sergeant Gary Carruesco told the Los Angeles Times that an officer fired several rounds and killed him.
The retired grandfather was pronounced dead at the scene at about 12.30am on Monday. No gun was found on him, but Serna was carrying a wooden crucifix.
Serna's son Rogelio later told the Los Angeles Times that his father did not own a gun and suffered from delusions brought on by his dementia.
"My dad did not own a gun. He was a 73-year-old retired grandpa, just living life," Rogelio said, "He should have been surrounded by family at old age, not surrounded by bullets."
Bakersfield Assistant Police Chief Lyle Martin told CNN that eight detectives are looking into the shooting.
"I cannot tell you enough that this community is affected by Mr. Serna losing his life," Assistant Chief Martin said, "This is a tragic incident for their family, for the community as a whole, and for the police department."
The police officer who fired the fatal shots has been placed on administrative leave as the police department investigates what happened.