The Soul of America: how can the past teach us about a fractured present?

How did a Pulitzer prize-winning historian, who might typically be giving lectures, signing books or roaming dusty archives, find himself making a speech at the Democratic national convention to millions of viewers?

“I think this is an all-hands-on-deck moment if you believe in the efficacy of an American experiment that, for all of its failings, is worth defending,” Jon Meacham, biographer of Thomas Jefferson, Andrew Jackson and George HW Bush, explains by phone.

Related: Facing Trump, a historian appeals to America's soul: 'I think we'll survive'

Meacham’s 2018 bestseller, The Soul of America, was presciently titled, since Biden and others have framed this year’s election as a battle for America’s soul. The book has now spawned an HBO documentary of the same name.

The film finds Meacham journeying from observer and chronicler to something of a protagonist “in the arena”, as Teddy Roosevelt put it. Director KD Davison says: “We were with him on the road for about a year and I was watching this transition take place.

“Jon has a deep sense of morality. He’s deeply spiritual and I think that where we are now is just so spiritually unnerving for people who are deeply kind. Reason is incredibly important to him but then also goodness and kindness and compassion and love. For people who are deeply compassionate, this presidency has been a difficult thing to watch. It makes perfect sense to me that’s the way he would go.”

Such is the galvanising effect of Donald Trump. Bob Woodward, another presidential biographer, broke his own protocol by declaring Trump unfit for office at the end of his latest book. Pete Souza, White House photographer under Barack Obama, cast off his anonymity to become a trenchant critic of the 45th president (the subject of another fine documentary, The Way I See It). Scientific American made the first presidential endorsement in the magazine’s 175-year history.

Does Meacham sense himself crossing the floor from political historian to political activist? “Not in any permanent way,” he says. “I think this is an existential election. Biden asked me to speak at the Democratic convention. I agreed as long as there weren’t any restrictions and there weren’t.”

The 51-year-old, born in Chattanooga, Tennessee, adds candidly: “I don’t know where I would have been on the issues of the civil rights movement if I’d been born 20 or 30 years before. I like to think I would have been on the right side but I don’t know. But I do know what the stakes of this are and I know what side I’m on and I want to be clear about it.”

The Soul of America gently traces Meacham’s life and career and devotion to the principle that the past is never dead, it’s not even past. Growing up, he recalls, he could still find bullets left over from the civil war. When he was 10 he was already campaigning for Ronald Reagan.

“I was a strange child,” he jokes. “It was a tactile thing for me. I can’t really explain the Reagan thing except that he captured my imagination and made the public square interesting. He was sort of my gateway drug. So blame him.”

Like the book on which it is based, the film is designed to calm, console and cajole to action, making the case that America has been through dark times before and emerged on the other side. It runs through the women’s suffrage movement, the incarceration of Japanese Americans during the second world war and the McCarthyite communist witch-hunts.

Meacham says: “Trump is the fullest manifestation of the darkest forces in the history of the republic and, if we go back to a phrase from the 18th century, eternal vigilance is the price of liberty. No single election is going to make all these forces go away but the task of civic life is, can you make the darker forces ebb as opposed to flow, and that’s what’s on the ballot.”

The documentary points to another example, the struggle to pass civil rights legislation in the 1960s, and in one segment Meacham is heard quoting Mark Twain’s “History doesn’t repeat itself but it does rhyme”. Many heard a rhyme in this summer’s Black Lives Matter protests after the police killing of George Floyd and, noting their scale and diversity, suggested that this time change will be more lasting and profound.

Meacham is not so sure. “Inherent in protest and reform, there are advances and there’s retreat, there’s progress and then reaction. I think it’s too early to tell. We’re in a significant moment both in terms of structural racism, structural partisanship and also what is the long-term impact economically and culturally of the virus.

“To quote Churchill, this is not the beginning of the end, it’s probably the end of the beginning. Because how many of these jobs don’t come back? What does that do to the economic distribution? I think it’s wide open. We are poised, I hope, for significant advances in combating structural racism but that requires eternal vigilance and energy.”

Dozens of statues of Confederate soldiers who fought to preserve slavery were removed or torn down during the racial justice uprising. But statues of Washington, Jackson and even Abraham Lincoln were assailed too. Where to draw the line?

“The first question is, was this person devoted to the constitutional experiment and a journey toward a more perfect union? If they were for ending the constitutional order then get off public land, take it down. So once that is clarified, then make an assessment of, on balance, is this person worthy of commemoration?

“If memorials are only for perfect people, we will have none. I don’t mean to sound overly Anglican about it but I really do think this is one where it’s case by case and requires a sense of history and the totality of a given person’s contributions.”

To take one example, Jefferson wrote that “all men are created equal” but enslaved more than 600 people and fathered at least six children with Sally Hemings, an enslaved woman. His memorial still stands tall in Washington, separated by water from a more recent tribute to the civil rights leader Martin Luther King.

Meacham reflects: “There’s nothing more American than the fact that Martin Luther King now stares in perpetuity across the Tidal Basin at Thomas Jefferson. So, yes, there should be a Frederick Douglass memorial, one of the most important Americans who ever lived. Yes, there should be Harriet Tubman. I would err on the side of additive rather than subtractive.”

On statues and contested history, as on so much else, Trump has been both clarifying and polarising. By every available metric, Meacham says, the country is more structurally divided now than it was a generation ago when Ronald Reagan won re-election by carrying 49 out of 50 states. The poll on 3 November will not produce anything like that. Nor will it fix everything overnight.

“People who think that the kingdom of heaven is going to come if the election goes one way or the other are just wrong,” the historian continues. “What we don’t know yet is whether Donald Trump was the last gasp of an angry white sense of grievance or whether there are gonna be more gasps.”

In history, Meacham appears to have found a language that takes the sting out of today’s tribal politics, echo chambers and alternate realities. He is that deeply unfashionable thing in hyperpartisan times: a middle-of-the-road moderate who can see both sides of an argument.

Davison, the director, who grew up in Texas, says: “What is so crazy to me about Jon is that he can speak to a packed room of super-progressive lefty liberals in Boston and then go down to a packed room of super-conservative rightwing folks in Alabama and have the same response. He knows where he is. He knows how to speak to people and he knows how to find a common language. It’s this really fascinating thing to watch.”

Asked how he cracked the code, Meacham responds: “Well, I say the same thing to everybody so that helps. I talk about all this stuff in a historical frame, not in a partisan one. That’s not to say it’s not inherently political but you don’t need me to tell you about the president’s latest tweet. There are plenty of people who can do that. If you have taken pains to lend an ear to what I might have to say, then I feel a certain obligation to add value to the cacophonous unfolding conversation in our country.”

It is not hard to understand why Biden tapped Meacham to speak at the convention in August. Both preach civility, bridge-building and working across the aisle, risking scorn from the hardliners on both sides. The pair speak occasionally, Meacham says. “He’s a good, good man. He’s running a sound campaign in a cataclysmic time and I think that, in so far as dignity and decency are on the ballot, if Americans want dignity and decency, they have a terrific candidate to choose.”

Would Meacham accept some kind of role from a President Biden if it were offered? The answer is not likely, but he finds a characteristically elegant way to say it: “I love my country and I will do whatever service I can do to it, but I seriously doubt that my being officially involved in anything would be good for the journey toward a more perfect union.”

• The Soul of America debuts on HBO on 27 October with a UK date to be announced