Donald Miralle/Getty James Caan and Hugh Hefner in 2003
James was previously married four times: first to Dee Jay Mattis, from 1961 to 1966, then to Sheila Ryan in 1976. He later tied the knot with Ingrid Hajek in 1990. The two divorced in 1995, and he wed Linda Stokes later that year.
"I have four wives and five kids. I apparently don't know the difference [between sex and love]," James joked in his 2003 Esquire interview.
Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human-interest stories.
Ron Galella Collection via Getty James Caan and family at the premiere of Misery in 1990
James' son Scott Caan spent time at the Mansion with his father when James briefly lived there during Scott's childhood.
In a 2010 interview with PEOPLE, James recalled with a laugh, "I'd have Scott bring me girls."
"No one could say no to a cute little boy," he added of Scott, now 45.
Scott's mother is Ryan, who died in 2012 at age 60.
Bei/Shutterstock Hugh Hefner and James Caan in 1983
A tweet from James' official account Thursday read, "It is with great sadness that we inform you of the passing of Jimmy on the evening of July 6."
"The family appreciates the outpouring of love and heartfelt condolences and asks that you continue to respect their privacy during this difficult time," the tweet added.
A cause of death was not made immediately available.
James is survived by sons Scott, James, Jacob and Alexander, as well as daughter Tara.