The singer, songwriter, and composer Randy Newman had a fascination with the legend of Faust that approached obsession. Beginning around 1981, he worked for some 15 years on an original retelling of the much-retold story of spiritual brokerage. In his version, which he conceived as a musical dark comedy, the Lord and the devil make a bet for eternal custody of the soul of an impressionable student at the University of Notre Dame. Productions were mounted at the La Jolla Playhouse in San Diego and the Goodman Theatre in Chicago. Critics found much to admire in the songs, while theatergoers sniffed and turned away; the customary dream of a Broadway opening went unfulfilled.

When I was working for Entertainment Weekly in the ’90s, we regularly surveyed readers on the likability of our content. The week we covered Randy Newman’s Faust, that article was ranked “least appealing.” The editors puzzled over this and decided not to blame Faust. Could there be something about Randy Newman that people just didn’t like? Listening closely to the music he has made over many decades in all its varied forms, I’ve since come to accept that Randy Newman’s greatest work is indeed awfully hard to like. But I think its very unlikability is what makes it great.

For more than 60 years now, Newman has been composing and performing highly distinctive and meticulously wrought work: songs for pop (or poplike) albums that, at their best, have been intellectually sophisticated, unorthodox, and unsettling—as well as spit-take funny and profoundly emotive. In his new book, A Few Words in Defense of Our Country: The Biography of Randy Newman, Robert Hilburn, a former pop-music critic for the Los Angeles Times, set out to come to terms with Newman’s life and work. His book is straightforward, helpful in clarifying the intentions underlying Newman’s most challenging songs. This is an authorized telling, written with the participation of its subject, who contributes comments with restrained candor and wry, arch wit. Hilburn, whose previous books include solid, comprehensive biographies of three other major songwriters of the rock era—Johnny Cash, Paul Simon, and Bruce Springsteen—knows his territory. It’s a realm in which Newman clearly fits, being gifted, white, and male, but where he is also an outlier, a pop composer steeped in classical music and Tin Pan Alley (New York’s turn-of-the-20th-century popular-music publishing scene), and a late-blooming performer who—unlike Hilburn’s previous subjects—has never been much of a rock star.

[Read: The least-known rock god]

Newman has had several overlapping careers: In his earlier years, he found success writing songs for others to sing—a successor to tunesmithing specialists such as Irving Berlin, Jerome Kern, and Dorothy Fields. Then, as singer-songwriters came into vogue in the late 1960s, he took up performing and went on to make more than a dozen solo albums. Along the way, he found himself moving into the de facto family business, composing instrumental scores for feature films, as his uncles and cousins did for hundreds of Hollywood movies. (Randy’s uncle Alfred Newman alone wrote more than 200 scores and got 45 Oscar nominations.) Through the scale of their exposure, rather than the depth of their insights, the songs Randy Newman wrote for hit kid-friendly movies—“You’ve Got a Friend in Me,” from Toy Story; “I Love to See You Smile,” from Parenthood—are surely what he is best known for today. “I Love to See You Smile” was even licensed for a Colgate commercial. There’s an aural smile in the tune, though I can’t say I love to hear it play.

Although a great many other songwriters are equally adept at composing toothpaste jingles, sing-along ditties for kids’ movies, and orchestral scores for films, few songwriters of any period compare with Newman for the psychological complexity and the dramatic force of his greatest songs, many of which are also his most harrowing. As a backroom writer, he had already demonstrated an extraordinary ability to conjure landscapes of despair, as in “I Think It’s Going to Rain Today,” sung by Judy Collins and Dusty Springfield: Tin can at my feet / think I’ll kick it down the street / that’s the way to treat a friend. Newman began to deal with bleaker themes in his first solo album, released in 1968, which he closed with “Davy the Fat Boy,” a story told by a man entrusted with the care of his childhood friend, whom he turns into a carnival freak attraction.

In his third album, Sail Away, he conjured a wide range of emotional and social nightmares by employing multilayered irony and unreliable narrators: “Political Science,” a flag-waver about the fun in dropping the bomb—Boom goes London, boom Par-ee! More room for you and more room for me; “You Can Leave Your Hat On,” a burlesque entreaty to kink; “God’s Song (That’s Why I Love Mankind),” sung by a sadistic Lord who revels in contempt for his worshippers; and nine more, including the title song, a mordant—and infectious—rallying cry for the slave trade. In the height of the singer-songwriter era, when earnest autobiographical confessions were prized as tokens of authenticity, Newman’s use of sarcasm and unlikable protagonists was an act of literary radicalism in pop music. At the same time, the sheer unpleasantness and grimness of his story songs made them elementally darker than the stagy goth cosplay of Alice Cooper and the scowling, black-cloaked metal bands that rock critics drooled over. The darkness of Newman’s work was internal and subtextual: a horror from within, not painted on for show.

Newman’s most audacious songs are, by intent, hard to stomach. They’re deceptively amoral morality tales that Hilburn casts as social commentary: critiques of racism, avarice, political violence, and the hollow pettiness of American life. As Newman once said about “Roll With the Punches,” a song sung in the voice of a cruelly racist, xenophobic jerk, “I disagree completely with everything the guy says in the song.” If listeners don’t like what they hear, that means the songs succeeded. They’re not here to be liked. They’re here to be confronted.

[Read: Bob Dylan reveals himself through 66 songs]

Newman, who is now 80, has continued to make solo recordings probing the discomforting lower reaches of the human psyche. (His most recent album, Dark Matter, was released in 2017, and includes an attack on Vladimir Putin.) Though his albums stand as testaments to his work as art, the airy songs he has written for Monsters, Inc., A Bug’s Life, Cats Don’t Dance, the Cars series, the Toy Story franchise, and other movies stand for his art as work. They’re good for business, and he knows it. In a vivid scene in his book, Hilburn describes the moment when John Lasseter, the director of Toy Story, first shows Newman finished scenes from the film. Newman pulled his wallet from his jacket, laid it on a table, patted it, and said in a stage whisper, “This movie is going to be very nice to you.”

There are moments of great warmth and beauty in Newman’s songs for family films. I think immediately of “When She Loved Me,” from Toy Story 2, the ballad of lost love sung by a doll about the girl who outgrew her. The first time I heard it, at a multiplex with my kids, I broke out crying in the theater, and remembering it now chokes me up a bit. Not all of Newman’s movie songs are simplistic or corny, though some certainly are. I don’t believe he should write nothing but songs about America’s violence and racism. But, looking over the course of his career, I can’t help but see it as the Faust story in reverse: the hero selling his soul to the God of goodness and light.