Penn’s edgy good looks and undeniable talent might have secured him a place among Hollywood’s A-list leading men; instead, he has largely neglected headline roles in big-budget films in favor of decidedly unheroic roles in darker, more understated films, with varying degrees of success. He starred as a lovesick, jealous husband in Nick Cassavetes's She’s So Lovely (1997), co-starring John Travolta and Penn’s real-life (second) wife, Robin Wright Penn (whom he fell in love with while filming State of Grace in 1990 and married in 1996 after a turbulent five-year romance). Although Penn (who also served as executive producer) won the Best Actor Award at the Cannes Film Festival, She's So Lovely did not attract a wide audience. Penn appeared in two other major films in 1997: the poorly received U-Turn, directed by Oliver Stone and co-starring Nick Nolte and Jennifer Lopez, and the hit action-thriller The Game, starring Michael Douglas. In Hurlyburly (1998), Penn reprised a role he had played on the Los Angeles stage in 1988, co-starring with Kevin Spacey as a brutal, misogynistic Hollywood agent. In 1998, Penn also starred in the critically acclaimed World War II drama, The Thin Red Line, directed by Terrence Malick.

In 1999, the unpredictable Penn took Hollywood by surprise when he garnered a second Academy Award nomination for Best Actor, turning in another searing, darkly complex performance as the dissolute jazz guitarist at the center of Woody Allen's Sweet and Lowdown, alongside fellow Oscar-nominee Samantha Morton. In 2000, he starred in the romantic Up in the Villa, with Kristen Scott Thomas, and The Weight of Water. His third directorial feature, the thriller The Pledge, starred Nicholson and Robin Wright Penn.

Penn and Robin Wright Penn live in Marin County, California, with their daughter, Dylan, and son, Hopper. Wright Penn, best known for her roles in The Princess Bride (1987), Forrest Gump (1993), and Message in a Bottle (1999), also appeared in Hurlyburly and The Crossing Guard. Penn’s mother, Eileen Ryan, is an actress who appeared in Magnolia, released in late 1999. His father, Leo Penn, who died in 1998, was an actor-turned-director who was blacklisted during the 1940s and 1950s when he refused to name names of Communist sympathizers in Hollywood. Penn has two brothers: Christopher, an actor who appeared with his brother in At Close Range (1986), and Michael, a rock musician.