Oliver was born in Westminster, London. His mother, Betty Jackson, is a fashion designer, and his father, David Cohen, is his mother’s business partner. His mother is English, and from a Protestant background. His father is from an Egyptian Jewish family that moved to France in the 1950s; they were Orthodox Jews. Oliver attended French school at the Lycée Français Charles de Gaulle in London and says he has a slight French accent.
He attended the Youngblood Theatre Company at weekends, then began washing vases for a florist as a job to financially support himself between auditions, stating: “It wasn’t great but when you’re that broke you’ll do anything.” He attended the Lee Strasberg Theatre and Film Institute in New York City due to not getting into drama school in London. His aim was to “do a foundational year and then reapply in London” but left after four months because he was offered a job.
Oliver landed a small role in the television series, Hollyoaks when he was 15, and he later appeared in the ITV series, The Time of Your Life in 2007. In 2008, he played Phillip White in the BBC adaptation of Lark Rise to Candleford and in the first episode of Bonekickers. He also played Marcus in the short film, The Rooftopsmiths by Len Rowles with Natasha Freeman as Imogen and Philip Marden as Joel. In 2010, he played Damon in the romantic comedy film, Going the Distance starring Drew Barrymore and Justin Long, and a hitman in the action film, Faster, opposite Dwayne Johnson. In 2011, he starred as Prince William, Duke of Cambridge in a series of Funny or Die videos with Allison Williams as newlywed Kate Middleton. Oliver then starred opposite Cynthia Nixon in the miniseries, World Without End as Ralph. He also appeared in Mr Selfridge with Jeremy Piven where he plays the role of Roddy Temple.
In 2013, Oliver was cast as journalist Jonathan Harker in the NBC series Dracula. In 2014, he was cast as Misha in the film adaptation of Shamim Sarif’s 2004 novel, Despite the Falling Snow. In March 2014, Oliver was cast as James, Duke of York in ITV’s four-part period drama miniseries, The Great Fire.
It was announced in July 2014 that he would star opposite Sarah Snook in the 2015 ABC convict miniseries, The Secret River, an adaptation of Kate Grenville’s 2005 novel of the same name. In 2015, Oliver also starred in The Healer as Alec Bailey. In the same year, he was cast as Lucas in Emerald City, which premiered in 2017 and was cancelled after one season.
In 2018, he starred in the anthology web television series, The Haunting of Hill House. The first season of the series received critical praise. He next starred opposite Elisabeth Moss in the science fiction horror film, The Invisible Man, portraying the titular role; the film was released in 2020 to positive reviews. He can currently be seen in the second entry in The Haunting series, The Haunting of Bly Manor, as Peter Quint.