In 2013, Breyer, along with Justice Ruth Bader Ginsburg, faced calls to retire.