The jurist and the second most senior Associate Justice of the Supreme Court of the United States lived here.