A partner at Brown Brothers, the oldest private bank in America, lives here. He attended Harvard Business School.