The county was formed in 1851 from parts of Buncombe County and Yancey County. It was named for James Madison, fourth President of the United States (1809–1817).