The Walters (father and son) collected a lot of art around the time of the Civil War and afterwards. When the son died, he left all that art to a public museum. It's not the largest museum, but it has the highest quality art. And matches my own tastes quite well. They have the largest collection of Gérôme's work in the US. It's a very diverse collection, but it tends towards a smaller amount of excellent examples rather than large numbers of more mundane works.
The Peabody Library is right around the corner, and also worth a stop.