The first library in Carmel was started in 1896 by the Ladies Literary Guild also known as the Wednesday Literary Club. With the help of a school teacher named Luther Haines, the group began lending books from the telephone company building on Main Street. In 1904 an official board of trustees was formed to run the Library. They appointed the first librarian, Miss Mabel Wells, in 1904.

The library received a grant from the Carnegie Corporation to build a library building in 1911. It was built in 1913 and housed 5,847 books.

By the late 1960s, the Library had outgrown the Carnegie building. A new building was built on Main Street in 1972. More than 21,000 books were moved from the Carnegie building to the new building by a group of volunteers who called themselves “The Book Brigade.”

Carmel City Hall operated out of the old library until the late 1980s. Today the old library is a restaurant. It is said that the ghost of Isaac Bales, the library’s janitor, is still on duty. He has been seen standing next to the bar in the basement of the restaurant. One evening as the upstairs was closing, a customer saw an older gentleman with white hair and a droopy mustache come down the stairs, tip his hat and stand at the bar. The customer was about to strike up a conversation when the gentleman disappeared.