According to their web site, you can visit the place without a US visitor visa from the Canada side.
As the only entrance to the library is in the US, all non-American visitors passing through US territory to enter the library must understand that their movements will be monitored by the US Border Patrol and the RCMP.
Passports and visas are not necessary to enter the Haskell Free Library & Opera House. However, please be aware that both the US Border Patrol and the RCMP have the right to ask to see personal identification and detain anyone who is not of legal status. We do encourage you to bring your papers with you, just in case.