This sometimes occurs when you try to open a reference library. The library freezes and this error message appears.

Open EndNote without opening the library, and click on Tools, Recover Library ...
Use this feature to save a "recovered" copy of your library with a new name.