It seems to be related to the integration of amsthm into amsbook. If I remove amsthm from the list of packages, the same error message occurs (obviously). Yet, by changing the documentclass to any other (e.g. scrreprt) and using the amsthm package, the list of theorems works perfectly. How can I use amsbook in combination with thmtools?
welcome to the forum!
That question here was posted before TeX StackExchange was opened. Our forum has a long history, back to 2007.
So, a good occasion to answer it! I would do it differently to the suggestions at StackExchange.
defining the undefined \@dotsep would remove the error, yes, but the lists would be inconsistent: the list of theorems would have dots, list of tables and figures would not have dots.
copying and editing the corresponding internal macros takes time and space (53 lines there), and can lead to an error if thmstools would be changed.
So, instead of patching, we could (re)define \@dottedtocline to behave like ams book's \@tocline, that is, to call that one. We won't have dotted toc lines anyway.
Simple solution, 1 line plus the usual \makeatletter/other: