My colleague, Mr. Hill, has a small but well-chosen library, where it is said （that） he spends most of his spare time developing his mind.

这个定语从句仍然是嵌入式定语从句，where为it is said的外置主语从句中作地点状语。在嵌入式定语从句中，关系词如果在从句中作主语，that是必须省略的。但如果关系词作主语以外的成分，则that通常省略，但不省略也是可以的。

有人会认为it is said为插入语。这种观点只能解释部分从句，对有些定语从句就无法解释。用嵌入式定语从句的观点可以解释所有此类从句。哪种解释更为合理，你自己决定。

我们可以用一个并列分句替换这个定语从句：

My colleague, Mr. Hill, has a small but well-chosen library, and it is said that he spends most of his spare time there/in the library developing his mind.

你可以清楚看到，定语从句本身就是个主从复合句。there/in the library是that从句中的地点状语。把并列句改为定语从句时，只需用关系副词替换指示副词there，并引导这个分句即可。that可以省略，也可以保留。