I was to have returned you the book that day. 我本来预定那天把这本书还给你。（实际上由于某种原因而没有还）

       I was to return you the book that day.  我预定那天把这本书还给你。（实际上可能还了，也可能没有还）

        但是加上but I was too busy that day就可以判断两个句子意思实际上一致：

        I was to have returned you the book, but I was too busy that day.

        I was to return you the book, but I was too busy that day.

        这两个句子在意思上没有区别，都是明确表示那天没发生还书这件事。