He spoke as though I were a child that needed to be distracted. I was sore, but not with him so much as with myself. I thought of the happy life that pair had led in the cosy studio in Montmartre, Stroeve and his wife, their simplicity, kindness, and hospitality; it seemed to me cruel that it should have been broken to pieces by a ruthless chance; but the cruellest thing of all was that in fact it made no great difference. The world went on, and no one was a penny the worse for all that wretchedness. I had an idea that Dirk, a man of greater emotional reactions than depth of feeling, would soon forget; and Blanche's life, begun with who knows what bright hopes and what dreams, might just as well have never been lived. It all seemed useless and inane.

这一句it seemed to me cruel that it should have been broken to pieces by a ruthless chance 中，should have been broken = should have done/been done 是推测还是虚拟语气呢？怎么理解其用法和含义？

另外，but not with him so much as with myself. 可否说成  but not so much with him as with myself. 呢？

no one was a penny the worse 怎么理解这一结构和含义？  who knows 是插入语吗？为什么是have never been lived 被动语态呢？