They had failed in what they had set out to do.

had set out to do 早于 had failed，所以，使用过去完成时。

句子原本用 failed，但语境中有另外一个过去的参照动作或时间（未出现），因此使用 had failed，说明早于这个过去的参照动作或时间。

总之，都是比出来的：had failed 早于那个未出现的参照物，而 had set out 早于 had failed。先着手打算做某事，后来失败未做成。