It wasn't that part - it was the 'invisible and intangible' aspect of things. Trapping people in the near past or future where you are, too, is just asking for trouble, because they can continue their battle with you, and it'll be your present self who ultimately gets affected - unless they're basically turned into phantoms. Then, it's an ingenious game of keep-away - they can't stop you until they catch up with you, and they can NEVER catch up with you, because one is always a little bit ahead and can't backtrack, while the other is always a little bit behind and can't catch up.