How do we know time travelers aren't constantly changing the past?

We take it for granted that the past is fixed. History always happened the way we remember it happening. But how do we know for sure that that's the case?