I was watching the Bill Nye Science Guy doc a couple days ago, and the antagonist there (Ken Ham, who believes the Earth is 6,000 years old) was saying, in a debate against Nye, that all we can know for sure is the present, we can't observe the past directly so claims of Earth's age are mere conjecture.

This is not the kind of invisible past you're talking about (you're talking about mistakes even a rational person could make), but I was nevertheless reminded of it.