Your memories of this “context” could have been altered. Maybe this instance of reality just started (and as such there is no context)
So it doesn’t matter if the past really is real, if you think they are real, they matter?
The act of inserting this context into the system is itself context, so I’m still existing in the context of what came before
If somebody tricks you and you fall for it, it doesn’t mean what you were tricked into believing isn’t context, or that the fact you were tricked at all isnt context.
That’s basically the Boltzmann brain thought experiment: https://en.m.wikipedia.org/wiki/Boltzmann_brain