Sorry to be a little rude but if a time traveler altered time and you didnt go that would create a loop paradox something extremely dangerous to you and everyone in that time zone.
That's not rude. It's questioning and challenging concepts and ideas, which ultimately results is greater understanding and refined perception. That's a good thing.
If someone was acting as an imposter or decoy, or for whatever reason had to go back in time pretending to be you, they would have to study and know you in order to mimic you properly as to fool whoever they intend to fool. Their entire operation is dependent on their ability to pretend to be you. Everything they base it on is ultimately dependent on their perception of you.
Thus if they are basing it on what they learn from studying you, you can use it as to discern whether it is you or someone else by emanating bullshit. You know yourself and life, thus you know what is bullshit and what is not. They don't. When they try anyway, it shows.
If they had to be there to know something, but do not have access, they have to research and learn what transpired. If their sources are false, they will base your acting on a false premise and subsequently eventually demonstrate their ignorance, demonstrate that they do not know because they were not there.
When tracking bits and pieces of information through networks and associations, events and circumstances, people place and things, etc etc... across time loops, this effectively facilitates a means of deductive reasoning, a way of cutting through the bullshit.
For example; You and four of your friends spend a weekend at a cabin in the middle of nowhere where an alcohol induced ridiculous nonsensical conversation transpires. You discuss "Billy bat builder building billy-bats before breakfast because Bertha beats beets behind barns before breakfast because bovines bite beets but burp beet breath."
If you were to ever hear "Billy bat builder building billy-bats before breakfast because Bertha beats beets behind barns before breakfast because bovines bite beets but burp beet breath" again, you know it HAD TO originate from one of those four friends you were with at the cabin, or that someone was eves-dropping on the conversation.
It narrows it down real fast.
If you openly tell a story about a weekend party stay at a cabin in the woods with friends partying, but change it to "Peter piper picked a peck of pickled peppers and painted pinatas", then anyone studying you, mimicking, pretending they were there that weekend, is claiming "Peter piper picked a peck of pickled peppers and painted pinatas" and not what actually happened. They show they are an imposter because they do not know the real conversation about "Billy bat builder building billy-bats before breakfast because Bertha beats beets behind barns before breakfast because bovines bite beets but burp beet breath."