Wandering trader interact check?

Long story short, I was using invisible villagers for a survival datapack, because I wanted to use their trade menu as a terminal to unlock upgrades. That worked all well and good, but I realized there was no way to get rid of their profession XP or make them immune to the hero of the village effect. So, I swapped to wandering traders. They work better in all aspects except one, the "minecraft.custom:talked_with_villager" scoreboard I was using does not activate from wandering traders. I want to prevent players from trading with wandering traders that dont belong to them, I was doing this by detecting when they talked to a villager that wasn't theirs and immediately killing and replacing the villager, but now that I can't check for the talking, I need an alternate method.
TL;DR if anybody knows a way to either check when a player talks to a wandering trader, or another way to prevent players from talking to wandering traders that are not linked to them via scoreboard, please let me know :)
of course, the console wandering traders have special tags and everything.