Adding temporary modifiers to armor

I’m struggling with figuring out how to add modifiers to armor temporarily, more specifically how to remove modifiers. See, I need players with specific tags to get extra defense from leather armor, for example. However, when they unequip the armor or give it to another person, its name, durability, enchants, etc. need to be untouched but its armor bonus should be reset. I would be OK with it fixing itself when a player without the tag equips it via inventory changed advancement too, really I just want to know how to remove the item modifier because I can’t figure that out (aside from moving to armor stand, modifying, moving back, but i want to avoid that if possible)