Making it so that only an item with a certain tag can be in the inventory slot

What I have:
execute unless entity @p[nbt={Inventory:[{Slot:25b,Tags:["badge"]}]}] run item replace entity @p inventory.16 with barrier{display:{Name:'{"text":"Badge Slot","color":"gray","bold":true}',Lore:['{"text":"Slot To Hold a Badge","color":"gray","italic":true}']},Tags:["badge"]}

What I want: if an item with the tag badge is put into the inventory slot, it will stay there instead of being replaced by Badge Slot, and then when you take the item with the tag badge out of that inventory slot, the Badge Slot item comes back