Change CustomModelData of item when removed from offhand

I know it's a long shot, since I'm fairly sure it's not possible, but still worth a try.
Basically, I want an item to change its model when held in the offhand, and revert to its default (custom) model in all other cases. 
Changing CustomModelData when in offhand is fairly easy to do, the real problem is that afterwards I have no control on what happens to the item. I can detect when it gets removed, but at that point I already lose track of where the items went (and it can be a plethora of options like mouse held, item frame, ground, container, another inventory slot, ...).
Any insight on this?