Help with Item Modifiers

I am attempting to modify the custom_model_data of my item with a command.
My item json:
{     "model": {         "type": "minecraft:model",         "model": "custom:item/hat",         "tints": [         {             "type": "minecraft:dye",             "default": -1902854         },         {             "type": "minecraft:custom_model_data",             "index": 0,             "default": 16711718         }         ]     } }
My item modifier (generated with misode):
my_dp/data/custom/item_modifiers/hat_change.json
{     "function": "minecraft:modify_contents",     "component": "minecraft:container",     "modifier": {       "function": "minecraft:set_custom_model_data",       "colors": {         "values": [           5897984         ],         "mode": "replace_section",         "offset": 0       },       "conditions": []     },     "conditions": []   }
The command I'm using:
/item modify entity @s weapon.mainhand custom:hat_change
The result is in the attached image.
The item itself is working as expected, but I can't get the item modifier to work. Not sure what I'm doing wrong. I'm pretty sure everything is in the right place.
edit: I'm trying to change the custom_model_data color btw.