Why raw JSON text selector don't change their starting position using execute at or positioned

So this is the command that I'm using for a project of mine:
execute as @e[type=villager] at @s on passengers if entity @s[type=text_display,tag=info_display]  run data modify entity @s text set value '[{"selector":"@e[type=villager,sort=nearest,limit=1]"},{"text":"|","bold":true},{"score":{"name":"@e[type=villager,sort=nearest,limit=1]","objective":"score"}}]'
I noticed that even by putting "at @s" indicating to run at the villager position the selectors inside the raw JSON text won't be affected, meaning that they still check from the text_display entity position. This is a problem when there are villagers closer to the text_display than the actual "vehicle" villager. Is there any way I can workaround this "bug" (idk if it can be called that).