What if, instead of getting your buffs from eating meals, you had to have sex with someone. Have sex with a different person, end up with with a different buff. For some reason, it's both known and accepted that this is the way things work. No one thinks twice about it. So naturally the prince would be sent out into the world with a variety of people who could buff him up. And naturally collaborating with, say, Ardyn on missions would include a bit of pre-mission buffing up as only Ardyn can give.

Just roll with it.