He's dedicated pretty much his whole life to Noct. When would he have found the time? He would've probably rationalized it as never being able to give another person the attention they deserve because of his duty/loyalty to his prince.

+ Bonus points for emphasis on body modesty too (inspired by - SPOILERS - how uncomfortable he was about his revealing attire for the Assassin Festival)