Maybe he makes a bad first impression. Maybe they're not as comfortable with Noct spending time around "common folk" as they like to think they are. Maybe something suspicious showed up in the background checks they ran on him. Maybe they think Prompto is 'friends' with Noctis for ulterior motives (maybe he's a super-poor orphan living on his own). idk.