Ardyn was never forced out of Lucis. He remained the Accursed King. The other side of his family line continued, keeping their relation to the king and their connection to the crystal's magic a carefully guarded secret. Everything is going fine(or as fine as it can be under the rule of a demon king) until 5 year old Noctis is chosen by the crystal to defeat the Accursed.