Why rebuild when you can roam?

I think the principle here is that while the 'undesirables' Thor woke up (which damn well should not have included Amora *grumble grumble they were lovers grumble*) went to explore the world and see what it offered, the Aesir for the most part remained in Asgard.

Think of birds being shoved from the nest, on a much larger scale.