Eventually six of the seven kingdoms of the First Men were conquered by the Andals. Only the Kings of Winter successfully resisted the Andals, holding off their invasion in an area of Westeros known as the Neck.
This caused the continent to be split between north and south. The north was occupied by First Men, both existing northerners and others who didn't want to give up their old gods and succumb to Andal rule. The south's kingdoms adopted the Andal's religion and were ruled under them.