he didnt let his family come because only the airbenders can navigate the air temples
his family would not be able to survive, and even if they could, they would be split by gender, so the farmer and the children would be separated from the mother
What's so hard about inviting the entire family at least. I'm sure that would convince some people. It seems like they are trying to change what it means to be an airbender in this season. There's no need to be a monk.