Affordability concerns are moving up the income ladder