Town may separate from Bayside affordable housing