Ah! The thing is that the adults did normally live in Cittàgazze and Spectres were not too much of a problem - there were a quite a few Spectres but they moved around the world so life remained the same for most people who continued to raise families. However, after Lord Asriel made a window into Cittàgazze, a huge amount of Spectres escaped and swamped all over Cittàgazze so the adults fled to the hills alongside the children.

This is why there is still recently made food on the tables when Lyra enters Cittàgazze, the adults only fled the city a few days before. Eventually, because the Spectres don't harm children, they came down from the hills but there were still adults alive and un-Spectred up there. I expect that most adults don't live very long though, but certainly to an age where they can breed.