You’re correct about this being rich people’s houses - I note it’s an upper class house in the title - but a poor Roman citizen would be more likely to live in an Insula, an apartment made of wood or brick. If in a rural area, small wood cottages were common for poor farmers, though in some particularly rocky areas stone is known to be used. The kind of hovels you’re thinking of would have been something more common in unromanized areas of Spain or Britain.