Shelter is definitely pretty important in, for example, Maslow's hierarchy. Obviously there are less and more fundamental rights, and the degree to which something can effectively be realized as a positive right depends on the capabilities and wealth of a society, but surely it’s worth considering what society should work collectively to provide.
I also don’t think it’s at all reasonable to claim that freedom from arbitrary arrest and detention, the right not to be tortured, and other basic human negative rights are at risk - ethics mist be continually improved and adapted as society changes, and the mere fact that abject suffering exists in the world should not mean we can’t decide that there are higher order human needs within the effective purview of society.