And no poor non-Hispanic people driving through on El Camino, getting from A to B? I don't know the local geography, but I would assume that a main thoroughfare would get _some_ non-residents.
As far as I can tell, there are ~no poor people living in Atherton, Palo Alto, or most of Menlo Park, unless you count startup founders, retirees who live on low retirement incomes in $5mm present-value homes, and the like. There are poor people living in RWC (sort of), who are almost exclusively Hispanic (at least from the neighborhoods I saw; I was looking at houses in North Fair Oaks and RWC at one point.) Even the poor people in RWC didn't seem that poor, more like working class or middle class (which is poor by comparison to Atherton.)
There are poor people in EPA and Belle Haven (Menlo Park east of 101). They would be very unlikely to take ECR through Atherton, though, since 101 is closer. (they used to be almost exclusively black, now it's partially black and partially hispanic and partially pacific islander). And even there, real estate is now becoming absurdly expensive again.
There are weird poor (white, and other) people living in the inaccessible parts of Portola, but they seemed really old, or rarely left their homes, and they probably weren't actually poor, just "people who live in the middle of the woods and are kind of weird."