I'm guessing it's easier to start in that area and it's not "nowhere."
Merced is north of Fresno which is a larger city than many states have at all. It has a half a million people and an international airport.
Bakersfield is also a fairly large city. They are only small by California standards. In most states, either of them would be the biggest or one of the biggest cities in the state.
I actually did a fair amount of research on rail stuff when I was a college student hoping to become an urban planner. I wrote an alternate rail plan for Solano County, California as a student.* So I'm not just talking out my ass here.
No did I suggest that they would have created high speed rail just for that area if it weren't intended to then go on to SF and LA.
But when I was homeless in Fresno, I was already reading articles about the project and I think they were already doing some of the work at that point. So I've actually read a reasonable number of articles about this project as well.
I think it makes sense to do the middle section first. Work out some of the bugs in a relatively low cost leg of the project and then try to connect it to the more dense parts of the state after the "easy" section of track is successfully up and running. It's a test drive for the project to prove the concept and make it more feasible to complete the more challenging sections, basically.
You could make an argument that the big mistake was running I5 along the west side of the valley instead of just improving HW99. A tell is 70 years after they built it there is still dick all nothing between Sacramento and LA.
Fresno only gets 11 inches of rain a year and there's a lot of farming around there. Fresno County also has an amazingly rich history of water development, both technologically and in terms of hashing out state laws for water rights.
California uses two different and competing standards with different historical roots. They use both the standard of "first come, first served" and also the idea that you can't just completely screw people over down river.
Most water in California is shipped in. Fresno County has more water security than most parts of the state and its rich history of water development means that in some years, it is actually raising the local water level while most of the US is draining underground aquifers that will never recover from it, resulting in land subsidence.
I considered remaining in Fresno permanently if only because of climate change and the expectation that water is what future wars will be fought over. Fresno has better water security than most of the state, and not due to rain.
It's also like closing in on 3am here and maybe I should shut up and get off the internet. (shrug)
Edit: Some old comment by me about books I have read about water in Cali that anyone into water in Cali might enjoy:
It's possible with rail. You just need to avoid stopping trains on the main line. Switch off on to a siding to serve the station. Switching should be at the fastest safe speed; slow trains do not belong on the main line.
a. You show up, only to discover that the train just left. Since it isn't properly scheduled, you just have to get lucky. Enjoy waiting an hour or two for the next train.
b. You show up, only to discover that there are too many people to fit on the train. Since you don't have a seat reservation, the train fills up and you don't get to board. Enjoy waiting an hour or two for the next train.
Train travel is much more pleasant than air travel and being able to just turn up and go, and carry your own luggage, is a big part of that. This goes beyond fixing what's not broken - why are you trying to make the pleasant thing like the less pleasant thing?
Reality: that won't happen
It costs money to run trains that might be mostly empty, so they are run less often, which makes them less used, which makes them run less often, and so on.
At best, reality looks more like this: https://www.mbta.com/schedules/CR-Lowell/timetable
That's 21 trains per weekday. Weekend days get only 8 trains.
Amtrak is much much worse. At that service level, you could get 1 train per week, seasonal.
Reliable trains are a good thing. Frequent trains are a good thing. Your proposal doesn't help with either of those things and would waste money that could be spend improving them instead.
Also Bakersfield Metro is 850,000 people. Fresno is 970k.