In regards to your 4th point, our cropping systems in Australia are a bit different to the states. We already grow crops in a rotation, Wheat, Lupins (Nitrogen fixing) and Canola. Depending on where your farming your might also be growing cover/summer crops.
For a whole-systems re-design, I think that beyond just tailoring genetics, nutrition, and herbicides to site-specific conditions, we need to be thinking about all of the interspecific interactions, spatial designs, and other natural processes that we can be harnessing. For example, we're currently constrained to operate farm machinery in a grid, or perhaps on contours. One way to improve that would be to start adding complexity to the grid. On a simple level, you could have one row of grains and one row of pulses as a means of breaking up pest migration. Slightly more complex would be to use intercropping or to perhaps co-locate crops that provide pollination services, moisture storage, or other benefits to each other. More complex would be to start using trap crops more frequently or taking advantage of natural seed predators with interspersed natural vegetation. Even more would be to incorporate trees (agroforestry) not only on the edges of the field but also within the field. Finally, once machinery is sufficiently sophisticated and the logistical hurdles are surmounted, you could break away from the grid pattern and start experimenting with other configurations such as concencentric circles or other shapes - the possibilities are endless.
We do grow crops in rotation in the states, although more in some locations than in others (about 15% of farmers where I'm from in Montana use pulses rotated with grains). Cover crops are a great option, though of course in dryland semi-arid regions early termination to prevent moisture losses is always an issue.
We've been using variable rate to tailor nutrient applications for a few years now[0], and have started contracting with a WeedIt to do spot spraying(major reduction in chemical application).
We've definitely discussed "reforming" some of our paddock structures to improve yield, or mitigate weed potential (sowing east-west vs north south). Although if you were going to commit to major production changes like that, you'd probably want to bundle them together. Changing sowing direction would mean restarting CTF, so you'd be best of doing another operation such as deep ripping at the same time.
We've experimented with wetting agents, as we have quite bad non-wetting soil, and the results have been quite good. It's amazing to watch the before and after[1]
Do you think we're going to need a shift in machinery before we can break out of conventional cropping patterns/methods? I'm still somewhat skeptical that a swarm farm type setup would work in our broad-acre situation, but I think a downsizing of machinery (1 * 60m bar -> 3 * 20m bar), could work if they also move to fully autonomous & electric drive. Although, automation for field operations is still quite "simple" compared to the logistics of managing seed/fert refils.
[0] A few example maps - https://hectare.ag/nooka [1] The effects being discussed here are different, but the video shows the same idea https://twitter.com/ipaddockapps/status/708066704954249216
Do you use / have you used a Yeomans' plough, and/or investigated keyline? These are well suited to broadacre remediation and improvement (compared to, say, the details around inner zone planning of permaculture). I've seen evidence of these techniques around the Hunter, near Yeomans' original property, and the methods tend to be highly regarded.
I actually work as an engineer/designer for a small business based in Queensland who build ploughs, including non-inversion deep rippers (somewhat in competition with Yeoman). We have in the last ~8 months sold a heap of rippers into WA in partnership with a local manufacturer, targeting clay pan breakup, pH remediation, etc. with a lot of success. It's interesting the way deep tillage and other mechanical soil management ideas are experiencing something of a renaissance out west at the moment.