Indeed, from having a natural unit for speed there are some steps until determining units for all primitive quantities, but nonetheless the units of all primitive quantities, including time and distance, can be determined from the available natural units.
In physics, there are primitive quantities and derived quantities. The derived quantities are defined using the primitive quantities, e.g. velocity is defined as the ratio between distance and time. The primitive quantities are not defined using other quantities, but they are assumed to exist and to satisfy certain axioms, e.g. of being a scalar quantity or a vector quantity.
To determine the numeric value of a quantity, a unit of measurement for it is necessary. The units of measurement are either fundamental units a.k.a. base units, or derived units, which are defined using the base units.
Ideally, the base units should be the units of the primitive quantities, because this leads to the simplest expressions for dimensional formulae and for derived unit definitions. Unfortunately, reality is not so simple, because for 3 of the primitive quantities, i.e. for time, distance and action, it is not possible to make a physical device that would provide directly units for them in such a way that those units would have good enough stability and reproducibility.
On the other hand, for other 3 quantities that are ratios involving these 3 primitive quantities, it is possible to choose units with excellent stability and reproducibility. Thus their units can be chosen as base units and the units of those 3 primitive quantities must be derived from them.
The determination of the units for time, distance and action starts with another base unit, that of plane angle. The plane angle has the advantage that it is possible to choose a unit for it by convention. Most physics books choose the radian, but the radian is an extremely poor choice, both for numeric computations and for practical measurements. This is why despite claiming that the radian is the unit chosen for plane angle, most physics books are inconsistent by using other units implicitly in many places, where using radians would be too inconvenient.
In practical measurements, the unit of plane angle that is always used is the cycle, not the radian. So starting with the cycle as the unit of plane angle, and with 2 natural units for velocity (the speed of light) and for angular momentum (Planck's constant), one more base unit is needed, which is the unit of energy and work. That unit is chosen arbitrarily as the work required for the transition of some atom between 2 states corresponding to different electronic energy levels. Currently there are several candidates for the future redefinition of SI, including certain energy levels of lutetium, ytterbium, strontium, aluminum, calcium or mercury. Because it is not clear which is the best it is possible that instead of choosing only one of them, the geometric mean of the differences in energy levels will be used.
Having established the choices of the base units for plane angle, angular momentum, energy and velocity, the units of the primitive quantities action, time and distance can be determined successively from them. Angular momentum is action per angle, so we get the unit of action by multiplying a cycle with Planck's constant. Energy is action per time, so the unit of time is determined from the unit of action and the unit of energy. Velocity is distance per time, so the unit of distance is determined from the unit of velocity and that of time.
The practical determination of the units is more complicated than the chain of multiplications described above, because some of those multiplications cannot be realized in a convenient way.
In practice, the unit of energy is prepared by pumping some ions or neutral atoms of the desired type to the upper energy level used in the definition of the unit of energy, in which case that energy is transferred to light by stimulated emission, or the ions or neutral ions are prepared in the lower energy level, and the unit of energy is detected by absorption of light having the right energy.
In any variant, eventually we get light whose energy corresponds to the unit of energy. In coherent monochromatic light, all the 6 ratios possible between the 4 quantities time, distance, wave angle and action are constant, so if one of them, i.e. the energy, is known, all the other 5 are known, including the wavenumber and the frequency, so this provides units for wavenumber and frequency. By counting cycles, the unit of distance can be derived from the unit of wavenumber and the unit of time can be derived from the unit of frequency.
However, at frequencies in the range of infrared light to ultraviolet light, counting cycles can be done only in the spatial domain, not in the temporal domain, so a laser with known frequency can be used directly only for spatial measurements, e.g. with an interferometer. For time measurements the frequency of light must be divided down to the microwave range, where cycles can be counted with a digital counter, by using an optical frequency comb (which is a certain kind of laser oscillator, which produces pulses of light in such a way that there is a fixed ratio between the frequency of the pulses and the frequency of the light).
The exact nature of action is one of the current mysteries of physics. Action is the source of inertia and gravity. Action behaves like it were some kind of rotation angle of an intrinsic rotation movement that affects all matter.
Action for a particle is the integral of the Lagrangian over the worldline of that particle through space-time, so the Lagrangian is the projection of the 4-gradient of the action on the worldline. Thus action, like electric charge, remains a scalar in the theory of relativity (where some quantities that were scalars in classical dynamics, e.g. energy, are no longer scalars). The fact that both electric charge and action are true scalars, even in the relativistic theory, is an important reason why they are the best choices for the primitive quantities of dynamics.
It is possible that in the future action will have a fate like that of temperature. When the nature of heat was still not understood, temperature was considered a primitive quantity. Later, in the 19th century, when it became understood that all thermal phenomena are caused by a permanent movement of all particles of matter, it was eventually understood that temperature is not a primitive quantity, but just the average kinetic energy per degree of freedom.
Like heat is explained now through a certain kind of universal and permanent movement, it is possible that inertia and gravity will be explained some time in the future as consequences of some kind of universal and permanent rotation movements of all particles of matter, in which case for a particle action would be the rotation angle and kinetic energy its angular velocity.
Nonetheless, for now action must still be handled as a primitive quantity, with no explanation about its nature, but only with the laws of dynamics where it is involved.