This is basically my view on this matter. We can only take for granted the existence of the objective universe. Everything else must be derived from experiment, including concepts that seem fundamental like time or causality.
One interesting thing about causality is that the fact that the macroscopic world obeys cause-and-effect laws is usually attributed to the second law of thermodynamics. But where does the second law of thermodynamics come from? I've heard it stated (can't say if this is universally held by physicists) that the second law of thermodynamics holds because the initial state of the universe had very low entropy, while the final state of the universe will have very high entropy. That is the second law of thermodynamics is "caused" by both the initial and final states of the universe. So causality and time are something that we don't fully understand yet.