It comes down to two different effects: Antenna asymmetry and environmental effects.
A crude explanation for the antenna part of it is simple:
Say you have a PCB type antenna that is 4 cm across with a phase center that’s right in the middle.
From square on the signal travels through the air for the entire distance.
From the side the signal travels the last 2 cm through the PCB. FR 4 has a dialectic constant of just over 4, the signal is going to travel at half the speed for those last 2 cm and so add around 2 cm to your antenna delay.
In reality I saw an effect that was larger than this even for isolated antennas. I have no idea why.
Other effects are a lot harder to explain but as soon as you put anything else metallic near the antenna it’s going to do weird things to both the antenna gain pattern and also the antenna delay. I’m not RF expert so I can’t explain the mechanism but when I plotted measured range over time at a high enough rate and then averaged to reduce the noise there were some clear discontinuities in the measurements as the angle changed.