Check out this paper, which also conveniently focuses on approximating the function of the head/torso simulation using spheres.
The takeaway is that our hearing is very finely tuned. Beyond just what we actively hear, the brain is taking in all kinds of information that it uses to determine distance, height, location, etc. of the world around us.
The importance of the head related transfer function cannot be overstated. A well-made binaural recording is like seeing color for the first time, it truly transports you to the location. Part of that - an admittedly smaller part - has to do with the way sound interacts with our torso in addition to our head and ears. There are reflections from the torso that our brain is using to help orient us and what we hear, even if we donāt realize it.