When you use your car’s built-in speakerphone feature, the speakers in the car door reverberate, sending the sounds outside of the car. Maybe don’t do this while your car is parked. Everyone in the neighborhood or store parking lot can hear your very intimate or business-related conversation.
The OP’s scenario is doing this while parked. While you’re driving it wouldn’t matter so much because there are only so many words somebody can hear when you pass them at 24-45mph