I've never been in a balloon, but I've been paragliding, and the one thing I really noticed, and then felt stupid for not having thought about before, was that the wind wasn't noisy. In fact there was no wind, because you were travelling with it, not against it.