Is this an observation or a question? Because, yes--that's the way it works. I really don't think there are a lot of circumstances where ramping the audio would be of much use. Our neural pathways will readily accept speed shifts in visuals, but sped up or slowed down auditory signals get annoying pretty quickly.
If you're trying to sync audio on both sides of a speed change, I'd suggest checking out match framing. Your audio would be anchored and synced on the incoming side of the speed ramp, and then you'd have to move to the point where speed returns to normal (100%) and match frame and edit in the original audio.
Park your CTI (the blue line) where speed goes back to 100%, and make sure your video track (let's say V1) is targeted. Hit "M" on your keyboard, and the source clip will load into the source monitor with the CTI at the corresponding frame. Toggle over to the clip's audio, and edit into your sequence--the audio will be back in sync. You can then fill in the gap between the 100% speed audio clips with other, displaced audio, or just leave the audio out. I did this quite a bit over the past few months--worked great.