Hm, sharpness loss, colour shift, loss of contrast and vignetting sounds like a degradation of image quality to me, no matter how 'little' or 'negligible' it might be.
The sharpness is practically non existent, even with pixel peeping. Regular viewing, I'd bet money that not a soul would notice. The rest are easily fixable in post. Is it a degradation? Yes. I wouldn't call it "trashing" the image, though. Certainly worth doing if you seek the low apertures. Of course, my standards aren't everyone's. It's the type of thing you'd want to test yourself to see if it meets your own personal standards.