With filters it does matter. I'm not positive but I think you want to put the darkest filters furthest away from the lens to reduce reflections. With gels, I've heard it doesn't matter and I've heard that it does, but I tend to believe it doesn't.
People tend to put color before diffusion on a light. If you want to swap out a diffusion frame, or move the frame, you don't have to worry about your color following along. Easiest if possible to just clip it to the doors.
It also allows you to get as close as possible with the diffusion frame to the edge of the frame.
So, as far as if it gives a different look, no. But it makes it a lot easier to work to have it first.
I think this is good question and I am kind of stumped. Now for gels, I think RyanT has it right; you do what makes the most sense with what you have. However, for lens filters... I have some questions of my own on that. I can see situations where I would want to stack a graduated ND, a polarizer, a tad bit of diffusion and a little bit of color compensation in front of a lens. In that kind of scenario it just makes sense to me to stack them polarizer first, grad ND second, color compensation third and diffusion last.