The hours depend on what area of the world you're in, whether it's daylight savings time, etc. Forget focusing on exact hours and instead focus on the sun position. When it's most directly overhead, that's when the lighting looks the worst. That's the time you'd prefer to avoid.
As for the polarizer, no -- not at all. Instead stick out your index finger and thumb (like you're making a "gun") and point your thumb at the sun. Anywhere you can now point your index finger, that's a place where you could shoot effectively with the polarizer -- well, with the polarizer at its most effective, that is. When you're at 90 degrees to the sun, that's when it works best.
