Actors get used to coming at all times of day. It is best to do auditions starting in the afternoon to early evening. That way you can get actors with day jobs and night jobs to come in. Weekdays are best but Saturdays can work too. We just get used to being called in to casting offices when they are open. Don't forget you can also offer people the option to video tape their audition and send it to you on the web. If you like their video submission enough, do a call back and have them meet you in person so you can tell if you can work with them and if their performance is consistent!
I can only think of one, possibly 2 auditions over 38 years that was on a Sunday. Maybe they happen more often that I think but I have not seem them as the norm. Sorry the reply is so late as I got busy. I hope it was not too late.