Can we know how our script scored?

[Larry Rutledge]

This discussion has been had many times, over many fests, and is the principal reason why it is now the policy of the festivals to no longer publish any stats. It has only been recently that we've made individual placement available, but if all we hear are complaints than we will be forced to go back to only announcing the winners.

The festivals and its underlying scoring system has been looked at critically, during each fest, by a multitude of people. We have debated and tested many different systems in order to get the fairest possible system in place.

The rating system is just that, a rating system not a ranking system. The votes are not decided based on any other entry. Rather it is an pure average of total by number of votes placed. We did extensive tests and found that attempting to skew results by not voting on a particular entry and/or voting low on all entries except the one preferred entry does not greatly affect the overall average.

It's true that the more votes each entry receives, the more precise the average will be; but we have done extensive tests to prove that this is an extremely fair system for rating the entries within each fest. This includes having used it during the recent TimeFest ... where we saw hundreds of votes per entry and yet the average moved very little throughout the course of the voting period. All that changed was the precision of the average. This is because if an entry is truly strong, then the majority of the people will vote high for it and if it is truly weak, the majority of the people will vote low for that entry. Thus, if each person votes honestly, the results are similar enough to give a true representation of the overall ... and this is proven true even if one entry receives 100 votes and another only received 10 ... we have run this type of scenario as well and found that it didn't affect the outcome, unless the 10 all lied and each gave it a 1 and no one else counteracted with a higher, more accurate score.

On that front, we take an active role in the accuracy of these fests. I personally check every vote, and the other mods also have access to the core data to check it as well. I check the overall numbers (vote total, number of votes, and average) for each film, I then check each individual vote for each individual entry, and then, if necessary, check each vote given by each individual voter. It is very easy to see patterns emerge if someone tries to skew the voting.

Understand, I don't have to give any insight into the under workings of the sytem, but I want to make sure that everyone understands we don't take any of this lightly and are actively looking for the most accurate outcome possible at all times. This includes on going discussions amongst the mods to ensure that the rules and underlying delivery/voting system is the best, simplest, most accurate it can be and if we find anything that can be improved we set about to get it improved as quickly as possible (understanding that we have lives and families outside of this site as well).

The system is not up for debate. If you have a problem/issue with the system, please take it up with me. To post messages deriding the system when you have no understanding of what takes place behind the scenes is counter productive and undermines the very system we strive to make the best we can ... and we do it for all of you.

[RodThompson]

I just got my magical PM from Issac, and I am happy with the results. Not for the purpose of scoring...but the feedback. The best feedback I ever received was from a professional screenwriter, about my hideously-written first script. He said, "I can't get past page ten...I have a f---in' headache. Sorry dude."

He went on to acknowledge that screenwriting isn't an easy road, and in some cases can be harder than being a novelist. You have to take all of the details of the visual word and convert them into a SLUG and like two or three descript lines IF THAT! No one starts off on a Ted Elliot-esque level. Watch Little Monsters compared to Pirates of the Carribean and tell me that ole Ted hasn't improved.

Chapel: Take this score with a grain of salt. It's not the Nichol's fellowship, or even (God forBid the horror) the Zoetrope contest. This is a Jury-based contest where we're the judges for everyone else. Like Project Greenlight back in 01-02. Don't look at the score, but instead look deep into the lines of that feedback. Your peers judged you why the did according to their standards (some pre-pro, some amateur), so remember that and roll with it. Take the feedback and learn form it without letting numbers get in the way. You're a writer, not a Physicist...the only numbr you should worry about is the gross of your first film at the box office...many years from now. Yes...years.

Everyone that I read did a gnarly job, and I too am guilty of only voting for a few, so I apologize. So pat yourselves on the back, and raise the bar for the next fest.

See you Hobros at TwilightFest.

Rodimus 5000
Placed 10th

[David W. Richardson]

...but if all we hear are complaints than we will be forced to go back to only announcing the winners.

Excuse me??? Who has complained?? The only thing close to a complaint came from me, and it wasn't about the process. It also wasn't a complaint, it was an observation. I have said repeatedly now that knowing the rankings is VALUABLE as feedback. I don't see that as being a complaint, do you?

Yes, I was disappointed that Alien Abduction didn't do better. But I think that's only human nature. Knowing how it placed is at least as valuable as the comments and critiques. And the fact I was disappointed is no reflection on anything or anybody.

As far as I can tell, nobody else has even remotely complained about knowing their ranking. So please don't overreact and make things worse for them, okay?

[Larry Rutledge]

Read it again ... I never said anyone complained. As you point out in your quoted section, I merely stated that "if all we hear are complaints...". This is not the first time this discussion has been had, I know where it has the potential of going. I tried to clear up any confusion and give as much information as possible.

[David W. Richardson]

Read it again ... I never said anyone complained. As you point out in your quoted section, I merely stated that "if all we hear are complaints...". This is not the first time this discussion has been had, I know where it has the potential of going. I tried to clear up any confusion and give as much information as possible.

Oh. Sorry. I misunderstood what you were saying.

Then my opinion is that knowing the rankings is a GOOD thing, even if our feelings get a little hurt. I think it's very useful information, and would hate to see it taken away.

[RodThompson]

Honestly...I feel bad for anyone who isn't me...

<mourn>

[Tom Shortridge]

Honestly...I feel bad for anyone who isn't me...

<mourn>

So do I, Rod... so do I.

[David W. Richardson]

Honestly...I feel bad for anyone who isn't me...

<mourn>

Your modesty is refreshing!

[Larry Rutledge]

Then my opinion is that knowing the rankings is a GOOD thing, even if our feelings get a little hurt. I think it's very useful information, and would hate to see it taken away.

I agree ... and that's why I did the "pre-emptive strike". We've had the discussions for as long as we've had the fests ... initially we provided all the info but honestly people complained so much that we finally decided to keep the info completely private. It wasn't until I released this new rating system that we stuck our toe into the waters again by privately providing individual placement.

So far, we're ok ... as long as everyone keeps their cool These fests require a tremendous amount of time and energy on our parts, but we don't mind doing it because we enjoy seeing everyone get excited about their own, and every one else's, projects and we love watching you all grow as writer/filmmakers.

We do everything we can to ensure the fests are as easy and fair as possible.

[RodThompson]

... as long as everyone keeps their cool

Rod crosses the room, hiding his cool in his satchel. Looking about the forum, he thinks of giving it away, but instead lowers his head and walks out. Keeping his cool.