How the Judging Works
The judging procedures of most industry awards always seem to be veiled in mystery. Well at the Australia Web Awards we aim to be as open as we possibly can.
Validation
The first step in the judging process is the validation for HTML and CSS for all the sites submitted this is done by the Judging Sub-Committee. It is a semi-automated process.
The entries that pass the validation process with NO ERROR and 2 WARNINGS or less are then assigned to Judges.
Assigning the Judges
The judges are selected by opposing state and their ability to judge the various categories. Any localised bias is removed by not having judges of a particular state judge for that regional block. This is made easier when you have a number of international judges, as we do this year.
The judges are asked to look over the list of sites they have been assigned and declare if they’re in a conflict of interest. You may think this doesn’t happen, but it does. Let’s be honest here we are a small industry, the people at the top of their game tend to work together all the time. Hence conflicts of interest are bound to occur.
When there is a conflict of interest, we simply reassign the judge to another category. We have found over the years that the judges tend to work professionally in silos of site types, so avoiding the conflict is usually very easy.
After the judges are all settled into their categories they get to know who the other judges are, but that is it. Any official correspondence between them occurs with the knowledge of the judging sub-committee.
Judges are reminded that it's not public knowledge which category they are judging until the finalists are announced. Again this stops any incentives or the like.
The Judging Process
When a judge is scoring a category of web sites, they have no idea what scores the other judges are giving. This allows total independence, and no collaborative bias.
Judges this year will be encouraged to write a short comment about the site that may assist you if you’re not successful. Please note no correspondence will be entered into over these comments, this is purely a service to better the already high standard of the Australian Web Industry.
At the end of the judging process only the judging sub-committee knows the outcome and the regional finalists and the national finalists.
From the national finalists the overall winner is selected. If there is no clear overall winner then the judges are brought back to discuss the top finalists and the clear winner of the Most Outstanding site.
As you can see the process is very regimented and strict. With no collusion, and minimal bias. At no stage does anyone sit around a table drinking a few beers trying to decide which one of their mates will win a awards.
The Australia Web Awards are won through hard work, blood, sweat and tears, oh and talent too!
Reader Discussion
I noticed last year’s winner in the education category failed some aspects of accessibility. Should’nt that be a basic criteria for judging this category, considering all government .edu sites legally have to comply with at least AA if not AAA regulations?
Hi Phil
Firstly let me explain we don’t validate every page for HTML and CSS, we take a random selection.
We have 100’s of sites to validate. So it is possible that award winners will have invalid code. Aslo note they usually have valid code for the pages we test for the time we tested. Months, years later that will change as you expect. Same will apply with accessibility.
For Accessibility there is no good automated testing tool for WCAG 1 or 2. This means that every page will need to be manually accessed via the criteria for accessibility.
You have to understand that is done on a volunteer basis in our spare time. It would take weeks or even months to validate every site against WCAG 1 or 2 to any level of compliance.
What does occur is the accessibility aspect is judged by specialist in their field. Often a good number of sites will get a very low accessibility rating. And yes in some cases a site with poor accessibility will win. This reflects that the industry as a whole is producing site with poor accessibility. As all the sites in that category will have a low rating.
Maybe if they fail in the accessibility aspect we shouldn’t hand out an award at all, what do you think?
If we had the time and money I would love to manually vet all the site for accessibility compliance. Especially as its a passion of mine. But reality is I can’t. Unless I had more volunteers specialising in accessibility willing donate 50-100 hours.
Maybe in the future we will be able to consider this, thanks for you comment.