Judging

Big Ideas Toolkit

Judge Assignments

Each Pre-proposal and Full Proposal is reviewed by a minimum of six judges, in order to provide an average score that is a reliable and valid indicator of each proposal’s strength. Given the large number of Pre-proposals received (sometimes up to 70 in one track), Pre-proposal judges cannot read every proposal submitted within their assigned track. As a result, each judge is expected to read and score only a subset of the Pre-proposals submitted in their track—usually between six and eight. In contrast, with the exception of a few larger track like Global Health, Full Proposal judges are expected to read all of the proposals submitted in their track so they have a complete picture of the projects as they score each proposal (between five and ten, depending on the track).

It is worth noting here that Pre-proposal judges sometimes remark that it is difficult to rank each proposal without knowing its relative strength compared to all other proposals, since they only read a subset in the first round. Big Ideas thus provides Pre-proposal judges with paragraph-long summaries of all of the proposals within their track so they have a better idea of the types of innovations being proposed across the entire track (see the Pre-proposal Summaries Example – Food Systems in the Tools section).

Judge Training and Judging Timeline

After judges have been recruited for each round of the Contest, a training is held by Big Ideas staff to familiarize judges with the mission and goals of Big Ideas and the proposal judging process. This training highlights differences between Big Ideas and other innovation or business plan competitions, provides detailed explanations on evaluating various elements in the proposals, and walks judges through how to submit their scores via the online Contest platform (see the Judge Training Prezi and Judge Training Recording in the Tools section for more information).

The information provided during this training session is provided in a Judging Handbook that is available to judges on the Big Ideas website and is sent via email to all judges (see the Pre-proposal and Full Proposal Judging Handbooks in the Tools section). Additionally, the judge training is made available via a live webcast and a recorded version is made available to judges who are unable to attend the in-person training. After the training session, judges are given approximately three weeks to submit their scores and provide written feedback.

Judging Criteria

The judging criteria from the Pre-proposal round focuses on the extent to which the teams propose a creative solution to a social problem and the project’s intended social impact, while Full Proposals are judged primarily on the project’s potential social impact and the viability of the project plans (for a detailed description on the criteria used to assess each proposal, see the Judging Criteria example in the Tools section).

Judging Feedback

During both the Pre-proposal and Full Proposal judging rounds, judges are asked to provide both quantitative feedback (i.e., Likert scale rankings of the proposals’ strength in particular areas) and comprehensive qualitative feedback (i.e., feedback written to the team about areas of strength and areas for improvement in their proposals). At the end of each round, all applicants (finalists and non-finalists, winners and non-winners) are given the qualitative feedback so that they may use the judges’ advice to strengthen their proposals and/or project ideas in the future. Students report in surveys and informal conversations with Big Ideas staff that providing judges’ feedback is among the most useful resources provided through the competition. They say the feedback is helpful and informative to their team and that the level of detail in the feedback is rarely offered in other competitions. Furthermore, providing comprehensive written feedback to all teams ensures that both finalist and non-finalists receive a key takeaway from the Pre-proposal round. Numerous non-finalists have used the Pre-proposal feedback as a resource for further developing and refining their project for other funding opportunities and for resubmission to Big Ideas the following year.

Tips

Emphasize high-quality written feedback. During the judge recruitment and the review stage, stress the importance of providing comprehensive qualitative feedback to the applicants. Along with mentors, this feedback is cited by applicants as one of the most valuable resources provided by the Contest.
Follow-up and build in a deadline cushion. Judges are busy professionals and so it is imperative to send constant reminders and follow-up emails throughout the judging period. Even with frequent reminders, there will inevitably be judges who need an extension to complete their reviews, so allow some flexibility with the judging deadlines– typically 3 to 5 days and if possible, a weekend.
Suggest that judges be critical, but constructive, in their feedback. Judges will often be too complimentary in their comments, and other times too harsh. It is important to communicate to judges that their comments should be written as if they were directly addressed to the team. The most effective feedback raises questions and considerations that the team has not addressed in their proposal, without demoralizing the team from pursuing a solution.