Big Ideas believes that selecting appropriate judges and crafting appropriate judging criteria are critical to the Contest’s success: Without both of these components, Big Ideas would be unable to identify those with the most innovative and impactful ideas from the large pool of applications received each year.
Each year, Big Ideas selects a pool of judges (see the Mentor & Judge Recruitment section in the Mentorship chapter for detailed information on recruitment strategies) who have content expertise within a particular track. Judges in the Global Health track, for example, typically are faculty or professionals with expertise who have spent a significant portion of their careers working to address health-related issues domestically and abroad. Judges in the Art & Social Change track, similarly, may be faculty or professionals with expertise in the use of dramatic, performing, or visual arts to effect social change. Put differently, judges are selected on the basis of their credentials and their fit with the Contest tracks.
In the 2018-2019 contest, Big Ideas recruited 325 judges for the Pre-proposal round 337 Pre-proposals) and 110 judges for the Final round (54 Full Proposals). It is generally recommended that 1 judge is recruited for every anticipated Pre-proposal submission, and 2 judges recruited for each Full Proposal submission to ensure that each proposal is read by a minimum of 6 different judges in each round.
The incentives for judges to participate vary, but many choose to participate for one of three reasons: 1) the opportunity to give back while also getting a first-hand look at some of the most innovative student ideas as they are being developed; 2) the opportunity to build their own professional networks by attending Big Ideas events and mixers where they can meet other judges, professional mentors, faculty and students; or 3) a professional courtesy to the track sponsor. It is worth noting that, generally speaking, judges who are incentivized by “giving back” and “professional development/networking” tend to be more committed and reliable judges. Due to the similarities in outreach strategies between Judge and Mentor recruitment, the process was integrated in 2015 into one large outreach effort in the beginning of the year (see the section entitled Mentor & Judge Outreach Strategies in the Mentorship chapter for a full overview on the types of approaches used).
Each Pre-proposal and Full Proposal is reviewed by a minimum of six judges, in order to provide an average score that is a reliable and valid indicator of each proposal’s strength. Given the large number of Pre-proposals received (sometimes up to 70 in one track), Pre-proposal judges cannot read every proposal submitted within their assigned track. As a result, each judge is expected to read and score only a subset of the Pre-proposals submitted in their track—usually between six and eight. In contrast, with the exception of a few larger track like Global Health, Full Proposal judges are expected to read all of the proposals submitted in their track so they have a complete picture of the projects as they score each proposal (between five and ten, depending on the track).
It is worth noting here that Pre-proposal judges sometimes remark that it is difficult to rank each proposal without knowing its relative strength compared to all other proposals, since they only read a subset in the first round. Big Ideas thus provides Pre-proposal judges with paragraph-long summaries of all of the proposals within their track so they have a better idea of the types of innovations being proposed across the entire track (see the Pre-proposal Summaries Example – Food Systems in the Tools section).
After judges have been recruited for each round of the Contest, a training is held by Big Ideas staff to familiarize judges with the mission and goals of Big Ideas and the proposal judging process. This training highlights differences between Big Ideas and other innovation or business plan competitions, provides detailed explanations on evaluating various elements in the proposals, and walks judges through how to submit their scores via the online Contest platform (see the Judge Training Prezi and Judge Training Recording in the Tools section for more information).
The information provided during this training session is provided in a Judging Handbook that is available to judges on the Big Ideas website and is sent via email to all judges (see the Pre-proposal and Full Proposal Judging Handbooks in the Tools section). Additionally, the judge training is made available via a live webcast and a recorded version is made available to judges who are unable to attend the in-person training. After the training session, judges are given approximately three weeks to submit their scores and provide written feedback.
The judging criteria from the Pre-proposal round focuses on the extent to which the teams propose a creative solution to a social problem and the project’s intended social impact, while Full Proposals are judged primarily on the project’s potential social impact and the viability of the project plans (for a detailed description on the criteria used to assess each proposal, see the Judging Criteria example in the Tools section).
During both the Pre-proposal and Full Proposal judging rounds, judges are asked to provide both quantitative feedback (i.e., Likert scale rankings of the proposals’ strength in particular areas) and comprehensive qualitative feedback (i.e., feedback written to the team about areas of strength and areas for improvement in their proposals). At the end of each round, all applicants (finalists and non-finalists, winners and non-winners) are given the qualitative feedback so that they may use the judges’ advice to strengthen their proposals and/or project ideas in the future. Students report in surveys and informal conversations with Big Ideas staff that providing judges’ feedback is among the most useful resources provided through the competition. They say the feedback is helpful and informative to their team and that the level of detail in the feedback is rarely offered in other competitions. Furthermore, providing comprehensive written feedback to all teams ensures that both finalist and non-finalists receive a key takeaway from the Pre-proposal round. Numerous non-finalists have used the Pre-proposal feedback as a resource for further developing and refining their project for other funding opportunities and for resubmission to Big Ideas the following year.
Blum Center for Developing Economies
The University of California, Berkeley
bigideas@berkeley.edu
© 2023 Big Ideas All Rights Reserved
Design by Spencer Hamilton
