Over a year and a half ago, on April 1, 2012, the BJCP implemented a new exam structure. This new structure was created to reduce the backlog of written exams, to address the program’s ability to scale, and to embrace newer technologies available for certification.

Each month the BJCP sets a limit on the number of exam sites that may offer the exam. When the new exam structure was implemented, the Exam Directorate made the decision not to immediately raise the limit of eight exam sites per month until the Legacy exam backlog could be cleared, and until the average turnaround time for the new Tasting and Written exams was determined.

At the BJCP’s annual meeting held during the 2013 National Homebrewers’ Conference in Philadelphia, President Gordon Strong announced that the backlog had been cleared, and that the Exam Directorate had elected to change the monthly exam limit from eight to 10 sites beginning in October 2013. For the four months following, the directorate contacted each exam site administrator asking them whether they would like to move their exams forward on the calendar given the availability of new dates. With this shift, as well as the appointment of new Exam Directors and Associate Exam Directors to the Exam Directorate, the number of exam sites being offered has increased through 2015. Please keep in mind exam site scheduling more than two years in advance is not allowed.

In addition to the higher monthly exam limit, other changes have been made to the Exam program. Jeff Sanders, Assistant Communication Director, spoke with Lead Exam Director Steve Piatz, Exam Director Scott Bickham, and President Gordon Strong, to get their feedback on the changes.

The exam restructuring brought an online exam to the BJCP (the “BJCP Entrance Exam”), and in addition, decoupled a Tasting test from a Written test (the format of the Legacy exam). Why was it decided that there would be an Online exam instead of simply separating the Tasting portion of the Legacy Exam from the Written portion?

We needed to cut the workload on the graders, and also make the exam more approachable to more judges. With only 20% or so of the members ever reaching National or higher ranks, it was pretty clear that the written exam is only required in some situations. If we just separated the exams, we wouldn't cut the grading. We were also able to shorten the written exam since it's no longer needed to screen candidates at all levels; now it can be focused only on the higher levels. So we cut a burdensome requirement for new examinees, while reducing the grader workload significantly. I call that a win-win.

Overall, how would you say that the Entrance Exam has been received?

From our perspective it has gone well, probably better than expected. For the online exam, the CourseWebs team has helped us make some modifications such as the type of participant data that is entered, question updates and adding exams in other languages that have helped the exam meet the needs of our members and administrative team. The registration data can now be ported directly into the BJCP database, which was one of the keys in expanding the number of exam seats. There are occasional connectivity glitches that require manual correction, and some participants have trouble understanding why a different User ID is needed for an exam retake, but otherwise, it’s low maintenance. A different User ID is used to preserve the exam feedback so that they know which areas need the most attention when they prepare to retake the exam.

How and why was the online provider chosen?

The BJCP Exam Directors and Treasurer looked at a number of options, and CourseWebs had the best combination of payment and billing options, flexibility for modifying and adding exam material, reasonable rates and great customer service.

What are the typical passing rates for the Entrance Exam, and how many times does it typically take for someone to pass?

The passing rate is about 68%, including multiple retakes as well as those who take the exam without any judging experience or focused training just to see what the exam entails. This passing rate is comparable to historical data for the written portion of the Legacy BJCP exam.

With the Entrance Exam administered online, no one is proctoring to look over the shoulder and double-check for cheating. There’s also no way to verify that the person doesn’t have 10 other established judges helping him or her to take the new online exam. Do you worry that judges may be taking the Entrance Exam for other judges?

Not much. Those who pass the Entrance Exam, either alone or with help from references or other judges, are not BJCP judges until they pass the BJCP Beer Judging Exam. They are Provisional judges and have no BJCP rank. The passing rate of the Beer Judging exam is quite high (over 95%), which indicates that most new judges are taking the Entrance Exam seriously and learning the material that enables them to pass that exam and do well on the Judging exam.

With the Exam Program changes, there have been clarification to ranks for Program members who have yet to meet the Recognized rank. This has led to some confusion. For those who are confused, can you explain the difference between Novice, Apprentice, Provisional, and Rank Pending?

Novice is not used in BJCP literature. It was used in the past as either a synonym for Apprentice or a non-BJCP judge, both of which were misleading. We removed it from all our documentation quite a while ago. On score sheets, we now use "Non-BJCP" to indicate someone unaffiliated with the BJCP.

Apprentice is a BJCP rank given to someone who has taken, but not passed, a BJCP exam. The person is not a full BJCP member, and must pass the exam within 2 years to remain active.

Provisional judge is a status given to someone who has passed the online qualifier but not yet taken a BJCP exam. It is not a BJCP rank, and the person is not a BJCP member. It appears on BJCP score sheets.

Rank Pending is a status on BJCP score sheets to indicate that someone has taken a BJCP exam but who has not yet received their results. In the database, we use "N/A" for this term; it is a placeholder for the real rank. The person is basically equivalent to an Apprentice judge, but without the negative connotation of having failed the exam.

One criticism that has been expressed is that the online Entrance Exam is that it has shifted the wait time from those waiting to take the legacy exam to get a BJCP number to those having to now wait for a free space in a tasting exam. There have been changes made to raise the higher limit on monthly Tasting exams and the number of people who can take the exam per session. Do you feel this will help to address that criticism?

The exam slots were filling up well in advance of the launch of the new exam, so this criticism is somewhat misguided. Prior to April 1, 2012, exam slots were booked over six months into the future, and seats for those exams were filled immediately when the exam was scheduled. The exam grading backlog at that time was over six months in the other direction, so it was clear that something needed to change to address the demand and supply problem. We were a little surprised by the surge in demand for the judging exam, but it’s apparent from data from recent exams is that much of the demand is driven by judges who are signing up to take the judging exam as many as three times per year.

We strongly discourage people from retaking the same exam when the results of a previous exam are still pending. We've discussed having a policy that says a retake while a pending exam hasn't completed grading will just be ignored (just thrown away and not graded). We may well need to officially say that.

At the end of the day, the number of exam we can process (grade) is driven by the number of senior judges that volunteer to grade exams. When you look at the Exam Grading Process on the BJCP web site (/docs/ExamGradingProcess.pdf) remember that the Gantt chart shows a single exam flow, but that we start 10 additional exams every month. It takes a lot of effort by a lot of people to handle that many exams on an ongoing basis.

All National and higher judges are eligible to grade exams and are encouraged to help by volunteering to grade exams - send an email to exam_director@bjcp.org to volunteer. Graders earn non-judging experience points. Exam grading is a mandatory part of the service required to become a Grand Master Judge and every Grand Master level requires additional grading.

The online Entrance Exam now provides near instantaneous results. The tasting exam turnaround time has been shortened from 6 months to an average of 3 months, which some contend is still somewhat lengthy. Can you explain or provide some details as to what each exam goes through during these three months, and how the grading process works?

The exam is graded using the same process as we have used in the past, as spelled out clearly in the following documents on the BJCP web site at /docs/ExamGradingProcess.pdf and /docs/Scoring_Guide.pdf.

Essentially, the volunteer exam administrator coordinates the exam, makes copies of the exams for the participants, and sends the exams and other documents to the Exam Director. Two volunteer senior judges grading the exam, writing the Report to Participants (RTPs), and making preliminary score recommendations to and Associate Director. The volunteer Associate Director reviews the scores and RTPs and makes scoring recommendations to the volunteer Exam Director, who then does a final review, completes the RTPs and sends the final results to the volunteer Assistant Director. There are multiple steps in this process that easily add to the twelve month objective for completion. It is entirely done by volunteers who have lives outside of the BJCP that typically include a paying job, families, friends and occasional brew days.

In addition to raising the number of exam sites per month, and the increase in the number of Provisional judges who can take the exam at each site, there have also been changes to the number of Exam Directors and Associate Exam Directors. What are the qualifications for these roles and why were the new EDs and AEDs chosen?

The qualifications have not changed and are spelled out in the officer section of the BJCP website. By clicking on the officer title, the job description can be read (/jobdescriptionexamdir.php). Adding more EDs and AEDs was done because exam slots and seats were completely booked two years in advance.

The changes also include the ability to request large tasting exam sites, up to 48 people. Are more proctors required for these larger numbers? Do you have any fears about the quality of the beer sample being effected having to be poured for 48 people rather than 12 people?

We only have one data point, but it does look like the exam administration and grading process could be better streamlined to accommodate the large exam size. The BJCP Exam Directors are evaluating this option.

The same number of proctors are required - more would not improve the exam process and would add more work for the grading team. There is a need for several people to function as stewards to help get the beers served efficiently to a large group. The quantity of beer needed for a large exam set can be a challenge for the local exam administrator.

It sounds as if more and more BJCP members are assuming more responsibility as program leaders, including new EDs, AEDs, and graders? What is the training that these individuals receive in assuming these roles?

Historically, it's been a mix of on-the-job training and one-on-one mentoring, supplemented with a variety of best practice documents and an online forum for graders. We're looking at implementing a more rigorous and formal grader training program in 2014. Stay tuned.

Overall, what do you feel has worked in the revisions to the exam program?

To our knowledge, nothing has not worked, or there are relatively minor issues that are worth accepting for the many benefits of the new exam system.

What are some of those minor issues -- was there anything that may not have worked as well as expected and could be changed?

The quality of the proctor's score sheets is always a concern. For a while we were using a check-box style of score sheet for the proctors on the hope that would get us more complete information from the proctors. Based on feedback from graders and proctors we reverted to the more traditional sheet. We now strongly suggest that at least one proctor be a Master-level or higher judge to help address the score sheet quality concern. Without preapproval by the Exam Director, all proctors need to be National-level or higher. We maintain a list of pre-approved proctors on our web site at /apps/reports/proctors.php.

Do you feel the news of the higher limit on the number of monthly exams was well-received?

This change has not been completely implemented, so please check back with us in a year.

Is there a timeline to look at the exam program again for future changes?

The exam program has been continuously evolving since 1995. There will certainly be changes made as we move forward, with minor changes being implemented by the exam directors and major changes done with the approval of the BJCP board.

Obviously, a new set of Style Guidelines will impact the exams - remember there are three beer related exams and one mead exam and they all tie to the guidelines. The Beer Written Proficiency exam questions tend to require minor adjustment with the style guidelines - the graders need to be current on the nuances of the style guidelines. Many of the questions in the online exam question pool will need review when the guidelines change. The Beer Judging Exam doesn't require change as the guideline change but the graders will need to stay current with guideline changes. The mead exam will require adjustment as the mead style descriptions are adjusted. The introduction of a BJCP Cider Exam (set to debut in 2014) will bring with it new challenges the Exam Directorate is poised to address.

###