Drawing lessons from the EBU’s first virtual exam

In November 2021, the European Board of Urology held its first Virtual Oral Exam, an online alternative to the in-person oral exams held in previous years. With the exams completed, graded and analysed, the Chairman of the EBU’s Examination Committee Prof. Serdar Tekgül (Ankara, TR) can draw some conclusions and explain how the exam will proceed in 2022.

Prof. Tekgül is currently chairman of the Urology Department at Hacettepe University in Ankara, Turkey, where he also started his urological career in the early 1990s. Prof. Tekgül specialised in paediatric urology and served in several EAU positions over the years, including Guidelines panels, and as faculty member at several EAU events. He is currently also the President-Elect of the European Society for Paediatric Urology.

“My role in the EBU started about ten years ago,” said Prof. Tekgül. “First as one of the board members, and then I had a position in the examination committee where I served for about five years. I was elected Chairman of the examination board four years ago.”

“I have a lot of experience with exam committees, having chaired the exam committee when the first Turkish urological exams were started in the early 2000’s. I was also one of the founders the European Society of Paediatric Urology’s board exam before moving to the EBU.”

The EBU offers three different exams every year, and Prof. Tekgül is closely involved in, and ultimately responsible for, drawing them up. There is an in-service assessment, which is an online multiple choice question exam. It is a learning tool which provides validation comments on the questions; the analysis shows participants a comparison to their peers. Then there are two parts of the EBU exam: written and oral. The written exam was typically taken at set times in secured test centres in Europe but is also offered online since last year.

Tekgül: “I mainly help provide questions for the paediatric parts of the exam, but I also bring together the contributions from all of our experienced board members and finalise the exams each year. It’s a big responsibility!”

It is the oral part of the EBU exam that required the biggest adjustment to the pandemic conditions of the past two years. “For obvious reasons we had to postpone the exam twice, so we had a backlog of hundreds of candidates who were waiting to become a Fellow of the EBU (FEBU). We looked around for inspiration and explored the possibility of an online exam. Some universities had done this with success, but always limited to their own students. Doing it internationally like we would have to was completely new territory and extremely challenging. We decided to take some risks.”

The format would not be an interactive conversation between candidate and examiner considering the high number of candidates. Questions were prepared (in ten different languages) and the candidate had set times to respond orally. The candidate’s feedback was recorded by the computer. Case data and visuals like X-ray or CT scans were shown on screen.

“There is a disadvantage when you don’t interact, it’s not really like an exam. This was of course our concern, but creating interactivity with so many online candidates is not really possible. We did our best to avoid any technical issues: we provided a mock exam to prepare people for the format of the exam, and we had candidates check their connectivity beforehand. Despite these precautions there were of course some problems, which are inevitable when dealing with connectivity. In roughly 10% of the cases, we did not have usable video fi les. There are many reasons: technical, logistical, insufficient familiarity with the set-up and so on.”

In the end, this round of exams yielded 440 clear and evaluable candidates. Two online reviewers independently scored the candidates. If there was a 20% discrepancy between the two (as was the case 65 times), a third reviewer also checked. The final scoring of 440 people took about six weeks, a bit longer than the anticipated three weeks. At the end of the day, there was a 84% pass rate.

“Starting out, our concern was that not doing the exam live would not be a good imitation of an oral exam. But in the end we came to the conclusion that this is in fact more objective,” Prof. Tekgül.

“Every examiner has a different attitude or body language when asking questions, there can be (mis-) leading of candidates and there is always subjectivity in these interactions. But in the setup we used, there was no person, just the screen and the camera. We feel in the end that it was more objective. Questions were standardised, asked in a standard way. The pass rate was a little lower than the 95% pass rate of previous years and, we feel, more accurate.”

“I think it was a good outcome. It has in fact led us to decide that we are going to use the same system in the upcoming exam held in Warsaw. The same software and set-up, but with presence of a reviewer in the room. Not to ask the questions but to oversee the process. We will have forty different rooms, simultaneously. The results will still be reviewed online. We want the candidates to be on-site because it will eliminate all connection problems.” 280 people already registered for the June exams, as have around 50 reviewers.

The Examination Board has been collecting feedback from participants and faculty alike. “The people who passed were very happy. The people who had technical issues were less pleased. Some knew it was their own fault, others were less understanding. The majority gave good, constructive feedback. We understand, we empathise, and we will allow a compensation for this year’s exam. This was the first-ever attempt at this. We will have problems to be corrected, and are on the right track to a better exam. I hope these candidates understand we had to take some risks and we tried our hardest to avoid technical issues.”

“There are already a lot of possible solutions to solve the technical problems encountered, like using a dedicated app to record the candidate's responses and then sending the verified files to the EBU, rather than relying on a potentially unstable live connection. Having recorded answers also opens the door to using AI systems to help in the grading.”

As it stands, there are plenty of valued EBU colleagues to help with the reviewing and Prof. Tekgül is grateful for their help. “I would like to thank all of the reviewers and all of my colleagues who were very supportive and helpful in setting up this new exam.”

“One project we are working on, is to make this system applicable globally for the International Certification oral exam, open to non-European candidates.”

The next FEBU Oral Exam will take place in Warsaw on 27 June, 2022. Find out more about the EBU and its exams on www.ebu.com

This article first appeared in the March-May 2022 edition of European Urology Today.