Make exams easier to make them harder
By Sam Graham
You’re sitting in a bar. (No, this isn’t a joke.) Your friend tells you his great aunt just died. You offer sympathy, and say that dying must be a frightening experience. Your friend says that it’s not if you think about it.
You go home. You tell your husband or wife about a student who is given no freedom by their parents, and how it seems odd to you. More than that, the student doesn’t seem to mind. They’re missing out!
The next day at work, you see your colleague reading their Kindle in the break room. You’ve been thinking about buying one so ask him how it’s going. You’re convinced that if you do, though, you’ll still borrow the paper versions from the library.
Congratulations! You’ve just inadvertently completed the first section of one the toughest exams in the world, the 2012 entrance exam to All Souls College (link) in Oxford University. The questions you answered:
Is it rational to fear your own death?
Can we be forced to be free?
Are reports of the death of the book greatly exaggerated?
This blog piece actually started as a piece about the most ridiculous exams in the world, with the All Souls exam being my number two. But the more I looked at the exam, the more the format made sense. By asking open ended questions, candidates could show themselves at the best, instead of just how well they could avoid being stumped by the purposely tricky, obscure or technical questions that are so common in many exams.
All Souls is a college of academics and no undergraduates, where fellows are given space and freedom to follow their academic interests. What happens in the college is a bit of an enigma, with stories of superhuman intelligence, a wine cellar that’s impossible to empty and a once-a-century ceremony involving ducks (link). Past and present members include Isaiah Berlin, Derek Parfit, Amartya Sen, Joseph Stiglitz, T. E. Lawrence and others who are far too smart to have become famous.
One door to an All Souls fellowship is an exam. Around 50 already-elite aspiring academics take it each year, with around two winning a seven year paid fellowship, putting them on the fast track to academic glory.
A recently scrapped portion of the exam had candidates write an essay based around a one word prompt. Previous prompts (link) included possessions, time, originality, play, error, style and water. The Guardian interviewed (link) one successful candidate:
Elizabeth Chatterjee, 23, who passed in 2008 when the word was novelty, said that in her year candidates spent the first hour wandering around the exam hall, thinking and sipping water.
In this most extreme of exams – and one that a four year old could answer (though simply!) – students spent an hour piecing together what they knew into something presumably profound. Compare this with most exams where the answer to a question is either obvious to the student or not, with little space in between.
(Notice also that students were “wandering around”. Cheating isn’t a problem when originality is emphasised.)
Since the one word exam was retired, candidates do two general papers and then two specialised papers. The questions quoted earlier are typical of the general paper in that they’re fairly straight forward. The questions in the specialised paper sometimes require more knowledge to understand, but a first year undergraduate would be able to make a good stab of answering most of the questions. (I’ve put a sample of questions from each 2012 paper at the end.)
These exams flip the thinking process usually expected in exams. In most exams – even those testing higher level thinking – the question tells students the concept that they are being tested on. With these more open ended questions, students need to consider what, from all their previous learning, is relevant.
We all seem to want to build creative and critical students, but so often assignment descriptions, exam questions and their assessment criteria are so restrictive that students must be at least partly mechanical in their response – and sometimes ‘playing the game’ is the easiest path to high marks. The expectations of a successful candidate at All Souls are much broader and can be found near the bottom of this page (link).
The use of students’ learning after graduation doesn’t resemble this restrictive assessment, whether in a higher academic environment or in practice. Recognising the important contexts and questions is more important than being able to answer a question, especially since clear questions are rarely presented to us outside of assignment and exams. Transfer of knowledge is increased by asking broad questions.
And so, perhaps we need to make exams easier to make them harder.
Past exam papers can be found here (link). A sample of typical questions from the 2012 exam papers is below. Papers have somewhere between 20 and 80 questions, of which students answer three in three hours.
Are all Internet users equal?
How do you know whether you are happy?
What has the Occupy movement revealed?
Do we need a theory of entrepreneurship?
‘The most effective form of foreign aid would be free immigration.’ Do you agree?
What is the right level of government debt?
Is the contract of employment a distinctive category of contract?
‘The courts are courts of law, but they are also courts of justice.’ [LORD COLLINS, 2011]. Discuss.
Should States be able to claim extra-territorial jurisdiction?
‘To date, no stable political democracy has resulted from regime transitions in which mass actors have gained control even momentarily over traditional ruling classes’ [TERRY KARL 1990]. Discuss, either solely with reference to Latin America or more generally.
Is political participation in decline in Western societies and if so what explains the decline?
Why was it possible to create an effective treaty protecting the ozone layer but not a treaty preventing dangerous climate change?
Is asking which was the first novel the wrong question? If so, what’s the right question?
‘Women aren’t funny’ [CHRISTOPHER HITCHENS]. Discuss the use of humour in the work of as many female writers as you like.
Why should we care how the Iliad was composed?
Why were Spartans more prone to violence than other Greeks?
By the time of Emperor Constantine a pagan emperor ‘could no longer govern without the acquiescence and good will of his Christian subjects’ [BARNES]. Discuss.
What was distinctive about Muhammed’s message?
What was the relationship between art and power in renaissance Florence?
In what ways did the 1950s witness the ‘end of ideology’?