API 510 Chapter 19 – The Final Word on Exam Questions
19.1 All exams anywhere: some statistical nuts and bolts for non-mathematicians
The statistiscs of exam questions are unlikely to tax the brains of eminent statisticians for very long. Let’s say we want to choose 150 questions randomly from a set of, say, 900 if that’s how many we have. Now take one question, the classic query about, among other things, how many angels can realistically be persuaded to dance on the head of a pin. Let’s call this, for convenience, question P (P for pin). Every so often, when choosing the question set for our exam, question P will no doubt appear; but how often?
Take the first exam cycle; if we choose 150 questions truly randomly from one large set of 900, then the chance of our question P appearing in the exam is precisely 150/900 or 1 in 6 (or 16.67 % if you like). Put another way, over time it will appear once in every six exams. This means, of course, that if you keep on taking the exam again and again, five times out of every six any effort you have put into remembering the answer to question P (the answer is 14 incidentally) will have been totally wasted.
But hold on; the situation has changed. You now know the answer, so we want it to appear next time. At the next exam cycle, the chances of P appearing are once again 16.67 (but the cumulative probability of the two successive appearances, given that it already had only a 16.67 % of appearing last time) are much less . . . let’s say 1 in 36. Extending this out, the probability of P turning up in three successive exam cycles are getting pretty thin and in four successive cycles, miniscule at best. The odds against you are awful . . . it’s looking like your valuable knowledge of the P answer will most of the time be wasted.
Just when you thought things were looking bad . . . it gets 301 worse. Waiting in the wings is another question Q, to which you already know the answer (it’s the one about the length of the piece of string). It would be nice if this appeared regularly, so you could confidently tick the correct answer box marked 17 inches. Even better, what if both P and Q appeared in every exam draw now you could get them both right. What’s the chance of this?
- The probability of P and Q both appearing in the first draw are small – we know that .
- In the first and second draw, an order of magnitude smaller .
- In the first, second and third draws . . . . undeniably tiny .
- And, in the first, second, third and fourth draws . . . miniscule would be the operative word
Extending this out to say, five questions, P, Q, R, S and T, that you are certain of the answer, the chances of all five appearing in three successive draws are so near zero that the calculation would be enough to leave your calculator a blackened ruin.
But wait . . . if, by some stretch of the imagination, such a thing did actually happen, you would have to conclude that either these lottery-type odds had occurred or that there was maybe some other explanation. But what could it be?
Working the maths backwards would tell us that we could only be certain to get such a run of unlikely probabilities if we had not 900 questions to choose from at all but a significantly smaller set. This is the much-reduced set size we would need if we were still drawing 150 purely at random from one big set and our five questions P, Q, R, S and T all miraculously appeared in each of three successive draws. That’s one possibility.
There’s another way to do it. If we divide our bank of 900 questions into, say, 10 sets of 90 questions each, based on subject breakdown, and let’s say our exam draw will require that we draw 15 questions from each set to give us the 150- question draw. If our favourite questions P, Q, R, S and T each reside in a different set (which they will, as they are about different subjects), then the odds of all five appearing in three successive draws are reasonably believable, with a little imagination at least. Adding some regular preferences within each set would reduce the odds even further.
In the final act, with a little biasing towards certain questions in each small set our question-drawing exercise will throw off its mask of randomness and, right on cue, up will pop our P, Q, R, S, T combination with a flourish, like the demon king among a crowd of pantomime fairies.
So if you ever see this, the explanation may be one of the options above.
19.2 Exam questions and the three principles of whatever (the universal conundrum of randomness versus balance)
As with most engineering laws and axioms (pretend laws) you won’t get far without a handful of principles (of whatever).
The first principle (of whatever) is that, faced with the dilemma between randomness and balance, any set of exam questions is destined to end up with a bit of both. A core of balance (good for the technical reputation of the whole affair) will inevitably be surrounded by a shroud of some randomness, to pacify the technically curious, surprise the complacent and frustrate the intolerant – in more or less equal measure. There is nothing wrong with this; the purpose of any exam programme must be to weed out those candidates who are not good enough to pass.
Now we have started, the first principle spawns, in true Newtonian fashion, the second principle – a strategy for dealing with the self-created problems of the first. The problem is the age-old one of high complexity. Code documents contain tens of thousands of technical facts, each multifaceted, and together capable of being assembled into an almost infinite set of exam questions. We need some way to deal with this. The second principle becomes: selectivity can handle this complexity.
Tightening this down, we get the third principle: only selectivity can handle this complexity. There’s nothing academic about the third principle (of whatever); it just says that if you try to memorize and regurgitate, brightly coloured parrot-fashion, all the content of any exam syllabus, you are almost guaranteed to fail. You will fail because most of the time the high complexity will get you. It has to, because exam questions can replicate and mutate in almost infinite variety, whereas you cannot. You may be lucky (who doesn’t need a bit of luck?) but a more probable outcome is that you will be left taking the exam multiple times. Round and round and round you will go at your own expense, clawing at the pass/fail interface.
A quick revisit of the first principle (of whatever) suggests that being selective in the parts of an exam syllabus we study carries with it a certain risk. The price for being selective is that you may be wrong. Most of the risk has its roots in the amount of balance versus randomness that exists in the exam set. The more balanced it is, the more predictable it will be and the better your chances. Don’t misread the situation though; your chances will never be any worse than they would have been if you hadn’t been selective.
The third principle tells us that. Remembering this, you should only read the tables in section 19.3 if you subscribe to the three principles and you think selectivity is for you. If you don’t recognize the code references, clause numbers or abbreviations then you need to start again at the beginning of this book.
19.3 Exam selectivity
For the wise
|Question||Subject: open book|
|2||RT backscatter symbol|
|6||RT joint type|
|10||Wall thickness calculation|
|11||RT slag acceptance|
|12||Re-rating Fig. 8.1|
|14||NPS 2 nozzle to shell|
|15||PRV set pressure|
|18||Vessel head calculation|
|21||Dry MT temperatures|
|27||Charpy specimen length|
|30||Temporary repair dimensions|
|33||Defects at weld toes|
|39||CD welding (again)|
|40||Charpy values table|
|44||Elliptical head calculation|
|45||RT step wedge|
|47||CD welding in lieu of PWHT|
|50||Cooling water corrosion|
|For the hopeful*|
|Question||Subject: open book|
|1||Factual questions from API 510 sections 1–4 that ﬁt my experience|
|5||Hard engineering logic questions from API 510 sections 5 and 6|
|8||Experience-based questions from API 510 section 7|
|11||API 510 section 8|
|15||ASME VIII head and shell calculations (easy if you can use a calculator)|
|19||Pressure testing questions (may need to consult the parrot)|
|23||Easily found points from API 572 that are obvious to anyone in this inspection business|
|28||API 571 DM questions . . . I’ll have a guess at those . . .|
|34||NDE questions from ASME V
… .. No problem with my previous experience. I used to be an NDE technician, you know
|44||Easily found points from API 577 that I agree with|
|47||ASME IX exercise (can be quite tricky . . . hope they’re not too hard)|