Psychometric test reliability and validity and what to check
A psychometric test should be judged by the evidence for that particular test, its intended use and the people taking it. “Psychometric” is not a quality certificate. Reliability concerns consistency of measurement; validity concerns whether the evidence supports the interpretation or decision being made.Neither question can be answered by the appearance of a report or a single claim that a test is “accurate”. Before using results for a major course or career choice, ask what was measured and how the conclusions were reached.Consistency alone does not justify a career recommendation. Ask for evidence relevant to the test, its intended use and the people taking it.
Reliability asks about consistency
Scores contain measurement error. Conditions such as distractions, differences in administration or a person’s state at the time can contribute to variation. The US Office of Personnel Management explains reliability in terms of score consistency and notes that no assessment is perfectly consistent. Designing an assessment strategy.That does not mean you should expect an identical score every time you take a test. Ask how reliability was studied, for which population and for the scale you are interpreting. A technical statistic without that context is difficult to use.If a report treats a small difference between two scores as decisive, ask whether that difference is meaningful given the measurement uncertainty. You do not need to calculate the answer yourself; the provider should be able to explain the limits in plain language.
Validity asks whether the conclusion is supported
A consistent measure can still be used to answer the wrong question. Imagine a hypothetical assessment that measures accurate typing. Even if it measures that skill consistently, that alone would not justify a recommendation that someone should study architecture.For employment selection, OPM describes validity evidence in relation to job performance and explains that the appropriate evidence depends on the use. Evidence that a task resembles job duties is different from evidence that scores predict later performance. OPM assessment guidance.Apply that distinction when reading a career report. Ask what supports the move from a test result to a suggested occupation. A measure designed for one purpose should not acquire a broader purpose simply because the output looks detailed.
Ask who the comparison group represents
Some scores compare your performance with a reference group, often called a norm group. To understand such a score, you need to know who is in that group and why the comparison is appropriate. Age, educational background, language and the assessment setting can all be relevant to interpretation.The International Test Commission’s guidelines call for appropriate comparison groups, attention to context and caution against extending results to characteristics the test did not measure. ITC Guidelines on Test Use, section 2.7.A percentile is a relative position, not the percentage of questions answered correctly. For example, a reported position around the 75th percentile describes standing relative to a specified group; it is not “75 per cent suited” to a career. If a report shows a career-match percentage, ask for that separate score’s definition rather than assuming it is a probability of success.
Examine language and access before interpreting a result
If instructions were difficult to understand, the display was inaccessible or the session was interrupted, tell the provider what happened. Ask how the issue should be recorded and whether the result can still be interpreted as intended. Do not estimate an adjustment to the score yourself.A translated test also needs an explanation of its suitability. Ask which language version was used and what evidence supports it for the intended group. A translation by itself does not answer that question.Raise access needs with the assessment provider before the test wherever possible. The Open University recommends discussing needs early and gives examples of accessible formats and equipment to ask about. Confirm the arrangements available for your assessment rather than assuming that one solution suits everyone. Open University guidance on access to assessments.
Do not mistake fairness questions for a simple verdict
Ask whether the provider has examined how the assessment works across relevant groups and whether its use is appropriate for your situation. An unsupported claim that a test is “bias-free” does not resolve that question. Nor does a difference in scores, by itself, explain its cause.Keep the enquiry specific. You may need to understand the test language, the comparison group, access arrangements or the evidence for a particular use. A provider should address the relevant issue rather than respond only with a general statement about scientific design.If you cannot obtain a clear explanation, avoid treating the result as the sole basis for an important decision. You can continue investigating courses and work while seeking clarification.
Use a short evidence checklist
Before relying on a report, ask:
What is the exact test and version, and what was it designed to measure?
What evidence supports using it for this decision and this group of people?
Which score or scale supports the recommendation, and what are its limitations?
Who is the comparison group, if one is used?
How were language, access needs and any interruptions handled?
What other information should be considered alongside the result?
These questions help you evaluate the explanation. They are not a substitute for a technical review of an assessment or a new scoring system for tests.
Put the report beside other evidence
Consider a hypothetical student whose report suggests difficulty with a reasoning task, while their coursework shows strong performance on related assignments. The useful next step is to examine the difference: were the tasks comparable, what support was available, and were the test conditions appropriate? Neither source should be dismissed automatically.For a career choice, also examine interests, actual work samples, course requirements and practical constraints. Our guide to turning assessment results into a career roadmap focuses on that next step.If you enquire about testing through our career guidance service, ask us to identify the proposed assessment and explain the feedback process. A responsible conversation should make the result’s scope understandable before it becomes part of a decision.
