Validity & Reliability
CPD: Assessment for Teachers (by Dawn Cox), topic 1.1.1
Reliability and Validity of Assessment
Teachers should consider the reliability and validity of any assessment that they design/issue to pupils.
Definition of validity
- To Stuart Kime:
- The ability of the assessments to test what it is meant to test and to provide information that is valuable and appropriate for its purposes.
- An assessment is not valid in itself. Its validity is related to the inferences made from the results of an assessment and the factors that impact these inferences.
Definition of reliability
- Reliability is a sliding scale on the extent to which an assessment result is free from error.
- In practical terms, a reliable assessment means that if a pupil repeated the assessment, they would get the same result.
- For Ofqual, reliability is making sure that students get a fair result, regardless of:
- Who is marking the paper.
- Types of questions (eg: multiple choice or essay).
- Which topics are covered.
- Date of the exam.
The importance of the assessment
- According to Wiliam (2014), the importance of the assessment is not to test how well students did.
- Rather, its importance is in what we can say from the evidence gathered about what those students are able to do in other contexts and at other times.
- The challenge for teachers is to minimize unreliability where possible.
Important notes
When using assessments in the classroom it is important to consider if they provide an accurate picture of what we want to know about a pupil and their learning.
No assessment is 100% reliable or 100% valid, however, there are steps teachers can take to increase this.
Whilst validity and reliability are often discussed separately, they are linked. The reliability of an assessment affects its validity.
Important Questions about Reliability and Validity
Things to keep in mind when discussing reliability and validity of assessments.
Threats to validity
- According to Crooks et al. (1996), the following factors threaten the validity of an assessment:
- How it is administered.
- How it is scored.
- How performance in the assessment is used to make judgments and how we act on those judgments.
- How scores are aggregated across papers to define students' performance in the whole course from questions about its specific parts.
Questions on validity
- Assessment design:
- What part/s of the course is being tested?
- What is the difficulty level?
- Does it test what it needs to test, or is it testing things we are not interested in? E.g. if a pupil can interpret a question rather than answer it?
Questions on validity
- Performance:
- What factors might affect performance on the test? E.g. where and when it is taken, level of readability.
- Results:
- How will the results of the assessment be used? What inferences will be made?
Questions on reliability
- Administration:
- What is available to students when they complete an assessment?
- What time of day was it completed?
- Were there any distractions?
- How much time is available to complete the assessment?
- Subjectivity:
- Who is marking the work? Are they biased?
Questions on reliability
- Standards:
- What does a ‘good’ answer look like?
- What criteria makes a successful answer?
- Consistency:
- How will answers be moderated? Do all markers mark in the same way?
- Marking:
- What variables might affect how marks are allocated?
- Is there any pressure on the results of the marking that may influence how it is marked? Eg. accountability measures?.
Test yourself on Validity & Reliability
What does assess mean in Latin?
What is the job of teachers when it comes to assessment reliability?
According to Crooks et al. (1996), what are the threats to the validity of an assessment?
What can be said about the reliability of an assessment?
What can be said about the validity of an assessment?