Validity & Reliability

Test yourself

Reliability and Validity of Assessment

Teachers should consider the reliability and validity of any assessment that they design/issue to pupils.

Definition of validity

To Stuart Kime:
- The ability of the assessments to test what it is meant to test and to provide information that is valuable and appropriate for its purposes.
An assessment is not valid in itself. Its validity is related to the inferences made from the results of an assessment and the factors that impact these inferences.

Definition of reliability

Reliability is a sliding scale on the extent to which an assessment result is free from error.
In practical terms, a reliable assessment means that if a pupil repeated the assessment, they would get the same result.
For Ofqual, reliability is making sure that students get a fair result, regardless of:
- Who is marking the paper.
- Types of questions (eg: multiple choice or essay).
- Which topics are covered.
- Date of the exam.

The importance of the assessment

According to Wiliam (2014), the importance of the assessment is not to test how well students did.
Rather, its importance is in what we can say from the evidence gathered about what those students are able to do in other contexts and at other times.
The challenge for teachers is to minimize unreliability where possible.

Important notes

When using assessments in the classroom it is important to consider if they provide an accurate picture of what we want to know about a pupil and their learning.
No assessment is 100% reliable or 100% valid, however, there are steps teachers can take to increase this.
Whilst validity and reliability are often discussed separately, they are linked. The reliability of an assessment affects its validity.

Important Questions about Reliability and Validity

Things to keep in mind when discussing reliability and validity of assessments.

Threats to validity

According to Crooks et al. (1996), the following factors threaten the validity of an assessment:
- How it is administered.
- How it is scored.
- How performance in the assessment is used to make judgments and how we act on those judgments.
- How scores are aggregated across papers to define students' performance in the whole course from questions about its specific parts.

Questions on validity

Assessment design:
- What part/s of the course is being tested?
- What is the difficulty level?
- Does it test what it needs to test, or is it testing things we are not interested in? E.g. if a pupil can interpret a question rather than answer it?

Questions on validity

Performance:
- What factors might affect performance on the test? E.g. where and when it is taken, level of readability.
Results:
- How will the results of the assessment be used? What inferences will be made?

Questions on reliability

Administration:
- What is available to students when they complete an assessment?
- What time of day was it completed?
- Were there any distractions?
- How much time is available to complete the assessment?
Subjectivity:
- Who is marking the work? Are they biased?

Questions on reliability

Standards:
- What does a ‘good’ answer look like?
- What criteria makes a successful answer?
Consistency:
- How will answers be moderated? Do all markers mark in the same way?
Marking:
- What variables might affect how marks are allocated?
- Is there any pressure on the results of the marking that may influence how it is marked? Eg. accountability measures?.

1Introduction to Assessment

1.1Definitions

1.1.1Validity & Reliability

1.2Types of Assessments

1.2.1Summative & Formative Assessments

2Feedback

2.1Feedback

2.1.1Feedback

3Assessment Methods

3.1Assessment Methods

3.1.1Research on Assessments

3.1.2Rubrics & Comparative Judgment

3.1.3MCQs & Self-Assessment

Jump to other topics