CREATING AN ASSESSMENT FRAMEWORK
INTRODUCTORY
” You should look at this section if you have only a limited idea of what an assessment framework is and why it is important.”
When the decision is made to develop an assessment for a particular subject, grade level and/or skill, something important needs to happen before any assessment questions(referred to as items) are written. This stage involves deciding what to assess and how to assess it.
To provide a foundation for assessment development, all of these decisions are included in a document called an ‘Assessment Framework’. An assessment framework can be long or short, detailed or succinct, but is a vital component in ensuring that assessment is high quality, and collects useful information on student learning outcomes
An assessment framework needs to include a variety of information that will help guide test developers. This includes:
- Why to assess–the assessment framework should identify the purpose of an assessment–why is this assessment being delivered at this time, on this topic and to this group or groups of students;
- What to assess–the approach to decide this depends on whether the assessment is based on a curriculum or learning outcomes (for example, for grade 5 mathematics)or on competencies that go across curricula (for example, creative thinking);
- Definitions and descriptions–these make sure that everybody involved in developing the assessment material has the same understanding, for example of what grade 5 mathematics includes;
- For assessments based on curricula or learning outcomes these definitions and descriptions should already exist, but for competencies they may need to be created:
- Priorities–even when an assessment is based on a curriculum or on a set of learning outcomes, it is unlikely that it will cover everything as this would mean too much testing time for students;
- For large scale assessment, deciding which parts of the curriculum, or which learning outcomes, should be prioritised should be a formal process that brings together people with expertise in the subject, in the curriculum and in assessment;
- For school and classroom based assessment, teachers and school principals can decide what to prioritise based on their knowledge of subjects and the students they teach;
- How to assess–this includes considerations such as what format to use (e.g. paper based or digital); what item type to use (e.g. multiple choice questions or open response tasks); what skills to measure (e.g. should students be asked to recall information or to apply their knowledge to real world scenarios);
- What format to use–this includes elements such as the duration of the test, the number of items to include, the number of test booklets to use, whether or not to use images and so on
All of these are very important decisions and it is important that both the decisions and the reasons for choosing one option over another are recorded in the assessment framework.This will guide those people involved in test development and will also act as a foundation for elements such as data analysis and reporting.Importantly, an assessment framework should never be fixed–it should be reviewed regularly and adapted to fit evolving assessment needs over time, both after piloting items and also over time.
To find out more about how to develop an assessment framework, go to #Intermediate
INTERMEDIATE
” You should look at this section if you have some idea of what an assessment framework is but would like more details. “
For assessment professionals it is important to understand the advantages and disadvantages of different forms of assessment, and to know the purpose for using each one. In the past it was common to focus on the difference between formative and summative assessment, or assessment ‘for’, ‘as’ or ‘of’ learning. These days, however, ‘purpose pluralism’ is increasingly referred to, indicating that assessment has a number of simultaneous purposes.
An assessment framework is an important part of an assessment programme. It is:
- A statement of the purpose, priorities and content of the assessment that can be shared publicly, and can prevent misunderstandings;
- A summary of design decisions that can guide test developers in developing assessment items to make sure that these reflect the purpose and priorities;
- A way of ensuring that the assessment is consistent if there is a desire to monitor patterns in student performance over time;
- A rationale for design decisions and for any changes over time;
- A resource to identify how well the assessment instrument has achieved its objectives;
- (If developed collaboratively) A way of giving a sense of ownership over the assessment to help stakeholders take it seriously and pay attention to the results.
An assessment framework needs to define the purpose of the assessment as this will determine design. For example, if the purpose is to select students for entry to university, then the design will be very different to an assessment that aims to provide information on students’ strengths and weaknesses to inform improvements to teaching and learning.
Once the purpose has been determined, priorities need to be made. Prioritising what should be assessed involves a review of the curriculum, learning outcomes and other relevant materials. For large-scale assessment this decision should be made by subject, curriculum and assessment experts. For school or classroom assessment, teachers and principals can make this decision with reference to what is most important in their context.
Once the domain has been selected, it needs to be defined. This is important so that everyone has the same understanding about what it means. In addition, the sub-categories within the domain should be decided. These comprise both content and cognitive elements.
Domain content areas (also called sub-domains or content categories) include things like different types of text in reading (informative, narrative, persuasive etc.) or the sub-categories in mathematics. It is important to note that these vary slightly by grade level. For example, TIMSS includes three content domains in grade 4 mathematics (number; measurement and geometry; and data) but four in grade 8 mathematics (number; algebra; geometry; data and probability). Domain cognitive areas (also called skills, processes or aspects)include things like application, knowledge and reasoning.
Once the content and cognitive domains have been selected and defined, the next step is to work out what proportion of items should be devoted to each one. For example there might be more items on numbers than on data. Again, these decisions need to be justifiedand recorded for future reference as well as to guide test development.
Once all of the elements of assessment contents have been defined, it is also important for the assessment framework to include information on how the assessment will be provided to students. For example, will it be in paper and pen format or digital?; what types of items will be used?; how many items will the assessment instrument include?; how much time will students have to respond?; and how will results be reported?
The assessment framework may include information on the collection of contextual information from students, teachers, schools and parents but sometimes this is included in a separate document called a contextual framework.
To find out more about how to develop an assessment framework, go to #Advanced
ADVANCED
” You should look at this section if you already know about how to develop an assessment framework and are interested in more details. “
There are a number of complex decisions to be made in developing an assessment framework. None of these decisions have right or wrong answers–the design of the assessment must be based on the purpose of the assessment and the context in which it is taking place.
These decisions are very important, however. Decisions about what to assess have a huge impact on teaching and learning. This is a phenomenon called the ‘washback effect’ and refers to the way in which what is assessed reflects what are regarded as important educational outcomes. If an assessment programme focuses on what students can recall, this will encourage a focus in teaching and learning on rote learning. In contrast, if an assessment programme focuses on measuring students’ ability to apply their skills and knowledge to real world situations, this will stimulate a very different approach to teaching and learning.
It is important that all terminology used in an assessment framework is defined so that there is no room for misinterpretation. For example, it is not enough to state that grade 4 mathematics will be assessed. Instead, a detailed explanation is required about what this means and how the assessment will be designed to reflect the meaning.
For example, the assessment framework for TIMSS Grade 4 mathematics defines the domain as comprising number (50%); measurement and geometry (30%); and data (20%).These are then further subdivided as follows:
Number (50%): whole numbers (25%); expressions, simple equations, and relationships (15%); and fractions and decimals (10%)
Measurement and geometry (30%): measurement (15%); and geometry (15%)
Data (20%): Reading, interpreting, and representing data (15%); and using data to solve problems (5%)Moreover, definitions are also given for each sub-division. For example one component of measurement is defined as“measure and estimate lengths (millimeters, centimeters, meters, kilometers)”.
This level of specification may seem high, but it is important that the assessment framework includes all the information that test developers need. In addition to definitions this includes: the number of items needed per sub-domain; the kinds of items that will gather the most accurate information on students’ knowledge, skill and understanding; the mode of delivery that will be most suitable in the context, and so on. The assessment framework also plays an important role in ensuring that reporting reflects what the assessment actually measures.
The most challenging scenario is if an assessment programme intends to define something that is not documented elsewhere. For example, to measure problem solving skills among students in grade 5 and grade 10. In this case an immense amount of research and consultation with experts would be needed to define problem solving skills, the sub-domains of problem solving skills and the expected differences between grade 5 and grade 10.
Frameworks are never permanent. They have to adapt to changes in curricula and learning outcomes, available modes of delivery, methods of marking and so on. They should also change to adapt to results from pilot and early implementations of the assessment programme.
About the guidance levels
