DEVELOPING TECHNICAL STANDARDS
INTRODUCTORY
” You should look at this section if you have only a limited idea about technical standards and why they are important.”
When undertaking assessment, one of the key priorities is to make sure that the data collected is able to paint an accurate picture of the skills and knowledge of students. This can only be achieved if the assessment is of high quality.In classroom and school assessment, these standards may not be written down or formalised. But in large scale assessment, the definition of technical standards in the earliest phase of the programme is essential in ensuring that the data reported can be relied upon.
The document–called ‘Technical Standards’–is basically a summary of all of the elements that need to be in place in order to ensure high quality.It provides a solid foundation for the development, implementation and analysis of assessment. This includes the quality assurance of test items through a field trial (piloting), how students should be selected for participation (sampling), how the assessment should be implemented(fieldwork), what should be done with raw data (data management) and how data should be analysed (data analysis).
It is important to note that one big component of assessment–the development of assessment items–is not included in technical standards. Instead the Assessment Framework includes detailed guidance for how assessment materials (both cognitive items and contextual items) should be developed and how test forms should be built. The Assessment Framework and the Technical Standards should be used in parallel, and together provide the building blocks for a robust assessment programme
The process for developing technical standards is normally done with reference to a range of sources, including what has happened in other projects, technical standards developed for international assessment programmes and input from experts in a range of areas, including test development, sampling and data analysis. It is important that the technical standards defined are as robust as possible, but are also realistic and take account of the context. There is no value in defining technical standards that are impossible to meet
As well as guiding assessment staff in their work, technical standards also provide the basis for quality monitoring. Technical standards set the benchmark of best practice for the evaluation of the assessment instruments and the assessment process as a whole. Once technical standards have been agreed, it is important that they are adhered to. This requires the development and use of quality assurance processes at every step in the assessment programme.
During and at the end of an assessment programme, it is important to keep records of the extent to which technical standards are followed. Any reasons for not following them–which are often due to the context in which the assessment programme is being implemented–should be recorded. The objective of keeping these records is to provide as much transparency as possible on the assessment programme. The more transparency there is, the more confidence that educational stakeholders will have in the results.
To learn more about how to develop and use technical standards go to #Intermediate.
INTERMEDIATE
” You should look at this section if you already know what technical standards are but are not sure of how they should be developed or used. “
Technical standards refer to the summary of technical requirements that ensure that an assessment programme is of high quality, and can produce data that gives an accurate indication of how students are performing. They are a very important foundation document for any large scale assessment programme, to be used in parallel with the assessment framework.
Technical standards generally include reference to a wide range of elements of an assessment programme, including piloting, sampling, field implementation and data analysis. This means that when they are developed a range of different experts need to be consulted. The level of detail for technical standards should vary in relation to the purpose of the assessment and the context in which it is being implemented. In general, the more complex an assessment programme, the more detailed the technical standards should be.
Since technical standards are not only developed, but also need to be monitored throughout an assessment programme, it is common to set up a Technical Advisory Committee to oversee them. A Technical Advisory Committee can be a stand-alone entity, as is the case for international assessment programmes. Alternatively, it might be part of the overall committee that governs and oversees an assessment programme.
The technical advisory committee should be composed of people with expertise in the different elements. For example, if a sample is to be used, it is important that one of the members of the committee is an expert in sampling, not just in general contexts but with specific expertise in sampling in education.
The reason that a technical advisory committee should be used is that every assessment programme is different. The role of the technical advisory committee is to consider the specific purpose of an assessment programme and the goals that it intends to achieve. This includes the level of reporting (students, classes, schools, districts, etc.) and the specific policy goals and issues that the assessment programme aims to help inform.
It is also important that the experts understand the context in which the assessment programme is to be implemented. Technical standards need to as robust as possible, but also realistic. There is no point establishing technical standards that cannot possibly be achieved.
The technical advisory committee should determine the precise parameters (such as the acceptable level of error) that are suitable for that programme. As the programme is implemented, the role of the technical advisory group is to monitor the extent to which technical standards are adhered to. Where there are discrepancies–for example when standards for something have not been met–the role of the technical advisory group is to provide advice to the implementation team.
To find out more about technical standards and how to apply them, go to #Advanced
ADVANCED
” You should look at this section if you are already familiar with technical standards and would like to know more about their use. “
Large scale assessment programmes need to have technical standards defined before any development work happens.These should include parameters for technical areas around implementation (including piloting and sampling) and data analysis (including data management and psychometric analysis) that are fundamental to establishing a robust assessment. Standards for item and test development should be defined in an assessment framework and it is important that these two documents are used in parallel.
The technical standards should reflect the purpose of the assessment, the degree of accuracy required and the context in which it is being implemented.It is also important that the technical standards are monitored throughout the assessment programme and that adherence to them is regularly reported on.
The more transparency there is about technical standards, the more faith that educational stakeholders will have in assessment results. A Technical Advisory Group, comprising members with deep expertise in each of the topics covered by the technical standards, should be involved from the start of the assessment programme, monitoring and advising the work of the implementation team throughout.
Example technical standards from large-scale assessments of learning at the national, regional and international level include:
Pilot–A pilot should be conducted for each different group of students–for example if an assessment is to be conducted in more than one language, a pilot is required for each language. The goal of the pilot is to gather data for psychometric analysis of individual items and also to practice the logistics of field implementation. At least 200 students should respond to each assessment item in order to enable psychometric analysis of pilot data.
Sampling–When a sample of students is selected, key technical considerations include acceptable response rates, the degree of coverage of the target population, the use of stratification and the sample size. There is no such thing as an ideal sample size as this depends on the purpose of the assessment and the acceptable margin of error. Common targets include that the sample size should be large enough to yield a 95% confidence interval for all relevant estimates, and that the response rate should be 85% of sampled schools and 80% of students within those sampled schools.
Data Standards–Ensuring that the quality of data in an assessment programme is as high as possible is essential. Technical standards relating to data include details about test administration, test security, data capture, data entry, data verification and so on. For example if error rates in automatic data entry are more than 5% or in manual data entry are more than10% then data entry procedures need to be reviewed. Of course, this also implies that error rates need to be monitored and quality assured.
Psychometric Standards–Standards for psychometric analysis include a range of elements including reliability, validity, item difficulty, item discrimination, fit statistics, and differential item functioning. Together these define the technical quality of test data and the interpretations that can be derived from the test results.
