Standardizing Field Operations

 INTERMEDIATE

“You should look at this section if you already know something about how to standardise field operations but would like to know more.”

Field operations–also known as test administration–are often overlooked in assessment programmes. Many people incorrectly assume that they are the least important, and least technical, part of an assessment programme. In fact, field operations are one of the most critical parts of an assessment programme. If field operations are not done properly–leading to inconsistencies in the testing conditions for students–then the assessment programme will not result in accurate data that can be used for valid comparisons.

It is not possible to control every element of the conditions in which students are tested. For example there may be contrasts in school infrastructure, resources, lighting and temperature. In high stakes testing (such as for university entrance) these differences can be overcome by moving students to testing centres which have equivalent conditions. For other types of assessment, however, this is not possible and hence the goal of field operations is to ensure that as many conditions as possible are as similar as possible.

Field operations are underlain by four key elements: the development of a field operations manual; the recruitment and training of field operators; the monitoring of field operations; and reporting on field operations. Examples of the key considerations for each one include:

Development of a field operations manual–for small scale testing this many simply include details such as what instructions and how much time to give to students. For large scale testing, however, field operations manuals can be very detailed. They may include how to set up the test room, a script of instructions for field operators to read out to students, precise instructions on timing (including when reminders should be given) and how to collect test forms. When testing is done on digital devices, much of this can be automated so that, for example, the programme stops when the maximum time has been reached.

Recruitment and training of field operators–the major challenge in field operations is ensuring that field operators follow instructions. This cannot be achieved by simply giving them a manual. Instead, it is important that they are trained in how to use it, including role-plays where possible. The training should be the same for all field operators, with a training manual and training for trainers on how to conduct training sessions.

It is also important to select appropriate field operators. They should not have any prior knowledge of the students, should be familiar with working with students, should complete all required training and should return all documentation. It is common to recruit students at teacher training colleges or retired teachers to be field operators.

Monitoring of field operations–however well trained field operators are, some may not follow procedures. Hence some degree of monitoring should be undertaken. This may involve something like sending a monitor to 10% of test sessions. Monitors require their own manuals and training prior to monitoring activities.

Reporting on field operations–all field operators should collect certain information on the implementation of testing. This may include the names of students and making notes of any inconsistencies. These records should be kept in case there are anomalies in test data. For example a power cut, meaning that students have to take the test with insufficient light, is important to report as it may lead to poor performance among affected students.

To find out more about advanced field operations, go to #Advanced.

  ADVANCED

“You should look at this section if you are already familiar with how to standardise field operations.”

Field operations are vital to assessment programmes–done well, they ensure that assessment data can yield valid data. Done poorly, they can undermine the assessment programme and lead to results that are not accurate.

For example, a common error is to use classroom teachers to administer assessments to their own students. This is almost certain to ensure that test conditions will not be standardised. Teachers are naturally keen to help their students do well, and often their own performance is measured by students’ results, providing a perverse incentive for inappropriate field operations, and therefore undermining data accuracy.

Field operations cannot control all variables–it is often very difficult to control the conditions in which students are tested if, for example, infrastructure is poor. Hence the objective should be to standardise as much as possible (and take different testing conditions into account at the data analysis and reporting phase). It is important to note that a great deal of preparation is required before field work. Excellent project management and planning is required to ensure that everything runs smoothly, and this should start as early as possible. The rule of thumb is that ‘anything that can go wrong, will go wrong’ and there should be plans in place in case of any eventuality. The coordination of schools can be very challenging and this needs careful management.

The main tools of field operations are field operators, field operations manuals, monitors and reports. These should be used in piloting AND the main implementation. All of these include a number of key considerations:

Field operators–Field operators should be as objective as possible.If they already know the students this is unlikely, so it is important to recruit and deploy neutral but qualified field operators. It is vital that field operators have experience working with students. Assessment can be very stressful for students–especially young children-and people who are not able to communicate appropriately with students can worsen their stress.Students in teacher training colleges and retired teachers are often found to be suitable field operators.

Field operations manuals–For large scale assessment programmes, field operations manuals need to be detailed. They may include a script for field operators to read out; precise timing for instructions; a seating plan; details on how to hand out and take back test papers (or instructions for dealing with any problems that arise in digital modes of assessment);instructions on secure storage; resources required and templates for reports.

Training and monitoring field operators-Simply disseminating field operations manuals will not result in adherence unless field operators are also trained and monitored. This required detailed planning, training resources, the training of trainers and so on. Monitoring of a proportion (often 10-20%) of testing sessions is also highly recommended in order to identify any errors that may interfere with the accuracy of data.

Reports-In-depth records of monitoring and field operations should be kept, and referred to if anomalies are detected during data analysis. For example a power cut during a test session could have a severely negative impact on student performance and this needs to be taken into account. Field operators also need to be given guidance on what to do in these situations. Detailed logs should also be kept on the receipt, handling and dispatch of test forms or (in digital formats) and technical issues encountered.