Developing Contextual Questionnaires
INTRODUCTORY
” You should look at this section if you have only a limited idea of what a contextual questionnaire is and why it is important.”
An assessment framework is the overall plan of an assessment. Apart from describing what areas of learning are measured, it also defines what additional information to collect and how to collect it. This extra information is related to the context in which learning takes place. Collecting this information helps to improve reporting of assessment data, taking into account the factors that are likely to influence student achievement.
These contextual aspects need to be defined in the assessment framework. Deciding on the most important contextual factors to include is ideally done in a collaborative way, including a number of different stakeholders. This will be informed by knowledge of the context as well as factors that have been empirically proven to influence student achievement. Possible contextual components for inclusion include:
- Students – common characteristics such as gender and age, as well as subjective factors such as motivation;
- Home environment / parents – issues related to parents, such as level of educational achievement and occupation, and issues related to the home environment such as the availability of books and/or a computer;
- The school and the classroom – issues related to how the school and teachers support student learning, such as teacher-student ratios and resource availability;
- The wider community – issues related to the local community but also the region and country, for example funding for education, systems of government and GDP.
Depending on the purpose of the assessment, some of these may be given greater emphasis, while others may be omitted entirely. The fundamental consideration in deciding what contextual data to collect is to consider what is going to be done with it. There is no point collecting information for no reason, so there should be a clear justification for the inclusion of each piece of contextual information.
Beyond deciding WHAT contextual information to collect, and WHY, other questions are HOW, WHERE, WHEN and from WHOM. For example, young children are likely to be unable to provide contextual information about themselves, so parents or teachers would be more relevant to collect this information from.
Another consideration is the mode and timing of collection of contextual information. For example although survey instruments are commonly used, it might be relevant to collect contextual information from teachers and parents in a digital format such as a phone app, and it might be relevant to collect contextual information from students through an interview.
A key skill in collecting contextual information, and in developing contextual questionnaires, is the design of contextual items. Instead of cognitive items that are designed to gather data on skills and knowledge, contextual items are designed to gather data on behaviours, practices, beliefs, attitudes, perceptions and so on.
All items need to map back to the assessment framework. Items must be designed to collect truthful and complete information from respondents and questionnaires should be designed to be easy to use, and to include appropriate item formats that are simple for respondents to answer. They should be designed to be as short and efficient to administer as possible.
To find out more about linguistic quality control, go to #Intermediate
INTERMEDIATE
” You should look at this section if you have some idea of what a contextual questionnaire is but would like more details.”
The assessment framework defines not only what is to be measured, but also what additional contextual information is to be collected. Context-related information refers to all the individual, social and environmental factors which may influence student performance. This information needs to be collected so that reporting can take account of factors that have a positive or negative impact on student learning.
Careful consideration needs to be given to determining what contextual information to collect; who to collect it from; how to collect it; and when to collect it. All of these decisions should be made in relation to the reason for collection – there is no point collecting contextual information if it is not going to be used in reporting. Reference to contextual questionnaires from other studies can help inform the design of contextual questionnaires but it is essential that these are adapted to be as relevant as possible.
Contextual information is often collected about the following key aspects: students’ background and motivation; home environment; the school and classroom; and the wider community. Some of this information may already be available, for example from a census,
Once the required information has been identified, the challenge lies in designing one or more contextual questionnaires that collect that information in an efficient way as possible, imposing the least possible burden on respondents.
Instead of cognitive items that are designed to gather data on skills and knowledge, contextual items are designed to gather data on behaviours, practices, beliefs, attitudes, perceptions and so on. All items need to map back to the assessment framework. There are a few key design features that all good questionnaires should possess:
- Information about the purpose of the collection and privacy;
- A layout that promotes easy navigation between items;
- A range of item types to target different information;
- Simple instructions on how to respond to different item types;
- Simplicity, brevity, avoidance of ambiguity.
Contextual items are quite different to cognitive items, and tend to be more varied. Some of the most popular ones are:
- Categorical items (for example about gender or age) in which respondents have to select one or more categories;
- Ordinal items (for example about interest, agreement, frequency or satisfaction) with a range of options on a continuum from least to most; and
- Open response items which ask respondents to provide from between one word to a few sentences in response to a prompt
All items should follow the BRUSO rule – be brief, relevant, unambiguous, specific and objective. Categorical and ordinal items can be very difficult to design and the design process may include preliminary qualitative research with respondents in order to identify how to phrase items and which response options should be given.
Just like cognitive instruments, it is important that contextual instruments are piloted with target respondents to check that all items are able to collect relevant data and that respondents have a clear idea of what is expected of them.
To find out more about linguistic quality control, go to #Advanced
ADVANCED
” You should look at this section if you already know about contextual instruments and why they are important and are interested in more details.”
The assessment framework puts forward not only the rationale underlying assessment and the methodological strategy, but also additional context-related information to collect, which may be associated, potentially through a causal relationship, with students’ level of proficiency in the domain assessed.
These contextual factors are conceptualised as variables and operationalised into empirical indicators for their use in the survey. They are related to individuals – not only students – and institutions involved in the educational process. Their choice is dictated by the requirement of the assessment at hand, although previous research can help identify the relevant phenomena likely to exert an influence on student performance.
Common factors include those related to individual students (their attitudes, beliefs, perceptions, motivations, self-efficacy, etc.); their home environment (parents’ educational achievement, resources, socio-economic status, etc.); educational environment (school leadership, resources, teacher skills, pedagogy, etc.) and the wider community (population density, GDP, etc.). If alternative sources of information for any of these exist already (as is often the case for the wider community) then there is no need to collect additional information.
The decision to select particular contextual variables for inclusion should always return to the purpose of the assessment programme, and all contextual items need to be mapped to the assessment framework. Contextual information should not be collected unless it contributes to the purpose of the assessment and will be used in data analysis. Similar quality assurance processes should be used for the development of cognitive and contextual items, including piloting with representatives of the target population.
It is common for contextual questionnaires to be developed for different cohorts of respondents, for example for students, teachers and school leaders. Constructing contextual questionnaires involves two key processes – writing the contextual items and then designing the contextual instrument. While the contextual instrument is usually a questionnaire, this does not have to be the case. For example with some student cohorts short interviews may be more relevant.
The structure and the content of the questionnaire should be conducive to the collection of valid, reliable and complete information. Items must be clearly worded and unambiguous; the overall logic (branching) and layout of the questionnaire should make it unproblematic for the respondent to move for one section to the following one, and from one item to the next (e.g. an item cannot be split in two pages).
Sensitive information – such as about income – should not be collected unless absolutely necessary, and researchers need to be cognisant of the various types of bias that can arise when particular information is sought. For example a long questionnaire or overly complex items can lead to response fatigue, and hence the collection of inaccurate information. Another common form of bias is when respondents give ‘socially-desirable’ responses.
All items should follow the BRUSO rule – be brief, relevant, unambiguous, specific and objective. Writing good contextual items can be difficult and preliminary research, such as through interviews with representatives of respondents, may be required prior to the drafting stage. There are many reference materials available on good questionnaire design and these should be referred to, in addition to drawing on people with expertise in questionnaire design during the developmental phase.
About the guidance levels
