Using Secondary Analysis to Draw out Policy Implications
INTRODUCTORY
” You should look at this section if you have only a limited idea of what secondary analysis is and how it can help inform education policy.”
Assessment produces vast amounts of data. This includes data on students’ performance on assessment items, data collected through contextual questionnaires and data collected from test administration. If testing is done on digital devices, there is also likely to be data collected on how students approach and navigate the assessment (such as how long they spend on each item). In standard reports that are produced towards the end of an assessment programme, much of this data is never reported.
This means that most education systems who undertake large scale assessment have vast resources of unused data. During data analysis in an assessment programme, data analysts usually lack the time and resources to dig into this resource. Hence, undertaking secondary analysis on assessment data – when there is more time available to explore some of the interesting patterns and correlations – can be highly beneficial. Secondary analysis is particularly important in informing education policy.
One good example of the insights that can be gained from secondary analysis at the international level include PISA in Focus (https://www.oecd-ilibrary.org/education/pisa-in-focus_22260919) , in which topics of interest are illustrated using data collected across all PISA countries, with the goal of informing education policy. Similarly, the International Education Association, which is behind TIMSS and PIRLS, produces IEA Compass (http://pub.iea.nl/policy_briefs.html) which also focuses on issues of interest to education policy makers. Some recent topics include:
- Why don’t more girls choose to pursue a science career?
- Can equity in education foster social mobility?
- How is participation in sports related to students’ performance and well-being?
- Do both boys and girls feel safe at school – and does it matter?
- The importance of early learning activities at home for fourth grade student achievement
- How Chinese Taipei used TIMSS data to reform mathematics education
As these topics illustrate, the secondary analysis has involved pinpointing a particular topic of interest to education policy makers and exploring this in some depth by analysing the data in a way that was not able to be included in overall reporting.
Many countries that participate in PISA also do this with their own national data. For example in Australia there are a number of focused publications that explore topics of importance for education policy by looking at the PISA data for Australia and how it compares with that in other countries (https://www.acer.org/au/ozpisa/publications-and-data). Examples include a focus on anxiety about mathematics; a report on achievement motivation; a focus on students’ further education expectations; and a look at students’ sense of belonging at school.
All of these examples of secondary analysis are designed to provide policy makers with empirical insights to support revisions to education policy in key areas of concern, providing them with the evidence required to justify changes to education strategies and new interventions.
To find out more about how secondary analysis can help inform education policy, go to #Intermediate.
INTERMEDIATE
” You should look at this section if you already something about why secondary analysis is important for education policy and would like to know more.”
Large scale assessment generates many hundreds of thousands of pieces of data. Standard reporting only publishes information on a small fraction of the insights that can be drawn from these. There are, however, many more ways to analyse the data – particularly through looking at correlations between student performance on assessment items and data collected through contextual questionnaires. Data collected from test administration, particularly on digital devices, can also yield very interesting insights.
For education policy makers it is important to ensure that the value derived from the vast amount of time, money and resources spent on large scale assessment is optimised. This means that secondary analysis – in which a wide range of different types of analysis are undertaken in order to look at topics of interest to policy makers in a particular education system – should be a key part of any assessment programme.
Good examples of the insights that can be gained from secondary analysis at the international level include PISA in Focus (https://www.oecd-ilibrary.org/education/pisa-in-focus_22260919) and IEA Compass (http://pub.iea.nl/policy_briefs.html). Both of these use data from international assessment programmes to focus on particular topics that have profound implications for how countries support improvements in learning for all students.
For example, PISA in Focus #76 from 2017 uses international PISA data to look at how schools compensate for socio-economic disadvantage in the context of science education and finds that key factors include effective teaching practices, strong discipline and more opportunity to experience high quality science instruction and access to relevant materials. The advantage of secondary analysis of assessment data is that it is able to provide policy makers with empirical evidence that indicates which factors have a positive impact on educational outcomes. This means that education policy makers can rely on real justifications for interventions or new policy directions.
A report from the IEA provides information on how Chinese Taipei used TIMSS data to reform mathematics education. This is an example of another way in which secondary analysis can help education policy makers, allowing them to learn from successful actions by other countries that have led to educational improvements. In this example, two programmes were introduced:
- Providing supplementary classes in mathematics to low performing students.
- Introducing new approaches to mathematics teaching and learning including new instructional materials, maths camps and professional learning opportunities for mathematics teachers.
In both cases, assessment data was used to both stimulate policy and also monitor its impact, overall leading to not only better performance in mathematics but also to the introduction of new pedagogical approaches and greater motivation among teachers.
This example indicates the ways in which countries that participate in international studies such as PISA and TIMSS, and also those who undertake their own national assessment programmes, can use secondary analysis to stimulate and monitor education policy initiatives. Sometimes this is done by analysts within the teams that implement large scale assessment programmes. Another option, however, is to provide assessment data to other researchers, for example in universities.
When this happens, researchers are usually required to sign agreements about what they can do with the data. In addition, data is usually de-identified (for example, removing the names of schools, districts and even states) before being provided to the researchers.
To find out more about how secondary analysis can support education policy, go to #Advanced.
ADVANCED
” You should look at this section if you are already familiar with the importance of secondary analysis of assessment data in supporting education policy and would like to extend your understanding.”
Large scale assessment generates a vast amount of data, much of which is never analysed. This is a real waste as the data can be highly valuable for policy makers, enabling them to draw as much benefit as possible from assessment programmes (which cost a lot of time, money and resources). Secondary analysis – whether done by academic researchers or by data analysts in the assessment programme implementing team – can help shed light on issues of critical importance, and provide empirical evidence to drive policy reform.
A particular value in secondary analysis is that it enables the many possible correlations between student performance on assessment items and data collected through contextual questionnaires to be investigated. Internationally, two good examples of what can be achieved by secondary data analysis are the OECD’s PISA in Focus (https://www.oecd-ilibrary.org/education/pisa-in-focus_22260919) and the IEA’s Compass (http://pub.iea.nl/policy_briefs.html). Similar secondary analysis can also be done by countries, either drawing on their own national or state assessment data, or on data collected during international programmes.
Assessment data can be used to highlight a range of issues of importance to educational policy makers. Key among these are issues around equity – using secondary analysis to identify patterns of different performance between students and the contextual factors that are related to them. This can help the design of education policies that target greater support to students who are most disadvantaged, helping to lift the overall educational outcomes of the whole population.
Another area of education policy in which secondary analysis of assessment data can be invaluable is in revising teaching and learning practices. Large scale assessment often involves collecting data from teachers and school leaders on their approaches to teaching. When correlated with student achievement this can help to identify, for example, that student-centred learning practices are important in enhancing educational achievement, or that teacher skills and qualifications are an important factor in driving student success. Again, these kinds of insights can help
A third area in which education policy can benefit from secondary analysis is around infrastructure and management. For example, data may show that students at schools with school principals who have been teachers perform better than students at schools with school principals who are administrators with no teaching experience. Or it might indicate that the number of computers in schools does not actually correlate with better student importance. Such patterns provide an important stimulus to reshape education policy in order to ensure that all students are given the best possible opportunities for learning.
As these examples show, secondary analysis can be useful for many different aspects of education policy. In the education policy cycle, secondary analysis can help inform policy agendas, can support policy makers in designing specific interventions and can also support monitoring of the impact of policy initiatives and suggest considerations in their subsequent revision. Overall, the use of secondary analysis of assessment data in education policy is part of the drive to ensure that all educational decisions – from the classroom to the national government – are informed by empirical insights which can show evidence of need, as well as evidence of impact.
