Teaching and School Practices Survey Tool He Rauemi Uiui mō te Mahinga Kaiako, Mahinga Kura

How we developed the Teaching and School Practices survey tool

Why this tool?

How do you support busy leaders and teachers to take advantage of the strong body of evidence now available about effective teaching and effective school practices which support learning?

How do we know what is happening nationally, so that the Ministry of Education and other government education agencies can see what the strengths and needs are of our schools, and whether policy is having its intended effect?

The Teaching and School Practices Survey Tool is designed to answer both these questions. The Ministry of Education commissioned NZCER to develop and provide the tool, building on NZCER’s previous work developing and administering the Educational Leadership Practices survey.

The tool that you can access now is the result of an extensive development process, including:

  • a review of current and relevant research
  • identification of domains (topic areas) to cover
  • creating or finding specific questions that exemplified each domain and would be of practical use to schools
  • discussion of our work with a strong Advisory Group
  • in-house trial with recent teachers to refine the questions for the online trial
  • online trial of a set of questions with 38 schools
  • psychometric analysis of the results to see how the questions in each domain related to each other, showing that they were giving evidence of a common concept
  • refinement of the questions and reducing the total number, using comments made by teachers and principals in the trial, the results of the psychometric analysis, and Advisory Group discussion
  • feedback from trial school principals and the Advisory Group on the use of the tool.

A strong Advisory Group

It was essential to have a strong Advisory Group informing and supporting this work. It was also vital to bring together sector group leaders, the government agencies, researchers and professional development providers, to ensure different knowledge and expertise was brought into play, and that we forged common ground.

We thank the Advisory Group members for their stimulating and productive discussions, practical support with the trial, and their enthusiasm about the tool and its usefulness for schools and providing a national picture.

Our Advisory Group met three times:

  • to discuss background papers on the domains we suggested should be included in the surveys, with the research behind them
  • to give feedback and try out the draft items we developed to take to the trial, and
  • to discuss the trial results and help us finalise the items to include in the survey now available.

Our Advisory Group will meet again late in 2017 to discuss the national report coming from the use of the Teaching and School Practices survey tool in Terms 2 and 3.

Advisory Group Members

  • Robyn Baker (consultant, heading the TLIF team, formerly Director, NZCER)
  • Pauline Barnes (Education Council)
  • Mere Berryman (University of Waikato; replaced by Therese Ford for one meeting)
  • Mary Chamberlain (Evaluation Associates)
  • Whetu Cormack (NZPF President, principal Bathgate School)
  • Robyn Gibbs (Ministry of Education)
  • Mary Hall (NZSTA)
  • Stuart McNaughton (University of Auckland, Chief Science Advisor Ministry of Education)
  • Frances Nelson (convenor for the Centre of Leadership Excellence at the Education Council principal of Fairburn school)
  • Mark Potter (NZEI Executive, principal Berhampore school)
  • Sally Scott (Ministry of Education)
  • Paula Pope (Ministry of Education); replaced by Vicki Wilde at meeting 3)
  • Jenny Poskitt (Massey University)
  • Ro Parsons (ERO)
  • Claire Sinnema (University of Auckland)
  • Linda Stockham (Education Council)
  • Gary Sweeney (NZAIMS, principal Pukekohe Intermediate)
  • Denise Torrey (principal Somerfield school)
  • Melanie Webber (PPTA Vice President)
  • Mike Williams (SPANZ Vice President and President from 2017, principal Pakuranga College)
  • Howard Youngs (AUT)

38 schools trialled the Teaching and School Practices tool

38 schools took part in the trial, which ran during March 2017. They gave a sound cross-section of school types and socio-economic decile:

Contributing Full Primary Intermediate Secondary
(Years 7-15)
Secondary
(Years 9-15)
15 9 5 3 6

Decile 1 2 3 4 5 6 7 8 9 10
Number of Schools 5 4 3 1 7 2 6 3 4 3

440 teachers (165 primary, 66 intermediate and 209 secondary), 17 non-teaching school leaders (a total of 457 teachers) took part in the survey, and 36 principals.

Feedback improved the Teaching and School Practices tool

We asked for feedback at the end of each set of questions (by domain), particularly around any questions that did not make sense or where the wording could be improved. We also asked how long the survey had taken and for overall comments at the end. This feedback identified some questions that were not sufficiently clear. This reinforced the need to keep the survey as short as we could — while covering the ground and providing useful specific information. It gave us some assurance that the questions were seen as useful. We got some very positive feedback from principals on the insight they gained from the reports for their school, and their usefulness.

Psychometric analysis gives assurance

Psychometric analysis came next. We did this for three reasons:

  • To ensure that the questions we asked in each domain were conceptually coherent, and to check that each domain had a strong association with other domains. The analysis showed that we are able to construct robust scales using combined domains.
  • To see whether each domain had a range of questions in terms of how they were scored (in technical terms the ‘difficulty’ level). We wanted respondents to have information about practices that were relatively common, and also practices that were relatively uncommon.
  • To check whether the response scales for the questions gave us enough differentiation.

Shortening and strengthening the surveys

The NZCER team used the feedback from trial participants and the psychometric analysis to shorten the surveys. We still include a range of questions for each domain. The psychometric analysis also highlighted the need to change the response scale for the Teaching Practices survey, to provide more differentiation.

We took our shortened version to the Advisory Group for their review, to ensure that the survey would be clear and meet people’s needs. Some further refinements were made.

We then checked that the final set of questions for each domain had conceptual coherence. The psychometric analysis gave us assurance that the survey we provide here gives a valid picture for individual schools, for Kāhui Ako, and at the national level.

We’ll continue to develop the Teaching and School Practices survey tool

Wide use of the Teaching and School Practices survey tool in Term 2 and 3 2017 will allow us to do two things:

  • Provide a nationally representative picture of teaching and school practices and principal leadership, including looking at any differences related to school type and context
  • Develop a national benchmark so that from 2018 schools can compare the results for their school with a common standard.

We will work with our Advisory Group to ensure the usefulness of the national picture and benchmarks.

We will also make any improvements to the tool or the website that are indicated by the first year’s use.

We welcome any comments or thoughts you have about the Teaching and School Practices tool. Please let us know how you have used it also! Contact: tspsurveys@nzcer.org.nz

The NZCER development team

  • Cathy Wylie: responsible for the School Practices and Principal Leadership sections (Cathy originally co-developed the Educational Leadership Practices survey with Viviane Robinson)
  • Sue McDowall and Jenny Whatman: responsible for the Teaching Practices section
  • Hilary Ferral: responsible for psychometric analysis and design of reports back to principals (she previously undertook the psychometric analysis and reporting design work for the Educational Leadership Practices survey)
  • Heleen Visser: team leader, also responsible for leading the work with the on-line team
  • Rachel Felgate: responsible for statistical analysis
  • Our on-line team: Paul Shih, Chip Pison, Matt Nichols
  • Our school services team: Leah Saunders, Ben Gardiner

Any questions?

Please contact us at tspsurveys@nzcer.org.nz