Survey Question Bank

SQB events

Details will appear here.

Qb events

10 April 2008 - ESRC Question Bank one day conference

Survey Research conference organised by the University of Surrey:

‘Survey Measurement: Assessing the Reliability and Validity of Contemporary Questionnaire Items’

The Royal Statistical Society, Errol Street, London

This one day conference was the second in the series and brought together international experts who considered issues in survey measurement, in particular assessment of the validity and reliability of contemporary questionnaire items.

Chair: Professor Martin Bulmer, University of Surrey

Speakers: Dr Janet Harkness (University of Nebraska-Lincoln and ZUMA, Germany), Dr Pamela Campanelli (Independent Survey Consultant), Dr Edith de Leeuw (Utrecht University), Dr Patrick Sturgis (University of Surrey)

Click here to see the Programme

Presentations and Handouts:

Introduction: Professor Martin Bulmer

Powerpoint slides

Question design and question translation in multilingual survey research: Dr Janet Harkness

Since the Second World War, a considerable body of literature has amassed on aspects of translation production or translation assessment in survey research. Some researchers have seen translation as a key to comparability across populations investigated, others as an insurmountable problem best avoided whenever possible. One response has thus been to develop detailed translation guidelines (cf. guidelines in the European Social Survey and the US Bureau of Census) and establish procedures for translation assessment. Another has been to reduce translation needs altogether or, alternatively, to simplify tricky areas for translation such as answer scales by reducing answer formats to dichotomous choices (as in the Gallup World Poll). This talk takes a third perspective. When surveys are designed for multi population studies, translation, I suggest, is best seen and treated as a matter of design. The presentation therefore illustrates the close relationship between instrument design, instrument adaptation and instrument translation. In many instances, however, a source questionnaire may already be “written in stone” before translation is ever considered. Thus theoretical underpinnings and strategies for other language version production in this context are also considered.

Powerpoint slides

Field experiments for assessing question validity: Dr Patrick Sturgis

Perhaps the most common way of evaluating the validity of a survey question before a main-stage survey is to subject it to some form of small-scale cognitive testing. While such techniques are of great value in diagnosing potential problems relating to respondent understanding and interpretation, they are less useful for determining the validity of alternate forms. That is to say, when a problem is diagnosed and alterations made to a draft question, we generally have no way of knowing whether the modified version of the item yields more accurate estimates of the concept we seek to measure. In this paper, I shall advance the case for greater use of field experiments in questionnaire design. I shall illustrate my argument with a number of examples drawn from research undertaken in collaboration with Ipsos-MORI.

Powerpoint slides

Question design and measurement in mixed mode research: Dr Edith de Leeuw

Data collection in surveys can be carried out in several modes, such as face-to-face, telephone, self-administered mail questionnaires, and web surveys. With all these possibilities, the choice for a specific mode is difficult, and involves trade-offs between the strong and weak points of each mode. A new approach is to use multiple modes and combine all strong points in one single survey, which leads to a multi-mode or mixed mode survey. Mixed mode surveys are attractive because one can attempt to combine the strong points thereby compensating for the weaker points of the different modes. However, such a decision should not be made without careful thought and planning. Introducing a second, or even a third or fourth survey mode into the data collection plan, implies a more complicated, more expensive, longer, and more challenging survey implementation. Mixing modes also raises the question of data integrity: since a specific mode can have an effect on the responses that are obtained, mixing modes involves the possibility of confounding mode effects with substantive effects. When different modes are used, it is important that in the design phase of the survey steps are taken to minimize the impact of mode effects. This paper introduces the advantages and disadvantages of mixing modes and principles of measurement, and reviews strategies for questionnaire design, with the emphasis on strategies that can be used to minimize mode effects.

Powerpoint slides
Handout (opens in PDF)

What question testing methods can and can’t tell us: Dr Pamela Campanelli

The introductory part of this presentation focuses on differences between the testing used for single survey questions versus standardised multi-item scales, the pros and cons of this testing, and how validity and reliability are considered in each case. The remainder of the presentation discusses the range of “newer” methods currently available to test survey questions for both interview and self-completion modes (including expert review, systematic forms appraisal, interviewer rating form, cognitive interviewing, respondent debriefing, behaviour coding, and focus groups for question testing). What can and can’t these methods tell us? More importantly, where should future research into “question testing” take us?

Powerpoint slides with 3 page table (opens in PDF)