By Janus Boye
Caroline Jarrett is an expert on forms and surveys. Back in 2011 she was a speaker at the Boye Aarhus 11 conference. At the conference, she shared Design tips for complex forms (PDF) and also ran a popular workshop on surveys in practice.
If you want to validate an idea or wondering how something works, it is almost always a good idea to ask the customer, but response rates are generally down on all surveys. Bad surveys can both stress your customers and alienate them.
In her recent book 'Surveys That Work', UK-based forms expert Caroline Jarrett explains a seven-step process for designing, running, and reporting on a survey that gets accurate results. In a no-nonsense style with plenty of examples about real-world compromises, the book focuses on reducing the errors that make up Total Survey Error—a key concept in survey methodology.
Caroline is a self-proclaimed survey sceptic and recently hosted a member conference call on her new book. Below you’ll find my notes from the call, the slides, recording and links to further reading on the topic.
What is a survey really?
In a world gone hybrid, Caroline caught our attention by opening the call with a clever question:
What would you do for a dollar?
It turns out that to most people one dollar now included with a mailed paper questionnaire is better than ten dollars promised later when the questionnaire is returned. Social exchange theory suggests that people do quick cost-benefit analysis to determine risks and benefits of their activities, and the same goes for filling out a survey.
As shown in the diagram below, Caroline made the point that people will only respond if they trust you. After that your response rates will depend on their perception of the effort required, and the potential rewards.
Diagram shared by Caroline in the call - Survey responses depend on effort, reward and trust.
At the Boye conferences, we’ve used small paper-based feedback forms on each sessions for the past year. These are filled out immediately after each session and then handed in to a member of staff. For each filled out form, we then donate €1 to a charity. We did one year try emailing the survey afterwards and both the response rate and quality of responses dropped.
A filled out evaluation form from a past Boye conference. Note the charity donation
The charity donation is an increased reward for each respondent, and giving them the survey right there takes away the effort of remembering to answer it later. So for us, this trust/reward/effort triangle worked in practice.
So what is a survey, really? Caroline shared this textbook definition from the book Survey methodology, which provides survey methodology topics and technique:
The survey is a
systematic method
for gathering information from
(a sample of) entities
for the purpose of
constructing quantitative descriptors
of the attributes of the larger population
of which the entities are members.
Based on work and experience, Caroline changed this a slight bit to focus more on process and decisions, so this is definition of a survey that she uses:
The survey is a
process
of asking questions
that are answered by
(a sample of) a defined group of people
to get numbers
that you can use to make decisions
Let’s move on to look closer at what really interests Caroline and one of keys to any survey: The questions.
It’s all about the questions
Caroline has worked with forms for 30 years and co-authored Forms That Work in 2008 together with Gerry Gaffney. As Caroline said, working with forms is really about questions and she turned to the survey literature to learn more, especially about questions.
When you are working on a survey, Caroline’s advice is to start with this key question:
Why are we doing it?
If you have a good answer, then move onto these two key questions:
What do you want to ask?
Who do you want to ask?
Caroline came up with the Survey Octopus as shown below to illustrate this and to also show how all the issues are connected.
The New Yorker Cartoons shared this one on Instagram earlier this month: If you like cartoons and surveys—and cartoons about surveys—then oh boy, do we have news for you: we have a survey *about cartoons*.
The discussion reminded me of a cartoon that I enjoyed in the New Yorker.
Think about errors across the entire survey process
Unfortunately, many surveys have errors, but as Caroline said in our call: Errors happen, it’s all about minimising them to get the best possible survey. Or to quote her:
To minimise Total Survey Error, do many things well
Errors can happen all around the octopus, including measurement errors, lack of validity, sampling errors and non-response errors.
Caroline showed us a linear process for surveys, as in the slide below.
She specifically pointed out that cleaning and analysing the data is often underestimated as this can take as much time as the other parts. To illustrate her point, Caroline used to familiar example of this brief survey question
Would you recommend us to a friend or family member?
To quote her:
If you're buying a baby carriage, then the 'recommend to a friend' question would probably be obvious, interesting and appropriate. If you're in hospital having a miscarriage, it would be cruelly inappropriate.
Always test your questions by interviewing in context.
As a part of the Q&A in the call, we also spoke about this tweet by Deane Barker:
Avoiding any comments on Brexit, she highly praised the work of Erika Hall (@mulegirl) who wrote the book Just Enough Research.
Learn more about surveys that work
The book came out in July 2021 and was published by our friends at Rosenfeld Media. Have a look at this helpful excerpt, to find out whether the book is something for you.
Effortmark is Caroline Jarrett's consultancy business. On the Effortmark website, you’ll find much more about better forms and surveys. Caroline is also the co-author of Forms that Work: Designing Web Forms for Usability and of User Interface Design and Evaluation.
Finally, you can also download the slides (PPT) or even lean back and enjoy the recording from the call below.