Recommended Practices for Clearer, Smarter Surveys

The following list is not required for your survey to pass the review process, but it is strongly recommended for creating high-quality surveys in Iowa. These best practices were developed in collaboration with a Qualtrics XM Scientist and Qualtrics Technical Success Manager Hunter Markus. 

An XM Scientist at Qualtrics is a specialized consultant and strategist who helps organizations design and optimize their Experience Management - XM programs such as Customer Experience - CX, Digital Experience - DX, and Employee Experience - EX.

Survey Methodology

Several of the recommendations below, particularly scale ordering, recode values, and letting respondents skip non-essential questions, come from an internal CX training session on survey methodology. For the full context, including how surveys and forms differ in practice, see:

Qualtrics CX Survey Methodology Video

Survey Structure and Flow

• Use a general-to-specific flow: start with overall questions, then drill into specifics and attributes.
• Display one question per page unless the questions are strongly related or it is a rating question followed by an open-text question on the same topic. This serves two main purposes:
  1. It ensures respondents give full consideration to one question at a time.
  2. It reduces respondent scrolling, which makes the survey easier and quicker, especially on mobile devices.
• Limit survey length to 1-3 minutes, which is 4-15 closed-end questions with 1-2 open-text questions. This doesn't apply to forms with required data collection.
• Limit to three open-text questions in a survey unless it is a form or qualitative methodology.
• Include a customized end of survey message that applies to the survey/form they filled out. Include any follow-up instructions or expectations, etc.

Question Design

• Use common, simple language that is easy to understand. Follow guidelines from the Plain Writing Act:
  • Avoid jargon, acronyms, or technical language. If technical terms must be used, define them clearly.
  • Avoid questions with negations or double negatives that can confuse respondents - How much do you agree or disagree that our website is not easy to use?
  • Use personal pronouns such as you.
  • Aim for 5th-8th-grade reading level in your question construction.
• Avoid double-barreled questions - Please rate the employee's knowledge and friendliness.
• Avoid leading questions - How satisfied are you with our excellent customer service?
• Include Prefer not to answer for sensitive questions for example income, race or ethnicity, drug use, and so on, unless the answer is necessary to achieve the objectives of the survey.
• If a Qualtrics suggested wording is offered, use it. 
  • If a standard index already measures what you're asking about, for example the OECD Better Life Index, Health-Related Quality of Life Index, or Employee Engagement Index, use its wording instead of writing your own. See the Rating Scales Best Practices page for scale wording references.

Scale and Response Format

• The most important aspect of survey design is maintaining consistency within a survey or across waves of a survey. Do not use different number of scale points, direction of scales, etc. within the same survey.
• Use fully labeled scales when possible, with an odd number of scale points - typically 5, though 3 or 7 are also appropriate). For example, Very satisfied, Somewhat satisfied, Neutral, Somewhat dissatisfied, Very dissatisfied. An even number of points removes the neutral midpoint and forces a directional answer, so confirm that's intentional before using one.
• Avoid numeric-only labels and partially labeled scales unless you are using a scale that has been used on past waves of the current survey OR a standard question and scale that has been created to standardize measurement of a concept. For example, OECD Better Life Index, Health-Related Quality of Life Index, Employee Engagement Index, and so on.
• Use vertical scale orientation so the rating scales appear the same on desktop and mobile devices. Vertically oriented scales provide a superior survey experience on mobile devices and avoid potential bias caused by side-to-side scrolling. To remove potential bias caused by the survey mode, desktop, laptop vs. mobile device, it is ideal to use vertical scales all the time.
• When using a scale tied to an existing scale structure, default to negative-to-positive scale ordering: Poor-to-Excellent. Qualtrics's automatic Recode Values logic assumes the negative end of the scale is displayed first, leftmost on a horizontal scale, topmost on a vertical scale. Keeping this order means your recoded values, and any trend reporting built on them, stay accurate. See Recode Values on the Rating Scales Best Practices page for the full explanation, including how to handle a Don't Know/N/A option if one is included.
• Avoid matrix/grid questions whenever possible. There is evidence that a matrix question can lead to less consideration of the individual questions and more straight lining, where the respondent selects the same response options for all questions in the matrix. On mobile devices, Qualtrics displays matrix questions as an accordion, showing one row at a time, which reduces but doesn't eliminate this risk. A single rating question per page is still preferred where feasible.
• Randomize categorical response options to reduce primacy bias - Select your reason for visiting this website. Anchor any Other options to the bottom of the list.

Survey Formatting and Accessibility

• Use radio buttons instead of dropdowns when possible.
• If the survey is over 30 questions, consider adding a progress bar. Otherwise, don't use a progress bar.
• Avoid requiring responses to questions unless the data is necessary for survey logic or to achieve the survey objectives.
• Avoid requiring responses to questions unless the data is necessary for survey logic or to achieve the survey objectives. This guidance applies mainly to surveys. Use Request Response, a soft prompt, or display logic instead of Force Response where you can. Forms are different. Since a form is typically processing an application or transaction, requiring every field is often appropriate, as an incomplete form usually can't be acted on.