Recommended Practices for Clearer, Smarter Surveys

The following list is not required for your survey to to pass the review process, but it is strongly recommended for creating high-quality surveys in Iowa. These best practices were developed in collaboration with a Qualtrics XM Scientist.

An XM Scientist at Qualtrics is a specialized consultant and strategist who helps organizations design and optimize their Experience Management (XM) programs such as Customer Experience (CX), Digital Experience (DX), and Employee Experience (EX). 

Survey Structure and Flow

• Use a general-to-specific flow: start with overall questions, then drill into specifics and attributes.
• Display one question per page unless the questions are strongly related or it is a rating question followed by an open-text question on the same topic. This serves two main purposes:
  1. It ensures respondents give full consideration to one question at a time.
  2. It reduces respondent scrolling, which makes the survey easier and quicker, especially on mobile devices.
• Limit survey length to 1-3 minutes, which is 4-15 closed-end questions with 1-2 open-text questions. This doesn’t apply to forms with required data collection.
• Limit to three open-text questions in a survey unless it is a form or qualitative methodology.
• Include a customized end of survey message that applies to the survey/form they filled out. Include any follow-up instructions or expectations, etc.

Question Design

• Use common, simple language that is easy to understand. Follow guidelines from the Plain Writing Act:
  • Avoid jargon, acronyms, or technical language. If technical terms must be used, define them clearly.
  • Avoid questions with negations or double negatives that can confuse respondents ('How much do you agree or disagree that our website is not easy to use?')
  • Use personal pronouns such as 'you.'
  • Aim for an 8th-grade reading level in your question construction.
• Avoid double-barreled questions ('Please rate the employee's knowledge and friendliness.')
• Avoid leading questions ('How satisfied are you with our excellent customer service?')
• Include 'Prefer not to answer' for sensitive questions (e.g., Income, Race/Ethnicity, Drug use, etc.) unless the answer is necessary to achieve the objectives of the survey.

Scale and Response Format

• The most important aspect of survey design is maintaining consistency within a survey or across waves of a survey. Do not use different number of scale points, direction of scales, etc. within the same survey.
• Use 5-point, fully labeled scales when possible (e.g., Very satisfied, Somewhat satisfied, Neutral, Somewhat dissatisfied, Very dissatisfied).
• Avoid numeric-only labels and partially labeled scales unless you are using a scale that has been used on past waves of the current survey OR a standard question and scale that has been created to standardize measurement of a concept (e.g., OECD Better Life Index, Health-Related Quality of Life Index, Employee Engagement Index, etc.).
• Use vertical scale orientation so the rating scales appear the same on desktop and mobile devices. Vertically oriented scales provide a superior survey experience on mobile devices and avoid potential bias caused by side-to-side scrolling. To remove potential bias caused by the survey mode (desktop/laptop vs. mobile device), it is ideal to use vertical scales all the time.
• When using vertically oriented scales and tied to an existing scale structure, default to a positive-to-negative scale ordering (Excellent-to-Poor).
• Avoid matrix/grid questions whenever possible. There is evidence that a matrix question can lead to less consideration of the individual questions and more straight lining (where the respondent selects the same response options for all questions in the matrix).
• Randomize categorical response options to reduce primacy bias ('Select your reason for visiting this website'). Anchor any 'Other' options to the bottom of the list.

Survey Formatting and Accessibility

• Use radio buttons instead of dropdowns when possible.
• If the survey is over 30 questions, consider adding a progress bar. Otherwise, don’t use a progress bar.
• Avoid requiring responses to questions unless the data is necessary for survey logic or to achieve the survey objectives.