Recommended Practices for Clearer, Smarter Surveys

The following list is not required for your survey to to pass the review process, but it is strongly recommended for creating high-quality survey in Iowa. These best practices were developed in collaboration with a Qualtrics XM Scientist.

An XM Scientist at Qualtrics is a specialized consultant and strategist who helps organizations design and optimize their Experience Management (XM) programs—such as Customer Experience (CX), Digital Experience (DX), and Employee Experience (EX). 

Survey Structure and Flow

• Use a general-to-specific flow: start with overall questions, then drill into specifics and attributes.
• Display 1 question per page unless the questions are strongly related or it is a rating question followed by an open-text question on the same topic. This serves two main purposes: 1) It ensures respondents give full consideration to one question at a time; 2) It reduces respondent scrolling, which makes the survey easier and quicker, especially on mobile devices.
• Limit survey length to 1-3 minutes, which is 4-15 closed-end questions with 1-2 open-text questions. Doesn’t apply to forms with required data collection.
• Limit to 3 open-text questions in a survey unless it is a form or qualitative methodology.
• Include a customized end of survey message that applies to the survey/form they filled out. Include any follow-up instructions or expectations, etc.)

Question Design

• Use common, simple language that is easy to understand. Follow guidelines from the Plain Writing Act:
  • Avoid jargon, acronyms, or technical language. If technical terms must be used, define them clearly.
  • Avoid questions with negations or double-negatives that can confuse respondents (e.g., "How much do you agree or disagree that our website is not easy to use?")
  • Use personal pronouns such as “you”
  • Aim for an 8th-grade reading level in your question construction.
• Avoid double-barreled questions (e.g., 'Please rate the employees knowledge and friendliness.').
• Avoid leading questions (e.g., "How satisfied are you with our excellent customer service?)"
• Include 'Prefer not to answer' for sensitive questions (e.g., Income, Race/Ethnicity, Drug use, etc.) unless the answer is necessary to achieve the objectives of the survey.

Scale and Response Format

• The most important aspect of survey design is maintaining consistency within a survey or across waves of a survey. Do not use different number of scale points, direction of scales, etc. within the same survey.
• Use 5-point, fully-labeled scales when possible (e.g., Very satisfied, Somewhat satisfied, Neutral, Somewhat dissatisfied, Very dissatisfied).
• Avoid numeric-only labels and partially labeled scales unless you are using a scale that has been used on past waves of the current survey OR a standard question and scale that has been created to standardize measurement of a concept (e.g., OECD Better Life Index, Health-Related Quality of Life index, Employee Engagement Index, etc.).
• Use vertical scale orientation so the rating scales appear the same on desktop and mobile devices. Vertically oriented scales provide a superior survey experience on mobile devices and avoid potential bias caused by side-to-side scrolling. To remove potential bias caused by the survey mode (desktop/laptop vs. mobile device), it is ideal to use vertical scales all the time.
• When using vertically-oriented scales and tied to an existing scale structure, default to a positive-to-negative scale ordering (Excellent-to-Poor).
• Avoid matrix/grid questions whenever possible. There is evidence that a matrix question can lead to less consideration of the individual questions and more straightlining (where the respondent selects the same response options for all questions in the matrix).
• Randomize categorical response options to reduce primacy bias (e.g., “Select your reason for visiting this website”). Anchor any “Other” options to the bottom of the list.

Survey Formatting and Accessibility

• Use radio buttons instead of dropdowns when possible.
• If the survey is over 30 questions, consider adding a progress bar. Otherwise, don’t use a progress bar.
• Avoid requiring responses to questions unless the data is necessary for survey logic or to achieve the survey objectives.