Search the docs, videos, blog, apps, teams, skills and release notes.
At a glance
What it is
Produces a complete survey with question set in recommended order, question type selection (Likert, multiple choice, open text, matrix) with rationale, response scale definitions, and an analysis plan for each question.
A skill is a written procedure a bot loads when a job calls for it. This one is a single file, SKILL.md, and the whole file is on this page.
Each of these library teams gives its bots this skill. Import one and look inside first.
The whole file, exactly as it downloads. Read it before you rely on it.
SKILL.md
547 lines · 7,542 words
Use this skill when:
A user asks to build a customer satisfaction, employee engagement, product feedback, or post-event survey from scratch
A user wants to collect structured quantitative or mixed-method data from a defined respondent population and needs question type selection, scale design, and an analysis plan
Produces a complete survey with question set in recommended order, question type selection (Likert, multiple choice, open text, matrix) with rationale, response scale definitions, and an analysis plan for each question.
Nothing. It ships with Murage, which is open source. Your bots run on the AI plan you already have.
Yes. Open it in Settings → Skills to read, edit or duplicate it. Skill Guard scans it again after every edit.
Download the app, connect the AI you already pay for, and tell your Chief of Staff what needs doing. Plan on about ten minutes from install to a working team.
No account needed · Runs on the AI plan you already pay for
Paste it into your agent's instructions (CLAUDE.md, AGENTS.md or a custom GPT), or add it to a bot in Murage.
Ask Ember for the bot, then switch the skill on in its settings.
Step 1
Send your Chief of Staff this message. It proposes the bot and its role first, and creates it when you say go.
Send this to Ember
“Create a bot that uses the Survey Design skill, and show me the plan first.”
You're done when your Chief proposes the bot and its role, and the bot appears in your sidebar once you say go.
Step 2
You're done when you see the bot’s skills and the library.
Step 3
You're done when the skill shows as on for that bot.
Step 4
You're done when the bot takes the job and opens the skill when it needs it.
Prefer to do it yourself? Choose + in the sidebar, then New Bot.Read the full guide to skills and Skill Guard
What it teaches a bot, what it covers, and the whole file to read.
What it covers
When to Use
Process
Output Format
Rules
Edge Cases
Example
Survey: New Onboarding Experience Assessment -- [Q1 2026]
A user has a business or research decision to make and needs to know what to ask, how to ask it, and what to do with the answers
A user already has a list of questions they drafted and wants expert review of question type, ordering, scale balance, wording bias, and skip logic
A user is building a recurring pulse survey (weekly, monthly, quarterly) and needs a minimal-question set with longitudinal tracking methodology
A user needs to segment responses by audience attributes (tenure, role, geography) and must build that segmentation into the survey structure before launch
Do NOT use when:
The user wants to analyze qualitative interview transcripts or code themes from existing interview notes -- use qualitative-coding instead
The user wants to run a controlled experiment to compare two versions of something -- use ab-test-design instead, since surveys cannot establish causality
The user is researching a market for sizing, competitor mapping, or trend intelligence -- use competitive-intelligence instead
The user has already collected open-ended survey responses at scale and wants automated theme extraction -- use text-mining-protocol instead
The user needs a discussion guide for 1:1 moderated user research sessions -- surveys are the wrong instrument for that context
The user wants to conduct observational research, diary studies, or ethnographic fieldwork -- those require entirely different research designs
The respondent population is fewer than 10 people -- at that size, conduct structured interviews instead; scaled questions will produce unreliable statistics
Before writing a single question, clarify what decision this survey must inform and what would count as a successful outcome. This prevents the most common survey failure: collecting data that cannot be acted upon.
Ask the user: "What will you do differently depending on what the survey results show?" If the user cannot articulate at least two different actions they might take, the survey is not ready to be designed.
Identify the decision deadline -- if a product roadmap decision must be made in three weeks, the survey must launch immediately with a 7-10 day collection window, not a 30-day window.
Confirm whether this survey will be compared to past data (benchmark comparison) or stands alone (baseline measurement). Benchmark comparisons require using identical question wording and scales from prior surveys.
Determine whether statistical significance is a requirement. If the user needs to make claims like "Feature A is preferred by 70% of customers," they need to understand margin of error. For a population of 2,000 with a target margin of ±5% at 95% confidence, the required sample is approximately 322 responses. Use the formula n = (Z² × p × (1-p)) / e² where Z=1.96, p=0.5 (worst case), e=0.05.
Establish the primary metric the survey will produce -- a single "headline number" that summarizes the survey's central finding (overall CSAT mean, NPS score, engagement index score, etc.).
Research questions (RQs) are the analytical questions the survey answers -- they are NOT survey questions themselves. Every survey question exists only to answer one or more RQs.
Write 3-7 RQs before writing any survey questions. Fewer than 3 suggests the survey is too narrow; more than 7 suggests it should be split into two separate surveys.
RQs must be answerable with data -- "Why do customers churn?" is not a valid RQ because surveys cannot determine causation. Reframe as "What factors are most strongly correlated with customers who rate their likelihood to renew as 1-3 out of 5?"
Classify each RQ by analysis type: descriptive (what is the distribution?), comparative (does Group A differ from Group B?), or relational (do two variables correlate?). Each type requires a different question design.
Prioritize RQs as P1 (must answer -- survey fails without this), P2 (important but survey still valid without it), or P3 (nice to have). This prioritization drives question ordering and cut decisions if the survey becomes too long.
For each RQ, write out what the output will look like: "RQ1 will produce a mean satisfaction score that we will compare against our Q3 benchmark of 3.9/5.0." This forces clarity before any questions are written.
The sequence of questions is as important as the questions themselves. Cognitive load, primacy effects, and social desirability all interact with question order.
Opening section (1-3 questions): Begin with easy, non-threatening, behaviorally factual questions (how long have you used the product, how often do you use feature X). Never open with opinion or satisfaction questions -- respondents need 1-2 questions to orient themselves. Never open with demographics, which can feel intrusive.
Core section (5-15 questions): Place P1 research questions early. Use funnel ordering: start with broad/general questions and narrow to specific topics. Ask about overall satisfaction before asking about specific attributes -- if specific questions come first, they anchor the overall rating (anchoring bias).
Sensitive section (if needed): Net Promoter, compensation/budget questions, or workplace complaint topics go in the second half after the respondent is engaged. Sensitive questions placed in the first half increase abandonment by 20-40%.
Closing section (1-2 questions): One open-ended "anything else?" question and one optional demographic question. Demographic questions at the end have better response rates than at the beginning.
Apply the "newspaper test" to the flow: read the questions in order as a respondent would experience them. Does each question feel like a natural follow-up to the previous one?
Total question budget: Volunteer respondents (customers, general public) drop completion rates by approximately 5-10% per question beyond 10 questions. Incentivized respondents (paid panel, captive employee survey) tolerate up to 20 questions. Academic research surveys with highly motivated participants can go to 30-40 questions but must be disclosed upfront.
Each question type has specific use cases, analytical affordances, and known pitfalls. Match the question type to what the data needs to do, not to what feels easiest to write.
| Data Need | Question Type | Analytical Output | Key Constraint |
|---|---|---|---|
| Measure attitude strength | 5-point Likert | Mean, median, frequency distribution | Use when you need parametric statistics and have ≥30 responses |
| Measure subtle attitude differences | 7-point Likert | More sensitive mean comparison | Use when comparing subgroups or when scale sensitivity is critical |
| Standard loyalty measurement | NPS (0-10 numeric) | Promoter%, Detractor%, NPS score | Always use 11-point (0-10), never 10-point (1-10), to match industry benchmarks |
| Effort/ease measurement | CES (1-7 or 1-5) | Mean CES; lower is better (less effort) | Use "The company made it easy for me to..." phrasing, not "How easy was it?" |
| Categorical exclusive choice | Single-choice / radio | Frequency distribution, mode | Options must be mutually exclusive and exhaustive (include "Other") |
| Categorical multiple response | Multi-select / checkbox | Frequency per option, combination patterns | Report as "% of respondents who selected" not "% of responses" |
| Relative priority | Forced ranking | Average rank per item, rank order | Limit to 5-7 items; beyond 7, respondents rank randomly |
| Rate multiple attributes | Matrix / grid | Mean per row, attribute ranking | Limit to 5-7 rows; matrix questions on mobile are notorious for abandonment |
| Exact quantity | Numeric input | Mean, median, range | Use when you genuinely need a number, not an approximation |
| Exploratory insight | Open text | Thematic coding, word frequency | Maximum 2-3 per survey; always make them optional for non-critical surveys |
| Nuanced continuous attitude | Semantic differential / slider | Mean on a polar scale | Use for research where directionality matters (e.g., simple vs. complex) |
Key rules for question type selection:
Never use a matrix for questions that have different sub-topic scales -- all rows in a matrix must use the same scale to be valid.
Never use ranking when the goal is to understand absolute satisfaction -- a respondent can rank support #1 but still rate it as poor. Use rating first, ranking only for relative comparison.
Multi-select questions cannot produce mean scores -- they can only produce frequency distributions. Never analyze multi-select results with means.
Question wording is where most surveys fail. Apply these specific writing disciplines to every question:
Single-concept rule: Each question must ask about exactly one thing. "Was the support team fast and knowledgeable?" is invalid -- speed and knowledge are independent dimensions. If both matter, write two questions.
8th-grade readability rule: Use the Flesch-Kincaid readability standard. Sentences under 20 words. No industry jargon (say "customer support" not "post-sale success engagement"). Spell out acronyms.
Neutral framing rule: Avoid loading words. "How useful did you find our onboarding resources?" is neutral. "How helpful was our comprehensive onboarding documentation?" is loaded with "comprehensive" and "helpful."
Avoid presupposition: "What features do you use?" presupposes the respondent uses features. Reframe as "Which of the following features, if any, have you used in the past 30 days?"
Define the reference period: Questions about satisfaction or behavior must specify a time frame. "How satisfied are you?" is vague. "How satisfied are you with your experience over the past 3 months?" is specific.
Avoid negatives and double negatives: "Do you disagree that the product is not easy to use?" is cognitive torture. Reframe as "How easy is the product to use?"
Social desirability check: For any question where the "right answer" is obvious (e.g., "Do you care about data security?"), consider indirect phrasing or behavioral anchors instead.
For each question, document:
Question text (final wording)
Question type and response options
Which RQ it answers
Analysis method
Skip logic (if any)
Optional vs. required designation
Scales must be labeled at every point, balanced in polarity, and consistent throughout the survey. Mixing scales for similar questions (5-point for some, 7-point for others) creates analysis problems and confuses respondents.
Standard scales with precise wording:
5-point Likert Agreement: Strongly Disagree / Disagree / Neither Agree nor Disagree / Agree / Strongly Agree
7-point Likert Agreement: Strongly Disagree / Disagree / Somewhat Disagree / Neither Agree nor Disagree / Somewhat Agree / Agree / Strongly Agree
5-point Satisfaction: Very Dissatisfied / Dissatisfied / Neither Satisfied nor Dissatisfied / Satisfied / Very Satisfied
5-point Frequency: Never / Rarely / Sometimes / Often / Always (Note: these are ordinal, not interval -- do not compute means without caution)
5-point Importance: Not at All Important / Slightly Important / Moderately Important / Very Important / Extremely Important
NPS (0-10): "Not at all likely" anchored at 0, "Extremely likely" anchored at 10 -- no intermediate labels, just endpoint anchors
CES (1-7): "Strongly Disagree" anchored at 1, "Strongly Agree" anchored at 7 -- statement format: "The company made it easy to handle my issue"
CSAT (1-5 or 1-3): For quick post-interaction measurement, 3-point (Dissatisfied / Neutral / Satisfied) outperforms 5-point in response rate; for detailed tracking, use 5-point
Scale selection rules:
5-point scales: Use when response sensitivity is less important and survey fatigue is a concern
7-point scales: Use when comparing subgroups where small differences matter, when respondents are experts or highly motivated, and in academic or formal research contexts
Never use even-numbered scales (4-point, 6-point) unless you explicitly want to force a directional response (no neutral midpoint). Forced-choice scales increase social desirability bias.
Add "Not Applicable" only when there is a genuine possibility the question does not apply. Overusing N/A creates data gaps.
The analysis plan must be written before the survey launches, not after data collection. Post-hoc analysis decisions introduce confirmation bias.
For every RQ, specify: input questions, statistical method, output metric, and decision rule.
Descriptive analysis (frequency distribution, mean, median): Use for all scaled and categorical questions. Always report means with standard deviation for Likert data. Report medians for highly skewed distributions.
Comparison analysis (cross-tabulation, subgroup means): Requires segmentation variables to be built into the survey. Identify every comparison you want to make (tenure vs. satisfaction, role vs. NPS) and ensure the segmentation question is included. For comparing two group means, use an independent samples t-test. For three or more groups, use one-way ANOVA. For small samples (n<30 per group), report results as directional only.
Correlation analysis (do two variables move together?): Pearson correlation for interval data (Likert means). Spearman correlation for ordinal data. Report r value and significance (p < 0.05 threshold for most business applications).
Thematic coding for open-ended questions: Requires at least two independent coders for research-grade reliability. For business surveys, one coder with a defined category scheme is acceptable. Code to mutually exclusive, collectively exhaustive (MECE) categories. Report inter-rater reliability (Cohen's Kappa) if research-grade results are needed (target κ > 0.70).
NPS calculation: % of respondents scoring 9-10 (Promoters) minus % scoring 0-6 (Detractors). Passives (7-8) are excluded from the calculation. Industry NPS benchmarks vary widely -- SaaS B2B benchmarks are typically NPS 30-50 for leaders. Do not compare NPS across industries.
Decision rules: Every analysis output must have a threshold that triggers an action. "If CSAT falls below 3.5/5.0 for any cohort of 50+ responses, the relevant team lead is notified within 5 business days."
Before the survey goes live, apply these quality gates:
Cognitive interview test: Have 3-5 people from the target audience take the survey out loud (saying what they are thinking as they answer). This reveals misunderstandings invisible to the designer.
Completion time test: Measure actual median completion time on 10 testers. If actual time exceeds stated time by more than 25%, shorten the survey or adjust the time estimate.
Mobile rendering check: Verify that matrix questions, sliders, and open-text fields render correctly on mobile. Over 60% of survey responses now come from mobile devices. Matrix questions on mobile require respondents to scroll horizontally and have 30-50% higher non-response rates.
Skip logic validation: Test every skip logic path to confirm no respondent gets stranded on an inaccessible question or exits the survey prematurely.
Anonymity statement review: If the survey is anonymous, verify that no combination of demographic questions (role + tenure + department + location) could identify an individual respondent. This is especially critical for employee surveys with small teams.
Scale consistency audit: Confirm that all Likert items in the survey use the same number of points. Mixed-scale surveys disorient respondents and create analysis inconsistencies.
## Survey: [Survey Title]
### Survey Overview
- **Purpose:** [Specific decision this survey informs and the action it enables]
- **Target respondent:** [Defined segment -- active customers >30 days, full-time employees, etc.]
- **Distribution channel:** [Email link, in-app intercept, QR code, embedded web, SMS]
- **Expected sample size:** [Target responses / target population] ([Target response rate]%)
- **Minimum viable sample:** [Calculated based on margin of error requirements]
- **Estimated completion time:** [X minutes (based on pre-launch test)]
- **Survey open period:** [Launch date to close date]
- **Primary metric:** [The single headline number this survey produces]
- **Benchmark:** [Prior survey result or industry standard for comparison, if applicable]
### Research Questions
| ID | Research Question | Priority | Analysis Type | Survey Questions |
|----|------------------|----------|--------------|-----------------|
| RQ1 | [Specific, answerable analytical question] | P1 | Descriptive | Q2, Q3 |
| RQ2 | [Specific, answerable analytical question] | P1 | Comparative | Q4, Q5, Q6 |
| RQ3 | [Specific, answerable analytical question] | P2 | Relational | Q7, Q8 |
---
### Survey Questions
#### Section 1: [Section Name] (Opening -- Questions 1-X)
*Purpose: [What this section establishes]*
---
**Q[N]: [Question text]**
- **Type:** [Question type]
- **Required:** [Yes / No]
- **Response options:**
- [Option 1]
- [Option 2]
- [Option 3]
- [Other / Prefer not to say -- if applicable]
- **Answers RQ:** [RQ number(s)]
- **Analysis method:** [Frequency count / Mean and SD / Cross-tab variable / Thematic coding]
- **Decision rule:** [If X then Y]
- **Skip logic:** [If [answer], skip to Q[N]. Otherwise continue. / None]
- **Wording rationale:** [Why this phrasing was chosen -- notes on bias avoided]
---
#### Section 2: [Section Name] (Core -- Questions X-Y)
*Purpose: [What this section measures]*
[Repeat question block structure]
---
#### Section 3: [Section Name] (Closing -- Questions Y-Z)
*Purpose: [Qualitative and demographic capture]*
[Repeat question block structure]
---
### Response Scales Reference
| Scale Name | Point Count | Labels (Low → High) | Used In | Notes |
|-----------|------------|---------------------|---------|-------|
| [Scale name, e.g., Satisfaction] | 5 | Very Dissatisfied → Very Satisfied | Q2, Q4, Q7 | Interval-treated for mean analysis |
| [NPS] | 11 (0-10) | Not at all likely → Extremely likely | Q9 | Promoters=9-10, Passives=7-8, Detractors=0-6 |
---
### Analysis Plan
| RQ | Input Questions | Statistical Method | Output Metric | Decision Rule | Owner |
|----|----------------|-------------------|---------------|---------------|-------|
| RQ1 | Q2, Q3 | Mean + SD vs benchmark; t-test | Mean CSAT score (1-5) | If mean < [threshold], escalate to [team] | [Role] |
| RQ2 | Q4, Q5, Q6 | Cross-tab by Q1 segment; one-way ANOVA | % distribution by group | If [group] scores differ by >0.5 points, investigate | [Role] |
| RQ3 | Q7, Q8 | Pearson r; thematic coding | Correlation coefficient + top themes | If r > 0.4, theme qualifies as a driver | [Role] |
---
### Survey Administration Notes
**Pre-launch checklist:**
- [ ] Cognitive interview completed with 3-5 target respondents
- [ ] Completion time verified under [X] minutes
- [ ] Mobile rendering tested on iOS and Android
- [ ] All skip logic paths tested end-to-end
- [ ] Anonymity posture confirmed and stated in survey intro
**Anonymity statement (to include in survey intro):**
"[Your responses are completely anonymous / Your responses are confidential to the HR team but not anonymous]. Individual responses will not be shared with [relevant party]. Results will be reported in aggregate only."
**Distribution schedule:**
- Launch: [Date]
- Reminder 1: [Date -- typically Day 5 for a 14-day window]
- Reminder 2: [Date -- typically Day 11]
- Close: [Date]
**Minimum response check:** If fewer than [minimum viable n] responses are received by Day 7, expand distribution to [additional channel or population].Every question must map to at least one research question. If a question cannot be linked to a specific RQ, cut it. Unmapped questions increase survey length, reduce completion rates, and produce data no one uses. The most common version of this failure is demographic questions collected "just in case" with no analysis plan.
Scaled questions must use consistent point counts throughout the survey. A survey that mixes 5-point and 7-point Likert scales within the same section is analytically invalid. Respondents mentally recalibrate at each scale change, making cross-question comparisons unreliable. Choose one scale and apply it to all similar questions.
Never use leading or loaded language. Words like "excellent," "innovative," "helpful," "comprehensive," or "convenient" presuppose a positive evaluation before the respondent answers. Use neutral verbs: rate, describe, evaluate, indicate, select.
Overall satisfaction must come before attribute-level satisfaction. Asking "Rate our support, reliability, and onboarding" before "Overall, how satisfied are you?" causes contamination -- respondents anchor their overall rating on the specific attributes just evaluated, suppressing variance in the overall score.
Open-ended questions may not exceed three per survey for general respondent populations. Each open-ended question adds 2-4 minutes to completion time and requires qualitative analysis capacity to process. If more qualitative data is needed, design a follow-up interview study, not a longer survey.
The minimum viable sample size must be calculated before the survey launches. For a population of 500, a margin of error of ±5% at 95% confidence requires 217 responses. For a population of 10,000, the same criteria requires 370 responses. Surveys that close with fewer responses than the minimum viable sample must be reported with explicit confidence limitations.
Matrix questions (grid format) must be limited to 5-7 rows maximum. Beyond 7 rows, respondents engage in "straight-lining" (selecting the same column for every row) to complete the question quickly. Straight-lining is a form of response bias that inflates consistency and suppresses meaningful variance. Detect straight-lining during data cleaning by flagging responses with zero variance across all matrix rows.
"Not Applicable" must not be added by default -- it must be justified. N/A options create missing data. Use N/A only when a meaningful proportion of respondents (estimate > 15%) genuinely cannot answer the question. For all other questions, the absence of an N/A forces the respondent to engage with the question or skip it consciously.
Frequency scales (Never, Rarely, Sometimes, Often, Always) are ordinal, not interval. Never compute means on frequency scales and present them as precise measurements. Medians and frequency distributions are the correct outputs. If interval-level analysis is needed, replace frequency scales with numeric frequency questions ("How many times in the past 30 days did you...?").
The analysis plan must be written in pre-registration form before launch. Specify every comparison, threshold, and decision rule before any data is collected. Post-hoc analysis -- deciding what to compare after seeing the data -- introduces p-hacking risk and confirmation bias. This is equally important in commercial surveys as in academic research.
NPS must always use the standard 0-10 scale with specific Promoter/Passive/Detractor cut points. Never use a 1-10 scale or a 1-5 scale for NPS. The 0-6 Detractor / 7-8 Passive / 9-10 Promoter segmentation is fixed by the standard and cannot be adjusted without making results incomparable to any benchmark.
Employee surveys require mandatory anonymity architecture review. If a team or department has fewer than 10 people, suppress individual sub-group reporting and report only at the aggregate level. Identifiable individual responses in employee surveys violate psychological safety and produce dishonest answers.
When fewer than 30 responses are expected, standard parametric statistics (means, t-tests, Pearson correlations) are unreliable. Redesign the survey to emphasize qualitative data: reduce scaled questions to a maximum of 3-4 critical ones, expand open-ended questions to 4-5, and use open-text coding as the primary analysis method. Report scaled results as percentages with explicit disclaimers: "Based on 22 responses; results are directional only and should not be treated as statistically significant." Consider whether structured interviews would serve the research question better. If the user insists on a quantitative approach with a small sample, use non-parametric statistics: Wilcoxon signed-rank instead of t-test, Spearman instead of Pearson, and chi-square with Yates correction for small expected cell counts.
When a survey will be administered repeatedly (quarterly NPS, monthly pulse, annual engagement), question wording and scales must be frozen from the first administration. Any wording change invalidates trend analysis. Design the minimum question set (5-10 questions) that produces the headline metric for trend tracking. Use a "modular" architecture: a fixed core module (5-7 questions, identical each cycle) plus a rotating deep-dive module (3-5 questions that rotate each cycle to investigate different topics without adding survey fatigue). Document the version history of every survey iteration with change dates. When a question must be changed (e.g., to correct a bias discovered post-launch), treat it as a new question: run both old and new wording in parallel for one administration cycle to establish a "bridge score," then retire the old wording.
When a survey must serve both statistical measurement and exploratory qualitative goals, structure the survey in two stages: all quantitative questions first (scaled, categorical) in the core section, then qualitative questions last (open-ended) as optional closing questions. This ensures that respondents who hit survey fatigue and abandon after the core section have still produced analyzable quantitative data. Never interleave open-ended questions with scaled questions -- the cognitive switch between answering a 5-point scale and writing a paragraph disrupts flow and increases partial-completion rates. When analyzing the combined data, use "quantitative anchoring": first analyze the quantitative results to identify the key finding, then use the qualitative responses to explain and contextualize that finding, not to generate new hypotheses.
Standard Likert phrasing ("Neither Agree nor Disagree") is difficult to process for populations with low reading proficiency or non-native language skills. Use visual scales: a 5-point smiley face scale (😞 😕 😐 🙂 😀) for satisfaction, star ratings for quality, or a thumbs up/down binary for simple yes/no evaluations. Keep all questions under 12 words. Avoid negatives entirely ("Is the product hard to use?" should become "Is the product easy to use?"). For translated surveys, use back-translation protocol: translate to target language, then have a different translator translate back to English, then compare the back-translation to the original to identify semantic drift. Never use automated translation alone for survey instruments.
For surveys covering sensitive topics, three design changes are mandatory. First, lead the survey introduction with a clear, prominent anonymity guarantee -- state specifically who cannot see individual responses. Second, apply the "indirect question" technique for the most sensitive items: instead of "Have you experienced harassment at work?", ask "Have you witnessed or heard about any of the following behaviors in the past 12 months?" with a list. Indirect questions produce more honest responses because they reduce the personal threat of admission. Third, provide a "Prefer not to answer" option on every sensitive question -- its mere presence (even if not selected) reduces defensive answering patterns. For demographic questions involving race, gender identity, or disability status, use current government standards for category definitions (e.g., U.S. Census Bureau race categories or EEOC standards) to ensure comparability and legal defensibility.
When the user's audience receives surveys frequently (e.g., SaaS customers who receive NPS, CSAT, onboarding, and renewal surveys), response rates and response quality degrade over time. Solutions: (1) Implement survey suppression logic -- do not survey the same respondent more than once per 60-90 day window across all survey types. (2) Reduce to a single-question transactional survey (one CSAT or CES question immediately post-interaction) rather than a 10-question satisfaction survey quarterly. (3) Use survey rotation -- divide the respondent pool into equal thirds, and each third receives a different survey each quarter, ensuring full coverage across four cycles while each individual receives only one survey per year. (4) Consider passive data collection (behavioral analytics, support ticket sentiment analysis) as a partial replacement for explicit survey questions. Survey fatigue surveys are a real phenomenon -- if a user asks why their response rate has dropped from 25% to 8%, survey fatigue and suppression strategies are the first diagnosis.
For surveys distributed immediately after a specific event (webinar, conference, product launch, support interaction), the collection window must be short (24-72 hours maximum) because response quality degrades rapidly with time. The question scope must be tightly bounded to the event itself -- do not use a post-event survey to ask about general brand satisfaction. Apply CSAT or CES as the primary metric rather than NPS (NPS measures relationship loyalty, not event quality). For post-support surveys specifically, use Gartner's validated CES question: "The company made it easy for me to handle my issue" on a 7-point agreement scale. CES is more predictive of repurchase behavior and churn than CSAT for support interactions. For conferences or training events, use Kirkpatrick's four levels as the RQ framework: Reaction (did they enjoy it?), Learning (did they gain new knowledge?), Behavior (will they apply it?), and Results (will it impact their metrics?). Each level requires different question types.
Surveys in healthcare, academic research, financial services, or involving minors require specific design constraints. Academic and clinical surveys involving human subjects require Institutional Review Board (IRB) approval before launch -- the survey instrument itself is part of the IRB submission and cannot be modified after approval without re-submission. Healthcare surveys collecting Protected Health Information (PHI) must comply with HIPAA -- anonymous surveys are safest; if a survey collects contact information alongside health data, a Business Associate Agreement may be required between the survey platform and the organization. Surveys involving minors (under 18) require verifiable parental consent in most jurisdictions. Financial services surveys that could be interpreted as investment research must include standard disclaimer language. Flag these requirements explicitly to the user before proceeding with survey design.
Input: "We're launching a new onboarding flow for our B2B project management SaaS. We need to survey customers who went through the new onboarding in the past 60 days to understand whether it's working. We care about time-to-value, where people get stuck, and whether they feel confident using the product. We have about 300 customers who went through the new flow and we're hoping for at least 80-100 responses."
Purpose: Evaluate whether the redesigned onboarding flow achieves faster time-to-value, identify where users encounter friction or confusion, and measure post-onboarding product confidence -- to inform whether to scale the new flow to all customers or iterate further
Target respondent: B2B customers who completed the new onboarding flow within the past 60 days (n=300 eligible)
Distribution channel: In-app notification (Day 3 post-onboarding complete) + email link (Day 5 fallback for non-openers)
Expected sample size: 90-120 responses from 300 eligible customers (30-40% response rate -- higher than average due to recency of experience)
Minimum viable sample: 73 responses (population 300, margin of error ±10%, 95% confidence) -- below this, results are directional only
Estimated completion time: 4-5 minutes (9 questions, validated by pre-launch test)
Survey open period: 10 days from launch (close before 14-day mark when onboarding memory degrades)
Primary metric: Post-Onboarding Confidence Score (mean on a 5-point scale, target ≥ 4.0/5.0)
Benchmark: Pre-redesign onboarding satisfaction score of 3.2/5.0 (Q3 survey); no prior confidence score exists -- this is the baseline measurement
| ID | Research Question | Priority | Analysis Type | Survey Questions |
|---|---|---|---|---|
| RQ1 | What is the post-onboarding confidence level compared to the pre-redesign baseline? | P1 | Descriptive + Comparative | Q2, Q3 |
| RQ2 | Which specific onboarding steps produce the most friction, confusion, or abandonment? | P1 | Descriptive + Diagnostic | Q4, Q5, Q6 |
| RQ3 | How long did customers perceive it took to feel productive with the product, and does that differ by company size? | P1 | Comparative | Q1, Q7 |
| RQ4 | What would most improve the onboarding experience for customers who report low confidence? | P2 | Relational + Qualitative | Q2, Q8 |
| RQ5 | Are there differences in confidence and friction by the respondent's role (admin vs end user)? | P2 | Comparative | Q9 (segmentation), all core questions |
Purpose: Establish time-to-value perception before anchoring on specific satisfaction ratings
Q1: After completing the initial setup, how long did it take before you felt productive using [Product Name] for real work?
Type: Single-choice (radio)
Required: Yes
Response options:
Less than 1 day
1-3 days
4-7 days
1-2 weeks
More than 2 weeks
I don't feel fully productive yet
Answers RQ: RQ3 -- perceived time-to-value measurement
Analysis method: Frequency distribution; cross-tabulate with Q9 (role) and company size if available from CRM enrichment
Decision rule: If more than 30% of respondents select "More than 2 weeks" or "I don't feel fully productive yet," the onboarding flow requires a significant revision; escalate to Head of Product
Skip logic: None
Wording rationale: Opens with a behavioral/temporal question (not an opinion) to orient the respondent and avoid anchoring the confidence question that follows
Purpose: Measure the primary metrics (confidence, friction points) for RQ1, RQ2, and RQ3
Q2: Overall, how confident do you feel using [Product Name] after completing onboarding?
Type: 5-point Likert (Confidence scale)
Required: Yes
Response options: Not at all confident (1) / Slightly confident (2) / Moderately confident (3) / Very confident (4) / Extremely confident (5)
Answers RQ: RQ1 -- primary metric (Post-Onboarding Confidence Score)
Analysis method: Mean and SD; compare to baseline (no prior benchmark -- this is the baseline). Track as primary metric. Segment by Q9 (role) and Q1 (time-to-value).
Decision rule: If mean ≥ 4.0/5.0, proceed to scale new onboarding to all customers. If 3.5-3.9, iterate on specific friction points identified in Q4-Q6. If < 3.5, conduct follow-up interviews before scaling.
Skip logic: None
Wording rationale: "Confident" is used rather than "satisfied" because the specific goal of onboarding is capability-building, not general happiness. Placed after Q1 to avoid overall confidence contaminating the time-to-value perception.
Q3: Please rate your experience with each of the following aspects of the onboarding:
Type: Matrix (5-point Satisfaction scale)
Required: Yes
Rows:
Initial account setup (first-time login, team invitation)
Learning how to create your first project
Understanding how to assign tasks and deadlines
Connecting integrations (Slack, calendar, email)
Finding help resources when stuck (tutorials, tooltips, help center)
Scale: Very Dissatisfied (1) / Dissatisfied (2) / Neither Satisfied nor Dissatisfied (3) / Satisfied (4) / Very Satisfied (5)
Answers RQ: RQ2 -- identify which specific onboarding modules produce friction
Analysis method: Mean per row; rank all 5 modules from lowest to highest mean; flag any row with mean < 3.0/5.0 as a critical friction point
Decision rule: Any module with mean < 3.0 receives a dedicated design sprint in the next two-week cycle
Skip logic: None
Wording rationale: Matrix limited to 5 rows to avoid straight-lining. Rows correspond directly to the five steps in the new onboarding flow (they were documented before this survey was designed). Generic labels like "Overall experience" are excluded from the matrix -- that is captured separately in Q2.
Q4: Were there any steps during onboarding where you felt stuck or confused?
Type: Single-choice (radio)
Required: Yes
Response options:
Yes
No
Answers RQ: RQ2 -- binary filter before follow-up diagnostic question
Analysis method: Frequency (% yes vs % no); cross-tabulate with Q2 (do "stuck" respondents have lower confidence scores?)
Decision rule: If more than 40% of respondents select Yes, the onboarding has a structural problem, not just isolated edge cases
Skip logic: If No, skip to Q6
Wording rationale: Binary question used here deliberately as a filter rather than asking everyone to identify friction points. Asking "Which steps confused you?" of someone who was never confused produces forced or false answers.
Q5: [Displayed only if Q4 = Yes] Which part of the onboarding caused the most confusion or friction? Select all that apply.
Type: Multi-select (checkbox)
Required: No (displayed conditionally)
Response options:
Initial account setup
Creating my first project
Assigning tasks and deadlines
Setting up integrations
Finding help resources
Understanding what to do next at any given step
The amount of information felt overwhelming
Something else (please describe briefly): [short open text field]
Answers RQ: RQ2 -- specific friction point identification
Analysis method: Frequency per option (report as "% of respondents who encountered friction here" -- not as % of total responses). Compare Q5 frequencies to Q3 matrix means for consistency. If Q5 frequency ranking and Q3 mean ranking contradict each other, flag as an insight requiring qualitative follow-up.
Decision rule: Top 2 friction options by frequency become the two onboarding design sprint priorities
Skip logic: Shown only if Q4 = Yes
Wording rationale: Multi-select used because multiple friction points can co-exist. "Something else" with a short text field captures unanticipated issues.
Q6: How would you rate the quality of the help resources available during onboarding? (Tooltips, tutorial videos, help center articles)
Type: 5-point Likert (Satisfaction scale)
Required: Yes
Response options: Very Dissatisfied (1) / Dissatisfied (2) / Neither Satisfied nor Dissatisfied (3) / Satisfied (4) / Very Satisfied (5), plus "I did not use any help resources" option
Answers RQ: RQ2 -- specific diagnostic on help resource quality (a known weakness from pre-redesign feedback)
Analysis method: Mean excluding "did not use" responses; cross-tabulate with Q4 (did people who got stuck have lower help resource ratings?); report "did not use" rate separately as an indicator of self-sufficiency
Decision rule: If mean < 3.5/5.0 among those who used help resources, assign content audit to documentation team before scaling
Skip logic: None
Wording rationale: "N/A" option (did not use) is included here because a meaningful proportion of users may have completed onboarding without needing help -- forcing a rating would produce invalid data. This is one of the justified uses of N/A per Rule 8.
Q7: How does your confidence using [Product Name] today compare to your confidence immediately after completing onboarding?
Type: Single-choice (radio)
Required: Yes
Response options:
Much more confident now
Somewhat more confident now
About the same
Somewhat less confident now
Much less confident now
Answers RQ: RQ3 -- measures whether product experience after onboarding reinforces or erodes onboarding gains
Analysis method: Frequency distribution; flag any "less confident" response rate above 15% as a signal that the product experience (not just onboarding) is eroding confidence post-launch
Decision rule: If more than 15% select "Somewhat" or "Much less confident now," the issue extends beyond onboarding into core product usability -- a separate product satisfaction survey should be initiated
Skip logic: None
Wording rationale: Comparative question rather than a repeat rating avoids the respondent having to recall a precise past rating. "Compared to" framing is cognitively easier and produces more reliable data.
Purpose: Capture qualitative context and enable segmentation
Q8: What is the single most important improvement we could make to the onboarding experience?
Type: Open text (short -- 1-3 sentences; displayed with a visible character count of 280 max)
Required: No
Response options: Free text field
Answers RQ: RQ4 -- qualitative improvement prioritization
Analysis method: Thematic coding into MECE categories (Setup complexity, Guidance clarity, Pacing and length, Integration issues, Missing features, Other). Report top 3 themes by frequency. Cross-tabulate themes with Q2 confidence quintiles: do low-confidence respondents (score 1-2) identify different themes than high-confidence respondents (score 4-5)?
Decision rule: Any theme identified by more than 20% of respondents who also scored Q2 ≤ 2 is a critical remediation priority
Skip logic: None
Wording rationale: "Single most important" forces prioritization -- without that constraint, respondents list everything and themes become unranked. The 280-character limit encourages brevity without being restrictive.
Q9: What best describes your role when using [Product Name]? (Optional)
Type: Single-choice (radio)
Required: No
Response options:
Workspace Admin (sets up the account, invites team members)
Project Manager (creates and manages projects)
Team Member / Contributor (assigned tasks, works within projects)
Executive / Observer (views dashboards and reports)
Other
Answers RQ: RQ5 -- segmentation variable for all core questions
Analysis method: Cross-tabulate Q2 (confidence), Q3 (matrix scores), and Q1 (time-to-value) by role. Admins and end users have different onboarding journeys and are expected to show different friction patterns. Report role segments separately if n ≥ 20 per segment.
Decision rule: If Admin and Contributor segments show confidence means that differ by more than 0.5 points, design role-specific onboarding tracks
Skip logic: None
Wording rationale: Role descriptions include behavioral descriptors, not just titles, because B2B respondents use different role titles across companies. Optional because demographics at the end have higher N/A rates and this segmentation is P2, not P1.
| Scale Name | Point Count | Labels (Low → High) | Used In | Notes |
|---|---|---|---|---|
| Confidence (5-point) | 5 | Not at all confident → Extremely confident | Q2 | Treated as interval for mean analysis; distinct from satisfaction |
| Satisfaction (5-point) | 5 | Very Dissatisfied → Very Satisfied | Q3, Q6 | Treated as interval for mean analysis; both endpoints explicitly labeled |
| Comparative change (5-point) | 5 | Much less confident now → Much more confident now | Q7 | Ordinal -- report frequencies, not mean |
| RQ | Input Questions | Statistical Method | Output Metric | Decision Rule | Owner |
|---|---|---|---|---|---|
| RQ1: Post-onboarding confidence | Q2, segmented by Q9 (role) and Q1 (time-to-value) | Mean and SD; one-way ANOVA by role segment (if n ≥ 20 per group) | Mean Confidence Score (1-5); this is the new baseline | ≥ 4.0: scale new flow. 3.5-3.9: iterate on friction points. < 3.5: interview before scaling | Head of Onboarding |
| RQ2: Friction point identification | Q3 (matrix means), Q4 (stuck binary), Q5 (friction multi-select), Q6 (help resources) | Rank Q3 rows by mean; frequency distribution for Q5; compare Q3 and Q5 rankings for consistency | Ranked friction hotspot list; top 2 module priorities | Any Q3 row < 3.0 OR any Q5 option > 30% frequency triggers a design sprint | Product Designer |
| RQ3: Time-to-value by segment | Q1, Q7, segmented by Q9 | Frequency distribution for Q1; cross-tab Q1 × Q9 (role) using chi-square if n allows | % achieving productivity within 3 days; distribution by role | If >30% select > 2 weeks or "not yet productive," investigate whether role-specific tracks are needed | Product Manager |
| RQ4: Improvement prioritization | Q8, cross-referenced with Q2 quintiles | Thematic coding (MECE categories); frequency by theme; compare themes between low-confidence (Q2 ≤ 2) and high-confidence (Q2 ≥ 4) groups | Top 3 improvement themes; low-confidence theme list | Theme >20% among low-confidence group = critical remediation | Content and UX Lead |
| RQ5: Role-based differences | Q9 × Q2, Q3, Q1 | Cross-tabulation; subgroup means; one-way ANOVA if n ≥ 20 per role | Confidence and friction scores by role | If Admin vs. Contributor confidence differs by > 0.5 points, design role-specific tracks | Head of Onboarding |
Pre-launch checklist:
Cognitive interview completed with 5 recent onboarding completers (mix of roles)
Completion time verified at under 5 minutes with 10 internal testers
Mobile rendering tested (Q3 matrix rendered correctly on iOS Safari and Chrome Android)
Skip logic Q4→Q5 tested: respondents selecting "No" on Q4 correctly skip Q5
Anonymity posture confirmed: responses are anonymous; individual response cannot be tied to account or contact record
Survey introduction text (to display before Q1): "You recently completed onboarding for [Product Name]. We'd love to know how it went -- your feedback directly shapes how we improve the experience for new customers. This survey takes about 4-5 minutes. Your responses are completely anonymous and will not be shared with your account team or sales contact."
Anonymity statement: Your responses are completely anonymous. Individual responses cannot be linked to your account, company, or
SKILL.md
547 lines · 7,542 words
The text of SKILL.md is shown as it downloads, under the Apache-2.0 license, by Ferrox Labs. The license is the one in the file's own frontmatter.
Your new bot
Survey Design bot
Waiting in your sidebar once you say go.
No account needed · Runs on the AI plan you already pay for