Four reasons tests teach nothing
The skill names them up front. Tests end before they have enough data, so a lift seen after a few days turns out to be noise. Teams test what is easy, such as button colors, instead of what could matter, such as the value proposition. They run too few tests to learn quickly. And nobody owns the test, so anyone can stop it or read it differently.
Its check on velocity is blunt: count the tests your team shipped in the last 90 days. Fewer than five means speed is the first problem.
A hypothesis needs a mechanism.
A weak hypothesis says a green button will convert better. A strong one names the change, the metric, the smallest lift worth detecting, and the reason, written from the user’s point of view. If you cannot say why a change should work, the skill treats that as a sign you do not understand the problem yet.
A ranked backlog, reviewed weekly
Ideas are scored from 1 to 10 for impact, confidence, and ease, and the three scores are multiplied, so a weak score in any one of them pulls the idea down. Each backlog entry carries its hypothesis, the page, the primary and secondary metrics, the sample size, the score, an owner, and a status. The goal is six to eight fully written hypotheses ready to launch at any time.
Sizing and running a clean test
Sample size comes from four inputs: your baseline rate, the smallest effect worth detecting, a 95% significance level, and 80% power. The skill’s worked example is sobering.
A page converting at 3% needs about 30,000 users per variant to detect a 10% relative lift, which is 60 weeks at 1,000 visitors a week. When the math does not fit, it tells you to accept a larger effect or test on a busier page.
The live version always stays in the test as the control. The skill assigns visitors by user, not by session, and it avoids launching a test right before a major event. Each one runs a full business cycle, usually two to four weeks for B2B.
Reading the result without fooling yourself
When a test ends, the skill asks whether it reached its planned size, whether the p-value clears the threshold, and how wide the confidence interval is. It checks secondary metrics for side effects and treats segment wins as ideas for the next test, not as winners. It also separates a result that is real from one that is big enough to matter.
It stops a test early for only two reasons. One is real harm, such as a critical metric falling by 30% or more in the first 20% of the sample. The other is an outside event that corrupts the data.
The experiment program in the ab-testing file
Hypothesis structure, scoring, sizing, design rules, and the reporting template are in the ab-testing skill file on GitHub. It is released under the MIT License, “Copyright (c) 2026 ScayverGraphix.” The skills follow the Agent Skills spec published by vercel-labs/skills.
A first month, week by week
Week one finds the three steps in your funnel where the most people drop off. Week two runs five customer interviews about what felt uncertain. Week three turns those into hypotheses and scores them. Week four launches the first test with a stopping rule and a date to call it.
After that, the skill aims to keep one test live at all times. Ask your agent to build an experimentation backlog or calculate a sample size, or install it with npx skills add scayver/marketing-skills --skill ab-testing.
One controlled change at a time
Changing three things at once can make a page look busy without showing which barrier mattered. Our Conversion Optimization service works the same way the skill does: define one specific change, then review what the evidence supports.