Over the summer we got the same complaint in a dozen wordings: "Why is this marked hard? It took me twenty seconds." And its twin: "This is supposed to be easy and I got it wrong three times."
You were right to be skeptical. A difficulty label is only useful if it means the same thing on every question, and ours did not. So we rebuilt it. Every question on SHS Prep is now scored on a published rubric, the rubric lives at /difficulty-levels, and the scores that produced each rating are saved on the question so you can see why.
Why difficulty labels are usually wrong
Most prep material rates difficulty one of two ways. The first is a gut call: whoever wrote the question tags it. Two writers, two scales.
The second sounds scientific: call a question hard if a lot of students miss it. Real exams are calibrated this way, with controlled field tests. On a practice platform the data is noisy in ways that matter. Students click through questions at 11 pm. They abandon a mock exam halfway. And the student weakest at probability is the one drilling probability, so an ordinary probability question looks brutal when it is only unfamiliar to the people who attempted it. A rating built this way drifts from week to week and can never tell you why a question is hard.
We wanted a rating that is a property of the question, not a statistic of who answered it.
What we borrowed from prep books
Open a good SHSAT book and the front matter explains its star ratings: one star is one idea and one step, three stars is several ideas and a trap. Drills are sorted into tiers, and the hardest tier is pinned to the hardest questions on released official tests, so "hard" means something outside the book.
We took all three ideas: tiered drill sets, a printed rubric, and calibration against released official SHSAT questions. Then we went further than a book can. We show the individual scores, and the same rubric covers every question in both the practice bank and the mock exams.
The rubric
Each question family has a short list of aspects, each scored 0, 1, or 2 against a fixed description. The aspects that most decide how demanding a question is count double. The scores are summed and the total is cut into three levels. Here is the math rubric, word for word from the scoring module:
| Aspect | 0 | 1 | 2 |
|---|---|---|---|
| Type of thinking (counts double) | Recall a fact or run one familiar procedure | Apply a concept in a setting you have not seen it in before | Reason strategically through a non-routine problem |
| Steps to solve | 1 to 2 steps | 3 to 4 steps | 5 or more steps |
| Concepts combined (counts double) | One concept | Two concepts | Three or more concepts |
| Setting up the problem | Given directly in math notation | Word problem with a straightforward setup | Word problem or figure you have to turn into a model yourself |
| Arithmetic load | Whole numbers and simple fractions | Fractions, decimals, and percents together | Algebraic manipulation or messy values |
| Trap answers | Wrong answers are easy to rule out | One wrong answer targets a common mistake | Several trap answers, or "which must be true" logic |
| Answer format | Multiple choice | Grid-in or a single interactive answer | Multi-part interactive answer |
The maximum is 18. A total of 0 to 4 is Level 1 Easy, 5 to 10 is Level 2 Medium, and 11 or more is Level 3 Hard.
Reading comprehension has six aspects. Two describe the passage and are shared by every question on it: Passage length and sentences and Passage style. Four are scored per question: What the question asks (counts double, from "Find something stated directly in the text" up to "Analyze the author's craft, purpose, tone, or structure"), Where the evidence is, How close the wrong answers are, and Answer format.
Revising and editing has four: The rule being tested (counts double, from "A common rule (subject-verb agreement, basic commas)" to "A rare or subtle rule (dangling modifiers, tricky pronouns)"), How much you have to read, How close the choices are (counts double), and Answer format. Every level description for every aspect is on the difficulty levels page.
A worked example
A store raises the price of a jacket by 25%, then puts the jacket on sale for 20% off the new price. The sale price is $36. What was the original price?
- Type of thinking: 1. You know percents, but you have to run them in reverse, from the final price back to the start. Counts double: 2 points.
- Steps to solve: 1. Markup, discount, combine, solve. 1 point.
- Concepts combined: 1. Percents and solving an equation. Counts double: 2 points.
- Setting up the problem: 1. A word problem with a direct setup. 1 point.
- Arithmetic load: 1. Multiplying 1.25 by 0.8 is where careless work goes wrong. 1 point.
- Trap answers: 1. One choice is $36, for anyone who assumes a 25% markup and a 20% discount cancel. 1 point.
- Answer format: 0. Multiple choice.
Total: 8 of 18, Level 2 Medium. Under the question you would see: Medium, applies a concept in a new setting, combines two concepts, 3 to 4 steps. That is the heart of the SHSAT: one familiar idea used in a slightly unfamiliar way, a few steps, one trap.
Where the cut points come from
The cut points are not round numbers we liked. We scored released official SHSAT questions with the same rubric and set the thresholds so the hardest official questions land in Level 3 and a typical official form comes out near its real mix, about 20% easy, 60% medium, 20% hard.
How to use the levels
In Targeted Practice, every topic lets you pick Level 1, 2, 3, or Mixed, or a ramp that starts at Level 1 and climbs. The ramp is the right default when you are learning a topic. Pick Level 3 alone once Level 2 is boring. Try it on percentages or algebraic applications, where the gap between levels is very visible.
In timed section practice and mock exams, you choose a tier. Standard matches the real exam's mix. Foundation is about 40% easy, 50% medium, 10% hard, for a first mock or for rebuilding confidence. Challenge is about 10% easy, 50% medium, 40% hard, for students chasing a Stuyvesant or Bronx Science cutoff. Since the adaptive SHSAT gets harder as you do well, Challenge previews a strong test day.
Under every question, you get the level and a one-line reason built from the aspects that scored highest. Use it. If your misses cluster on "several trap answers," you have a reading problem, not a math problem. If they cluster on "5 or more steps," you are losing the thread mid-solution and should write more down.
Lessons are unchanged.
Why not use other students' results at all?
Because the rubric answers a question that data cannot: why. Knowing that 38% of students missed a question tells you nothing about what to do next. Knowing it scored 2 on "Setting up the problem" tells you to practice building the model from the words. And a rubric can be argued with.
We still track your accuracy at every level in every topic. That is your data, about you, and it belongs in the analytics. It just does not get to decide what a question is called.
When a rating looks wrong
Sometimes it will be. Report it from the question. A person reviews every report against the rubric, and if we agree, a teacher overrides the rating. The override is recorded on the question with who made it and when, so we can see later where the rubric itself needs work, not just one question.
A Level 2 that feels like a Level 3 is not always an error, though. The rubric rates the question, not you. A medium question in your weakest topic will feel hard, and that is the most useful thing the levels can tell you: it points at the topic, not at the test.
Read the full rubric at /difficulty-levels, then head to Targeted Practice and pick a level.
