What a million completed tasks reveal about good maths tasks

Children have now completed more than a million tasks in Mazio Maths. The obvious question would be how much practice that adds up to. The more interesting question is what we have learned about tasks along the way. Behind every Mazio Maths task lies a didactic model, an assumption about how children think and where they go wrong. Every attempt tests this model. A million attempts are a million small tests of our own assumptions, and we did not pass all of them.

Two tasks, one topic, worlds apart

Two Knobelbox tasks deal with the same key idea: reflection in a mirror line. In one, children choose the correct mirror image, and 89 per cent succeed at the first attempt. In the other, they place the reflected point in the grid themselves. Here the figure is 43 per cent, and nowhere else do children ask for a hint more often.

The format explains this difference; the topic is the same in both. Recognising a mirror image and constructing a reflection yourself are different cognitive actions, and between them lies exactly the learning path that teaching has to cover. For us as task designers there is a second lesson in this. When a task shows an unusually high error rate and an unusually high demand for help at the same time, the cause can lie with the children or with the task. Such tasks are automatically added to our review list, because sometimes the difficulty lies in a misleading representation rather than in the reflection itself.

Errors are rarely random

When many children fail at combinatorics tasks, it might look like random scatter. The diagnosis shows something else: by far the most common pattern is adding instead of multiplying. Three pairs of trousers and four T-shirts then give seven combinations instead of twelve. The same signature appears everywhere. In written column subtraction, children take the smaller digit from the larger one, whichever is on top. In reflection tasks, they check only one side of the figure or compare locally instead of looking at the whole shape. In probability, they treat “certain” as if it meant “possible”.

These are stable, comprehensible patterns of thinking rather than careless slips, and research has described them for decades. That is why Mazio Maths models a catalogue of 5,178 such misconceptions behind its tasks. A wrong answer is marked as wrong and, where possible, also assigned to a pattern. The difference matters in practice. Repetition helps with careless slips. With a misconception, repetition only produces the same error more reliably. What helps are tasks that make the pattern visible and resolve it, and this is exactly what the feedback and the hints from Mazio aim at.

One task becomes 17,825

A Mazio Maths task is a parameterised family of tasks. Number range, representation, arrangement and distractors are generated from parameters, within limits set on didactic grounds. In this way, children completed 17,825 unique variants in the Knobelbox alone within six months; the movement task type alone exists in 1,900 versions.

This does more than prevent copying. Parameters make difficulty controllable: the same task idea can be set in the number range to 20 or to 1,000, with round numbers or with bridging, as a picture or in symbols. They also make the task bank open to analysis. When one variant systematically stands out, comparing it with its siblings shows why.

How adaptivity actually decides

Adaptivity often sounds like magic. In Mazio Maths it is a sober feedback loop. Each task type has its own difficulty scale, and allocation responds to what a child has shown most recently. The data show whether this works: stronger children are allocated measurably harder variants. The success rate stays similar across all children, and that is the purpose of the design. Every child should practise in the range where tasks are neither trivial nor discouraging.

It is important what adaptivity means for us: it covers more than easier or harder. Often the better response to an error is the same demand in a different representation, an isolated sub-step or a contrasting pair. The reasons are set out in detail in the article Adaptivity goes beyond “easier or harder”.

Hints that lead to thinking

Hints are the most delicate part of a learning environment. Too little help frustrates; too much help takes over the thinking. Mazio Maths therefore works with a hint ladder: first direct attention, then ask a question about the solution path, then offer a representation, then activate a strategy. No step gives away the solution itself; the last step stays with the child.

Whether this ladder holds up can be checked in the attempt sequences, because each attempt records whether a hint came before it. The result is clear: after the first incorrect attempt, the next attempt succeeds in 65 per cent of cases with a hint and in 46 per cent without one. Even after adjusting for task type, a difference of around 14 percentage points remains, and even deep in the ladder, from the fifth attempt onwards, attempts with a hint are still ahead.

The most interesting part is the mechanism. Without a hint, the next answer after an error comes after a median of 4 seconds, which is trial and error. With a hint, 21 seconds pass. So the hint changes how a child works as well as what they know: it interrupts guessing and creates time to think.

In the units, help is built differently. There, Mazio responds automatically after every error, with a message written on didactic grounds for exactly this sub-task. More than 112,000 such error-and-retry moments show three things. After the first message, just over half of the next attempts succeed. Whether the message works depends on whether it is read: children who answer again within 30 seconds reach 38 per cent, and those who take more time reach 53 per cent. Many children answer faster than they can read, and this leads to a concrete product idea that we will test: a short pause before the next input. Third, the messages vary enormously. For the best sub-task, the help leads to the solution in 92 per cent of cases; for the weakest, in 8 per cent. Probability levels appear strikingly often at the lower end, where the misconception “certain means possible” sits; a text message does not resolve deeply rooted patterns. Such levels go straight onto the revision list, because where a help message reaches almost nobody, the problem lies with the message rather than with the child.

To be clear about the limits: these are observational data without random allocation. We cannot yet tell which type of hint works best at which point, whether a sub-question, a counterexample or a visualisation. To find out, we are currently instrumenting the ladder in finer detail.

Tasks are never finished

From the outside, a task looks finished once it is online. In fact, that is where its second phase of development begins. We keep a record for every task type: reported problems, a change log and quality scores from the process data. If a task stands out in the data triage, it is reworded, replaced, re-parameterised or given a new difficulty level, and the next wave of attempts shows whether the change has worked.

This is the real difference from a printed textbook, whose tasks stay as good as they were on the day it went to press. A task bank with a million attempts behind it gets a little better every week, because it learns something about itself from every single attempt.