Articles / Explainable vs black-box coaching

Why an explainable coaching decision beats a black-box one

Reading time about 7 minutes

When software tells you to change your calories or add a set, there are two very different ways it can have reached that conclusion. It can follow a set of stated rules applied to your data, in which case you can read exactly why it decided what it did. Or it can produce the answer from a large statistical model whose internal reasoning is not laid out, in which case you get the output but not an auditable account of how it got there. This is the difference between an explainable decision and a black-box one, and for coaching decisions it matters more than it first appears.

This article makes the case for the explainable approach. It tries to be fair to the alternative, because generative models are genuinely useful for some things, and pretending otherwise would be dishonest.

What each approach is good at

Generative models, the kind behind conversational assistants, are remarkable at language. They can explain a concept in a dozen ways, answer a rambling question, and produce fluent, tailored-sounding text on demand. For learning, for exploring ideas, and for turning a vague question into a readable answer, they are a real advance.

They have two properties that matter here, though. First, their output is probabilistic: the same question can yield different answers, and a fluent answer is not always a correct one. Researchers describe the tendency of these models to produce confident but unfounded statements as hallucination, and it is a well-documented, structural feature rather than a bug that has been fully solved [1]. Second, their reasoning is not transparent by default. You can ask a model to explain itself, but the explanation it generates is itself produced text, not a guaranteed trace of the actual computation. That gap between a plausible explanation and the real basis for a decision is exactly what the field of explainable machine learning exists to address [2].

Why a coaching decision is a special case

A coaching decision is not the same as a piece of explanation. When you ask software to tell you whether to cut your calories this week, you are asking it to act on your body, over weeks and months, based on your own health data. Three things follow.

• You need to be able to check it. A recommendation to change your intake is one you should be able to interrogate: on what trend, over how many weeks, against which target. If the reasoning is not visible, you cannot tell a sound decision from a confident-sounding mistake.
• It should be consistent. The same inputs should produce the same decision. If identical data can yield different advice on different days, you cannot learn what works, and neither can you trust that a change reflects your body rather than the model's variance.
• The stakes include safety. Calorie and training advice can be harmful if it drifts into under-fuelling or ignores warning signs. Rules you can read are rules you can audit for safety. A black box offers no such guarantee that a hard limit will always hold.

None of this means a generative model can never help around the edges of coaching. It means the core decision, the one that acts on your body, is a poor fit for an opaque, non-reproducible process.

What deterministic, explainable coaching means

A deterministic system applies stated rules to your inputs and produces a decision that is reproducible and inspectable. Deterministic coaching, not generative coaching, has three practical properties:

• Same inputs, same decision. Feed it the same data and it returns the same answer, every time. There is no run-to-run variance to second-guess.
• The reason is written down. Every decision comes with the basis for it: the trend it read, the threshold it compared against, and the rule it applied. You can agree or disagree, but you can always see the working.
• It can hold when the data is thin. Because the rules are explicit, they can include the rule do not decide yet, so the system waits for enough evidence instead of manufacturing a confident answer from too little.

This is not a claim that rules are cleverer than a large model. They are not. It is a claim that for a decision that acts on your health, week after week, being reproducible and auditable is worth more than being fluent.

The fair objection, and the honest answer

The reasonable objection to deterministic systems is that they are only as good as their rules. A rule-based engine cannot invent an insight its designers never encoded, and a badly designed rule set will give bad advice confidently. That is true, and it is the real cost of the approach.

The honest answer is that this cost is visible and fixable in a way a black box's is not. When a rule is wrong, you can find it, read it, and correct it, because it is written down. When an opaque model gives poor advice, you often cannot even establish why, let alone guarantee the same failure will not recur. For a system that people rely on to make repeated decisions about their own bodies, a fault you can locate and fix is a far better position than a fault you cannot even name. Both approaches can be wrong. Only one of them lets you see how.

Where this leaves you

If you want to explore ideas, learn the theory, or talk something through, a conversational model is a genuinely good tool, and it would be dishonest to say otherwise. If you want a repeated decision about your training and nutrition that you can check, trust to be consistent, and hold to a safety standard, an explainable, deterministic decision is the better fit. The two are not in a fight. They are good at different jobs.

The point is not that one method wins every contest. It is that for the specific job of deciding, repeatedly and safely, what you should do next with your own body, being able to see the reason is not a nice extra. It is the whole point.