The Limit May Not Be in the Model

Four Questions, Again

I asked an unnamed console-style AI product for a longer, iterative interview. It came back with four clarifying questions. Fair enough. I answered them and asked it to continue.

Then it gave me four questions again.

That was the part I could not stop thinking about. Four is not obviously a bad number, and I have no access to the product's internal implementation. But my request for a longer exchange did not seem to change the shape of the interaction at all.

Maybe the model could have handled more. Maybe it could not. The interface gave me no way to tell, because the product kept steering the exchange back to the same default.

This is where model capability and product capability get mixed together. We experience the whole system through the product, so a limit in the interface can easily feel like a limit in the model. A team can replace the model and still leave the surrounding behavior almost untouched.

I did not come away thinking, "Four questions is wrong." I came away wondering who owned the number. Was four chosen deliberately? Is there a current evaluation behind it? What evidence would cause it to change? Or did it simply survive another model release because nobody reopened the decision?

That question is bigger than one interview flow. As models improve, the features built around them should be expected to evolve too.

The Model and the Product Move on Different Clocks

Model capability can move quickly on a specific task. As one example, Anthropic reported that Claude Sonnet 4.5 scored 61.4% on OSWorld, a computer-use benchmark, compared with 42.2% for Sonnet 4 four months earlier.

That is a vendor-reported result on one named evaluation. It does not prove that every new model is better at every task, and it is not a universal growth curve. It simply makes the timing problem visible: the conditions under which a product default was chosen can change faster than the product revisits the default.

Product behavior often moves on a different clock. A team swaps in a new model, confirms that the existing feature still works, and ships. That is a sensible compatibility check, but it is not the same thing as asking whether the feature should now work differently.

The model upgrade can be real while the user's experience stays pinned to assumptions from an older release.

This does not mean every limit should rise whenever a model version changes. More interaction can mean more latency and cost. A newly capable model can introduce different quality or safety tradeoffs. Sometimes the old default is still the right one. Sometimes the right move is a lower limit or an adaptive one.

The problem is not that a default stayed the same. The problem is that staying the same can happen without a fresh decision.

Where Four Can Become Embedded

A question count sounds like one setting. In a real product, it can become part of everything around the model.

The interface may be laid out for a predictable number of prompts. State handling may expect a short exchange. Tests may stop at the current default, which means longer interactions are never evaluated. Cost and latency budgets may assume the existing path. Client integrations may rely on its shape. Teams learn to trust the behavior they already know.

Any one of those could explain the interaction I saw. I cannot claim that any of them does.

What I can say is that changing the model does not automatically revisit those surrounding choices. Once a default appears in interface copy, orchestration, tests, budgets, and rollout plans, it stops being a tiny implementation detail. It becomes part of the product contract.

If that contract is invisible, it is easy to mistake inheritance for intent.

Replace the Inherited Number With a Contract

The useful alternative is not "always ask more questions." It is to make the default reviewable.

I think of that as a capability-linked contract. It records why the current value exists, who owns it, what evidence could change it, how a candidate change will be tested, and what event schedules the next review.

For the interview example, I would start by pulling the question count out of the shadows. Make it versioned configuration, and make each served interaction traceable to the contract version that selected it. Google's guidance on monitoring recommends tracking software and dynamic-configuration versions so observed behavior can be connected to an intended change. The same idea applies here.

Next, test a useful range of values on the actual job the feature is supposed to perform. A longer exchange is only better if it improves the downstream result enough to justify the extra effort and cost.

Generic model benchmarks cannot choose that product default. OpenAI’s evaluation guidance recommends evaluations grounded in real application use and continued evaluation as an application changes. For this feature, that could mean asking whether the interview gathers the information needed for the next step while keeping quality, latency, cost, compatibility, and safety within acceptable bounds.

If a different default looks promising, give it a limited release. Google's canarying guidance describes exposing a change to a subset, comparing it with a control, and using attributable evidence to expand or roll it back.

Completion and abandonment are useful signals, but neither is the answer by itself. A short interview may get more completions and collect worse information. A longer one may lose some users and produce a much better result for those who finish. The owner has to read those signals alongside the outcome the feature exists to create.

The number can stay simple. The decision around it should be visible.

Close the Release Loop

This cannot be a one-time cleanup every few years. The review has to become part of the release loop.

A material model release should trigger the task-specific evaluation. A named owner reviews the result and decides whether there is enough evidence to try another default. That candidate goes through a canary against a control. Telemetry stays attached to the contract version that was actually served. The result informs both today's decision and the next review.

Observed behavior should be able to trigger the loop too. A repeated interaction pattern, an unusual abandonment change, a quality regression, or a latency shift can all be reasons to look again before the next model release.

The trigger starts a review. It does not predetermine the answer.

That distinction matters because "evolve with the model" can otherwise become a lazy rule to make every limit bigger. That would trade one unexamined default for another. A capability-linked product is willing to revisit the decision. It is not committed to increasing the number.

The evidence may say to keep four. It may favor fewer questions because the extra interaction adds cost without improving the result. It may support a higher value, or one that adapts to the task. Any of those can be a good outcome when the decision is current, testable, and owned.

An Upgrade Is More Than a Model Swap

Shipping a stronger model without reviewing the product contract around it is an incomplete upgrade.

The model may be able to do more while the feature continues to present boundaries that nobody has examined recently. Users do not benefit from capability that the product never evaluates, releases, or makes available.

So before calling the next model upgrade done, pick one consequential default. Name its owner. Name the product-level evaluation that could justify changing it. Then name the model release or observed behavior that will cause the team to look again.

The answer may be to leave the default exactly where it is. That is fine. The important difference is that it will be a decision made for the model and product you have now, not a number quietly inherited from the ones you used to have.