I Built an AI Telemetry System Because My Memory Is Too Forgiving
My memory is generous to me. It keeps the clean result and quietly misplaces the retries, the abandoned approaches, and the stretches when I was busy without being effective. I built Agent Telemetry because I wanted a record with less sympathy for the story I tell myself about my own work.
The dashboard counts tokens, sessions, and dollars. Those numbers are obvious. By themselves, they tell me very little. A large token total can mean that I completed hard work, repeated weak instructions, chose an expensive model, or simply left several agents running. The total cannot tell those stories apart.
I want evidence that accumulates slowly enough to answer four uncomfortable questions.
Are my engineering standards helping? Am I becoming more effective, rather than merely generating more activity? Are two $200 monthly subscriptions a good economic choice compared with the same observed usage at API list prices? Which projects are taking the most from me in time and attention?
A productive afternoon cannot answer any of them. A productive month may not be enough either. The system preserves comparable history so I can eventually stop guessing.
A standard needs something to answer to
I like explicit standards: acceptance criteria, review passes, tests, evidence, and a clear definition of done. I also know how easily a standard can become ceremony. A process can feel rigorous because it creates a lot of motion. More rounds, more findings, and more generated artifacts may show care. They may also show that the first pass was poor or that the process has learned to feed itself.
Usage data needs an outcome layer. For the governed feature-cycle loop that currently has one, the telemetry records represented specifications, review rounds, findings, accepted outcomes, duration, tokens, and API-equivalent cost. An accepted feature cycle is counted once even if several retained records contain terminal evidence. Acceptance efficiency asks what share of the feature cycles represented in a selected period actually reached acceptance. The metric catalog publishes those definitions and their caveats instead of leaving the labels to intuition.
Over time, I should be able to compare a change in standards with what happened around it. Did accepted cycles become more common? Did the number of rounds fall? Did findings move earlier, or did the loop simply produce more of them? Did cost and elapsed time rise for a result I consider worth the extra scrutiny?
What happened around it is different from what the standard caused. These are observations from real projects, not a controlled trial. Project difficulty, model changes, my own skill, and plain luck can all move at once. A trend can make a cherished rule look suspicious. It cannot convict the rule by itself.
I do not need a perfect causal model to notice months of recurring friction without a visible improvement in outcomes. I need enough evidence to decide what deserves a closer test.
I want to measure effectiveness without defining it as volume
Tokens are wonderfully easy to count. Engineering effectiveness refuses that convenience.
Fewer tokens may mean I learned to frame a problem well. More may mean I delegated a difficult investigation and caught a serious error. Rising output alongside stalled accepted work would leave me with a busier machine, not better engineering.
I care about the relationship among activity, outcomes, and rework. I want to know how many feature cycles reach acceptance, how much back-and-forth surrounds them, what kinds of findings recur, and whether the result holds across projects and time windows. Collection health belongs beside those results. A missing provider root, a late snapshot, or an unobserved field is part of the answer because a clean-looking trend built over a gap is not clean.
The system is strict about a small distinction I value: null means unobserved or inapplicable; zero means an observed zero. Coverage also begins at different times for different hosts, providers, projects, and outcome adapters. That makes the charts less tidy, which is preferable to a smooth line manufactured from assumptions.
The project's operating contract says usage volume cannot establish code quality, developer productivity, causal model superiority, or an invoice. The collector does not read prompts, messages, tool output, or code, and it does not need to. I am not building a surveillance transcript of my working day. I am preserving the metadata needed to ask better questions about it.
The subscription comparison is a counterfactual, not a receipt
I currently pay $200 per month to OpenAI and $200 per month to Anthropic. Four hundred dollars is real spend. I want to know what I receive for it, but “savings” is a dangerous word here.
The telemetry prices observed usage against the providers' published API rate cards. OpenAI and Anthropic publish model- and token-class rates; the project applies a dated price only when an observed model string has an exact match. Usage without that match stays unpriced. Best-effort ranges, when available, remain separate from exact dollars.
This produces an API-equivalent counterfactual: what the observed token mix would cost at the configured list prices. It does not recreate an API invoice, and it does not prove that I saved the difference. My behavior under a flat subscription is unlikely to be identical to my behavior under a meter. Some of the work may never have happened if every request had a visible marginal price.
The comparison still disciplines the economic story. I can see whether the API-equivalent value is routinely below, near, or above my fixed monthly spend. I can amortize those subscriptions across accepted feature cycles without adding that number to API-equivalent cost. I can ask whether a cheaper model remains cheap after review rounds and rework. I can notice when a flattering total depends on a large unpriced bucket.
If the subscriptions appear to deliver many times their monthly cost in API-equivalent usage, that is evidence of utilization, not proof of value. If they appear underused, that is evidence too, though not an automatic cancellation order. Availability, predictable billing, and the option to explore have value that a token ledger does not capture. The metric gives me a harder financial question. It does not volunteer the answer.
Projects spend attention differently
The cost I feel most acutely is rarely the token bill. It is the project that keeps reopening itself in my head.
Agent Telemetry attributes sessions, tokens, cost, host environment, and UTC activity to privacy-safe project identities. The public view keeps private names and paths out of the repository; high-cardinality records remain available through the machine-readable manifest. I can compare projects without turning the public dashboard into a map of my machine.
Compute cost and attention cost are not interchangeable. One project may consume a large model budget while I do little beyond reviewing a strong result. Another may use fewer tokens but demand repeated interventions, context rebuilding, and decisions that linger after the terminal closes. I need both views before I call either project expensive.
Round duration helps, but only as a rough friction signal. It is calculated from dispatch to verdict and includes queue idle. It is not a timesheet, and I will not convert it into labor dollars. Paired with rounds, findings, outcomes, and project allocation, it can show me where to investigate. A project that repeatedly combines long cycles, heavy rework, and weak outcomes deserves a different conversation from one that simply uses a costly model.
Outcome coverage is narrower than global usage coverage today. The governed feature-cycle loop has an explicit outcome adapter; every observed project does not. For projects without one, I can describe activity and allocation, not success. Pretending otherwise would give me a complete dashboard by deleting the distinction I built it to preserve.
The useful version of this dashboard is months away
The public page offers fixed 7-, 30-, 90-day, and all-time windows. The complete daily rows support exact custom UTC ranges. I care about those ranges because a snapshot is mostly biography: what I happened to be doing, with the models and constraints available that week.
A longer record lets me establish a baseline before changing a standard, then inspect comparable periods after it. I can separate a strange project from a broader shift. I can see whether apparent efficiency survives a model change, whether a lower-cost workflow moves work into more review rounds, and whether a project keeps consuming attention after its strategic value has faded.
The record has to earn its upkeep by changing a decision. A standard that adds recurring friction without a corresponding outcome signal should be simplified or tested directly. A supposedly cheap model that produces expensive rework should lose its price advantage in my reasoning, even if the API column stays small. A project that dominates attention while producing little accepted progress should be narrowed, paused, or ended. If I keep everything the same regardless of what the trend shows, I have built decoration.
I also expect the record to improve the questions. Today I have strong global activity coverage and narrower outcome coverage. Future adapters may add project-native evidence, provided they have honest attribution rules. I can tolerate gaps now because the collection records them. I care more that the next observation has somewhere honest to land.
I expect the first genuinely useful finding to be annoying. A standard I am proud of may add rounds without improving acceptance. A project I call strategic may be the main consumer of attention. The subscriptions may look extraordinary in API-equivalent terms, or merely convenient. Any of those results would be more valuable than a large number that agrees with me.
My memory will continue keeping the flattering edit. Months from now, it will have to argue with timestamps, denominators, coverage gaps, and a record that remembers the retries.