The Economics of Thinking: How We Keep AI Costs Under a Penny

Every AI-powered product has a dirty secret: the margin math is brutal. Your users expect the quality of a frontier model, your background jobs need to run on every conversation, and your pricing needs to look reasonable on a landing page. If you throw the biggest model at every problem, you're burning cash. If you use the cheapest model for everything, the quality tanks. The gap between those two extremes is where your product lives or dies.

systimus.io runs background AI operations on nearly every conversation — memory distillation, behavioral calibration, contradiction detection, fact classification. These aren't optional nice-to-haves. They're core to the product. If we ran all of them on a frontier model, the per-conversation cost would make the business unsustainable. So we built a deliberate model-tier strategy, and the constraint turned out to produce better engineering than unlimited budget would have.

The principle: right-size every call

Not every AI call requires the same level of intelligence. Extracting structured facts from a conversation transcript is a different task than generating a nuanced, personality-shaped response to a user. Classifying a memory as "durable" or "ephemeral" is a different task than detecting subtle contradictions between two beliefs. Each task has a minimum viable model — the cheapest option that produces acceptable quality.

We mapped every AI call in the system to one of three tiers. The cheapest tier handles classification, extraction, and structured data generation — tasks with clear right answers and low ambiguity. The middle tier handles evaluation and comparison — tasks that require judgment but within well-defined rubrics. The top tier is reserved for user-facing generation, where quality perception is the whole game.

This mapping isn't static. When a new model drops and the price-performance curve shifts, we re-evaluate which tier each task belongs to. What mattered was establishing the principle that model selection is an engineering decision made per-task, not a product-wide default.

Memory distillation: the volume play

Memory distillation runs after every conversation. It reads the transcript and extracts structured facts — preferences, technical details, behavioral patterns, ephemeral context. The extraction prompt is tightly constrained: output a JSON array, classify each fact, assign confidence.

This is a high-volume, highly structured task. The output format is rigid. The judgment required is modest — is this a preference or a fact? Is it durable or ephemeral? A lightweight model handles this well because the task doesn't require creativity, nuance, or extended reasoning. It requires reading comprehension and classification.

The cost per distillation run is a fraction of a cent. We run it on every conversation. If this task used a frontier model, the cost per run would increase by roughly two orders of magnitude, and we'd have to decide which conversations are "worth" distilling. That's a product compromise we'd rather not make. Cheap enough to run always beats smart enough to run sometimes.

Calibration: cheap enough to run often

Behavioral calibration — measuring whether the agent actually behaved as configured — runs standardized scenarios through the configured personality and scores the output. The scoring step is an AI call: given this scenario and this configured behavior, rate how well the output matches the intent.

Calibration needs to run frequently. Behavioral drift is gradual, and catching it early matters more than measuring it precisely. A cheap model running calibration weekly gives you better trend data than an expensive model running it monthly. We care about the direction of the curve, not the precision of any single point.

The quality tradeoff is real but bounded. A lightweight model's calibration scores correlate well with frontier model scores on the same scenarios — they agree on the direction and approximate magnitude of drift. The absolute numbers differ, but the actionable insight (this dimension is drifting, this one is stable) is preserved. Good enough to detect drift is the bar, not good enough to publish a paper.

Embeddings: the invisible cost center

Every memory gets a vector embedding for semantic search. Every memory recall query generates an embedding. Every contradiction check compares embeddings. Embeddings are the most frequently generated AI output in the system by a wide margin, and most teams underestimate their cost because each individual call is cheap.

The trap is dimensionality. Higher-dimension embeddings produce better retrieval quality, but they also cost more to generate, more to store, and more to search. We settled on a dimension count that balances retrieval quality against the storage and compute cost of running vector similarity at scale. The quality difference between our chosen dimension count and the maximum available dimension count is measurable in benchmarks and unnoticeable in practice.

Index choice matters too. Approximate nearest neighbor search is less accurate than brute-force search but orders of magnitude faster at scale. We accept the approximation because memory recall doesn't need perfect retrieval — it needs fast, good-enough retrieval. If the top-5 results are 95% as good as the theoretically optimal top-5, that's fine. The agent won't notice and neither will the user.

What we chose not to optimize

User-facing generation — the actual conversation with the agent — runs on whatever model the user or provider supports. We don't try to be clever here. When someone is having a conversation with their AI, the quality of that conversation is the product. Cutting costs on the thing the user directly experiences is false economy.

The background operations subsidize the foreground experience. Cheap distillation means the agent has memories. Cheap calibration means the agent behaves consistently. Cheap embeddings mean recall is fast. All of that makes the expensive, user-facing generation better, which makes the product worth paying for.

Infrastructure costs: the other half

Model costs get all the attention, but infrastructure costs are the other half of the margin equation. We chose a single-database architecture — application data, vector search, and background job queues all in the same database. No external vector database, no Redis, no separate job runner infrastructure.

This isn't a scaling philosophy. It's a solo-founder-stage pragmatic choice. Every additional service is another bill, another uptime dependency, another thing to monitor at 2am. The single-database approach means one service to keep running, one backup strategy, one connection pool to manage. The performance ceiling is lower than a purpose-built vector database, but we're nowhere near that ceiling yet, and we won't be for a long time.

When we hit scale constraints, the migration path is clear: vector search moves to a dedicated service, job queues move to a dedicated broker, and the application database goes back to doing what PostgreSQL does best. But premature infrastructure is a tax on velocity, and right now velocity matters more than theoretical throughput.

The margin math

The deliberate model-tier strategy produces a cost structure where background AI operations cost a small fraction of the user-facing generation. That means the product's margin is overwhelmingly determined by the user-facing model cost, which is the one cost that directly correlates with perceived quality.

Everything else — distillation, calibration, embeddings, contradiction detection — adds up to a rounding error on the per-user cost. Not because these operations are unimportant, but because we engineered them to be cheap by treating model selection as a per-task engineering decision rather than a per-product default.

The constraint produced the design. If frontier models were free, we'd probably run everything on them. But they're not, and the discipline of asking "what's the cheapest model that produces acceptable quality for this specific task" led to a system that's more thoughtfully designed than the unconstrained version would have been. Constraints are features.