we have become very good at measuring financial success.
salary.
net worth.
annual income.
investment returns.
property.
we put numbers against all of these and then use those numbers to decide whether someone is doing well.
but there is another number that rarely appears alongside them:
how much of their time did it take to produce all of this?
that omission matters.
because money is not the only thing we spend.
we spend time to make money.
and unlike money, time cannot be accumulated for later.
we tend to optimize whatever we measure.
if a company is measured primarily by revenue, people inside it will find ways to increase revenue.
if an employee is measured by salary and promotions, they will naturally optimize for career progression.
and if we measure personal success primarily by income and net worth, we should not be surprised when people organize their lives around maximizing them.
the problem isn’t that these are bad metrics.
they are useful.
the problem is that they are incomplete.
imagine evaluating a business purely by its revenue while ignoring how much it costs to generate that revenue.
we would immediately recognize that as an incomplete analysis.
yet we routinely do something similar with people’s incomes.
someone earns ₹50 lakh a year.
great.
but how much time did it take?
how much commuting?
how many weekends?
how much availability outside working hours?
how much mental bandwidth?
how much of the person’s day is actually theirs?
we call the first number income.
perhaps we should also measure the second one.
every rupee has a cost of production.
for a business, that might mean labour, capital, materials, rent and energy.
for an individual, one of the largest inputs is time.
this doesn’t mean that every hour should be converted into rupees.
life would become absurd if we tried to put a price on every dinner with friends or every evening spent reading.
the point is simpler:
earning money requires spending something that cannot be replenished.
and the headline salary doesn’t show the full cost.
consider two people earning the same amount.
one works from home, has predictable hours, rarely thinks about work outside those hours, and can take a month off without much disruption.
the other commutes two hours every day, works late most evenings, answers messages on weekends and cannot really disconnect.
financially, they may look identical.
in terms of life, they are not.
the difference is being paid in a currency that doesn’t appear on the payslip.
there is another reason this matters.
the value of money isn’t constant.
the first ₹10 lakh can be transformative.
it can create an emergency fund.
remove debt.
pay for education.
provide a financial buffer.
change the kind of job you can accept.
the next ₹10 lakh can still be valuable.
but eventually the relationship changes.
going from ₹1 crore to ₹2 crore can be meaningful.
going from ₹20 crore to ₹21 crore may barely change the way someone lives.
the exact numbers are different for everyone.
the principle isn’t.
the marginal value of money generally falls as you already have more of it.
time has a different problem.
you don’t become less in need of time because you already had a lot of it.
an additional free afternoon can still be an additional free afternoon.
an additional year with your children can never be made irrelevant by having enough money.
this creates an important question:
at what point is another unit of money worth less to you than another unit of discretionary time?
there isn’t a universal answer.
but there should be a point where you start asking the question.
money has an extraordinary property.
you can move its usefulness across time.
you can earn today and spend next year.
you can save today and use it decades later.
you can invest today and let it compound while you do something else.
money is remarkably portable through time.
time isn’t.
you cannot borrow an hour from tomorrow.
you cannot transfer an unused afternoon from your 20s into your 40s.
you cannot put this year’s summer into an account and withdraw it when you retire.
money can sit in a bank. time can’t.
this is more than a nice metaphor.
it is a fundamental difference between the two resources.
when you give up money today, you can potentially earn it back tomorrow.
when you give up an hour today, the transaction is final.
this is where the usual idea of “time is money” becomes incomplete.
an hour is not simply a unit of labour.
what that hour can mean depends on when you have it.
at 25, you may have the freedom to move to another country for six months.
at 35, you may have children in school.
at 45, your parents may need more of your attention.
at 55, your physical capabilities and interests may have changed.
none of these stages is inherently better.
but they are different.
the same sixty minutes can exist in all of them while offering completely different possibilities.
this is why I don’t think we should pretend that all future time is perfectly interchangeable.
you can always earn more money later.
you cannot necessarily recreate the circumstances in which you wanted to use your time earlier.
there is a difference between postponing a purchase and postponing an experience.
if you don’t buy a car this year, you can probably buy it next year.
if you don’t invest ₹1 lakh this month, you can invest it next month.
but if you miss your child’s school performance because you were working, there isn’t necessarily a future version of that exact event waiting for you.
if you don’t spend time with your parents while they are healthy, you cannot assume the same opportunity will exist ten years later.
if you spend your twenties working every weekend, you can take weekends off in your forties.
but you cannot recreate your twenties.
some decisions are reversible.
some are not.
time has more irreversible transactions than money does.
that should affect how we think about its value.
this idea can very easily turn into generic advice about quitting your job, travelling the world and “living your life.”
that’s not the argument.
sometimes working extremely hard is exactly the right decision.
early in a career, a difficult job can buy you skills that dramatically increase your future options.
building a company can require years of extraordinary effort.
learning something difficult can demand long periods of concentration.
taking a financial risk can require temporarily sacrificing leisure.
there is nothing irrational about any of this.
the important question is whether the trade is conscious.
what are you getting in exchange for your time?
money?
skills?
ownership?
experience?
freedom later?
status?
something you genuinely care about?
or are you simply continuing because everyone around you is continuing?
this is where the idea of “enough” becomes important.
people often imagine that financial independence means reaching some universal number.
₹5 crore.
₹10 crore.
₹20 crore.
whatever the number happens to be.
but there cannot really be one number.
someone with a simple lifestyle and modest obligations may need much less than someone with expensive tastes and large responsibilities.
“enough” is not an amount of money sitting in an account.
it is the point at which additional money stops changing your life as much as additional freedom would.
that point is personal.
and it can change over time.
but without some concept of enough, the optimization has no natural stopping condition.
there is always another promotion.
another property.
another target.
another level of consumption.
another person to compare yourself with.
the finish line keeps moving because the system was never designed to tell you that you have arrived.
there is a natural progression in how money can work.
early on, you exchange:
time → skills → money
then:
skills → higher income → capital
and eventually, if you’ve built enough capital:
capital → autonomy → time
this is what financial independence should mean beyond the usual spreadsheet definition.
not simply having a large portfolio.
but reaching a point where money gives you more control over your life.
the mistake isn’t exchanging time for money.
the mistake is continuing to make the same exchange forever without checking whether the exchange rate still makes sense.
at 25, another ₹10 lakh might be worth an enormous amount of effort.
at 45, after you have substantial financial security, the same ₹10 lakh might be worth far less than the time required to earn it.
the numbers changed.
but more importantly, your life changed.
there is no notification that arrives one morning saying:
congratulations. you now have enough money. you may begin optimizing for time.
career systems don’t work that way.
companies reward more responsibility.
society rewards visible achievement.
income is easy to compare.
net worth is easy to compare.
time is not.
you can see someone’s car.
you can see their house.
you can ask what they earn.
you cannot easily see whether they control their Tuesday afternoon.
so we keep optimizing the things that are visible.
and because more money is measurable, “more” feels objective.
“enough” feels vague.
this is one reason people can become financially successful without ever becoming time-rich.
they solved the money problem without noticing that money was supposed to help solve the time problem.
perhaps wealth should not be defined only by what you own.
perhaps it should also include what you control.
how much of your week can you decide yourself?
can you take a random Tuesday afternoon off?
can you spend a month somewhere without your income collapsing?
can you say no to a job that pays more but consumes your life?
can you spend an ordinary evening with your family without feeling that you’re falling behind?
these aren’t luxuries in the traditional sense.
they are forms of autonomy.
and autonomy is one of the things people are usually trying to purchase when they accumulate wealth in the first place.
there is a final trap here.
once you start thinking about time as currency, it is tempting to optimize every minute.
that would miss the point.
not every hour needs to be productive.
some hours should be wasted.
some should be spent walking without a destination.
some should be spent talking nonsense with friends.
some should be spent doing absolutely nothing.
the purpose of having more control over your time isn’t to squeeze more output out of every minute.
it’s to have the freedom to decide what deserves your attention.
a free hour does not need to produce anything to be valuable.
sometimes its value is precisely that nobody owns it.
perhaps the most useful way to think about this is not “work-life balance.”
that phrase makes it sound like work and life are two competing boxes that need to be kept at some ideal ratio.
life is not a spreadsheet.
the better question is:
what am I exchanging my time for?
early in life, you may deliberately exchange a lot of time for skills and money.
later, you may deliberately exchange some of that accumulated money for freedom.
the direction of the exchange changes.
that’s not laziness.
that’s not giving up ambition.
it’s simply recognizing that money was never the final objective.
money was a means.
and one of the things it can buy is control over your time.
so perhaps financial success shouldn’t end with the question:
how much money have I accumulated?
there should be another question beside it:
how much of my life do I control?
and eventually:
how much additional money am I willing to exchange my time for?
because the answer shouldn’t remain constant forever.
there may be a period when the trade is absolutely worth it.
there may come a point when it isn’t.
the important thing is noticing when that point arrives.
we spend years learning how to make money.
we learn how to invest it.
we learn how to protect it.
perhaps we should also learn how to spend it correctly.
not just on things.
on time.
because money can be earned again.
money can be invested.
money can wait.
time cannot.
and the strange thing about time is that you’re spending it whether you have decided what it’s worth or not.
so maybe the real question isn’t how rich you are.
maybe it is:
]]>how much of your life are you getting to keep?
the uncomfortable reason some careers compound while others quietly stay linear.
two people can start their careers in almost identical positions.
same age.
similar education.
similar intelligence.
roughly the same number of hours available every day.
five years later, they can be living completely different professional lives.
one has become unusually valuable in a particular area.
they have built things, met interesting people, developed a reputation, and accumulated opportunities.
the other may have worked just as hard.
they may even have more experience.
but their career has mostly remained a longer version of where it started.
it is tempting to explain this with talent, luck, or discipline.
sometimes those things matter.
but there is another variable that is easier to miss.
the two people may be getting very different returns from the same amount of effort.
one hour of work does not have a fixed value.
what matters is what that hour creates.
if the result disappears when the work ends, tomorrow starts almost from zero.
if the result becomes a skill, a system, a relationship, a product, an audience, or a reputation, tomorrow starts with something extra.
this difference looks small in the beginning.
over several years, it can become enormous.
imagine two engineers working on the same team.
both spend their days solving production problems.
one becomes extremely efficient at debugging.
the other notices that the same classes of problems keep appearing.
instead of fixing each problem separately, they start removing the cause.
they write tooling.
they automate repetitive checks.
they document failure modes.
they improve the deployment process.
after a few months, something interesting happens.
the first engineer has become better at solving problems.
the second has made some problems less likely to exist.
both have improved.
but their improvements have different lifetimes.
the first improvement lives mainly inside the engineer.
the second has been transferred into the system.
this is one of the simplest forms of leverage.
and it appears everywhere.
a consultant can repeatedly answer the same question, or turn the answer into a framework.
a teacher can repeatedly explain the same concept, or create material that explains it without requiring their presence.
a developer can repeatedly perform a manual operation, or automate it.
a manager can repeatedly resolve the same class of conflict, or change the process that creates the conflict.
the important distinction is not effort.
it is persistence.
some work produces output.
some work changes the machinery that produces output.
the second kind is unusually powerful because its effects survive the original action.
we tend to treat time as the scarce resource.
it is.
but time is not the only constraint.
the more interesting constraint is often the number of things that can benefit from that time.
consider two five-hour projects.
the first involves helping five people solve five individual problems.
the second involves building a small tool that eventually helps five thousand people solve the same problem.
the second project may fail completely.
perhaps nobody uses it.
perhaps the problem wasn’t important.
perhaps someone else already built something better.
but the possible outcomes are radically different.
this is where asymmetric opportunities become interesting.
some experiments have limited downside and unusually large upside.
you spend a weekend learning a new technology.
you publish something you know about.
you build a small prototype.
you talk to people in an industry you don’t understand.
you contribute to a project outside your normal work.
most of these things will produce little.
that is not necessarily a problem.
if the cost of finding out is small, the experiment does not need a high probability of success to be worthwhile.
the useful question becomes:
what happens if this works, and what happens if it doesn’t?
a failed weekend project costs a weekend.
a successful one might change what you work on for the next five years.
that is a very different equation from spending the same weekend optimizing something whose maximum possible outcome is already known.
there is a reason software has changed so many businesses.
a programmer can spend months building something and then allow a machine to execute the result millions of times.
the work happens once.
the output does not.
this is leverage.
the same principle exists outside software.
a book can be read without the author being present.
an automated process can execute without its creator.
a video can reach people while its creator is asleep.
a product can serve another customer without requiring another complete production cycle.
a distribution channel can send future work to people who already know where to find you.
these are different mechanisms, but they share the same property.
they separate output from the creator’s immediate time.
without leverage, your capacity remains close to:
output ≈ hours worked
with leverage, the relationship becomes closer to:
output ≈ hours worked × reach of what those hours created
the second variable is where large differences begin to appear.
two people can work for eight hours.
one produces eight hours of output.
the other produces something that eight hundred people can use.
the difference is not necessarily effort.
it is the number of places the effort can go.
there is another problem with career planning.
you can become extremely good at solving problems that nobody around you considers important.
this happens because opportunity is not distributed evenly.
some industries are growing quickly.
some companies are full of ambitious people.
some communities sit close to emerging technologies.
some cities concentrate capital, talent, customers, and founders.
some online communities perform the same function without requiring everyone to live in the same place.
what changes in these environments is not just the number of opportunities.
it is the number of opportunities you can see.
imagine two engineers with identical technical ability.
one spends most of their time inside a stable organization where the same problems have been solved for years.
the other spends time talking to researchers, founders, independent developers, and people experimenting with new technologies.
the second engineer will encounter ideas that the first engineer may never hear about.
not because they are smarter.
because their information environment is different.
this is easy to underestimate.
before you can act on an opportunity, you need to know that it exists.
you need to recognize it.
you need to know what to do about it.
the first step is simply exposure.
sometimes the highest-leverage career move is therefore not learning another thing.
it is moving closer to places where interesting things are already happening.
this is also why professional relationships are more valuable than a contact list suggests.
a network allows information to move.
someone hears about a problem.
you know someone who can solve it.
another person is looking for an engineer.
someone has just discovered a market that is growing faster than expected.
another person has already made the mistake you were about to make.
none of this requires a formal networking strategy.
it happens naturally when people repeatedly interact and have enough context to trust each other.
there is also an interesting property of being around people from different domains.
if everyone around you works on the same problems, your information becomes highly redundant.
if your network crosses industries and disciplines, unexpected combinations become possible.
a technical problem can meet a business problem.
a researcher can meet a product person.
a developer can meet a customer who has a problem nobody has built for yet.
these collisions are difficult to manufacture deliberately.
you can, however, increase their probability.
spend more time in environments where different kinds of useful information intersect.
eventually, opportunity depends on trust.
someone has to decide that you are worth betting on.
a company has to hire you.
a customer has to trust you.
a founder has to work with you.
someone has to recommend you to another person.
the problem is that nobody knows exactly how you will behave in the future.
so they look at the past.
what have you built?
what have you shipped?
who has worked with you?
do you finish things?
do you communicate well?
do you keep commitments?
do other people trust you?
this makes visible work unusually valuable.
suppose two engineers have roughly equal technical ability.
one says they are good at building distributed systems.
the other has written about distributed systems, contributed to production systems, built tools, and can show previous work.
the second engineer has created evidence.
that evidence reduces uncertainty.
this is what reputation really does.
it compresses a large amount of previous information into a signal that another person can use quickly.
over time, the signal becomes an asset.
a successful project can help with the next project.
a useful article can lead to a conversation months later.
a recommendation can lead to an opportunity years after the original work.
the strange part is that reputation can keep working even when you are not actively trying to create it.
there is another way people accidentally make their careers harder.
they optimize for isolated expertise.
become a better programmer.
become a better designer.
become a better writer.
become a better marketer.
all useful.
but the interesting value often appears between them.
a programmer who understands customers can make better product decisions.
a designer who understands engineering can design within real technical constraints.
a technical person who can communicate clearly can move ideas across organizational boundaries.
a founder who understands product, engineering, distribution, and sales can operate across several parts of the company without needing every problem translated for them.
none of these people necessarily needs to be the best programmer, designer, writer, or salesperson.
their advantage comes from the combination.
this changes the question you should ask about learning.
instead of:
what skill should I become exceptional at?
sometimes ask:
what skill would make the things I already know significantly more useful?
that produces a different kind of specialization.
you go deep somewhere.
then add adjacent capabilities that multiply the usefulness of that depth.
eventually, the combination itself becomes unusual.
the most confusing part of all this is that none of it looks particularly impressive at the beginning.
learning a skill doesn’t immediately change your life.
writing ten articles doesn’t guarantee an audience.
meeting twenty interesting people doesn’t guarantee an opportunity.
building a small tool doesn’t guarantee users.
reading deeply about a subject doesn’t guarantee a better career.
the returns are often delayed.
but delay is not the same thing as absence.
knowledge accumulated today can make tomorrow’s learning faster.
a reputation developed today can make next year’s introduction easier.
an audience built today can reduce the cost of distributing future work.
a system created today can remove repetitive work for years.
relationships developed today can surface opportunities long after the original conversation.
this is what compounding looks like when the asset isn’t money.
the previous result changes the conditions under which the next result is produced.
this is compounding.
it does not have to involve money.
a useful way to think about it is:
today’s work ↓ something remains ↓ that something improves tomorrow’s work ↓ tomorrow creates another asset ↓ the assets interact ↓ future work becomes more valuable
the first few iterations are often boring.
then the accumulated effects begin interacting.
that is when the curve starts to look different.
some career-changing events are genuinely random.
you meet the right person.
someone discovers your work.
a technology suddenly becomes important.
a market appears.
a company needs exactly the skills you happen to have.
there is no reliable way to schedule these events.
but you can influence how often you are exposed to them.
publish more things.
build more things.
talk to more interesting people.
try unfamiliar projects.
enter new communities.
learn technologies before you absolutely need them.
each action creates another possible path.
most paths lead nowhere.
a few lead somewhere useful.
occasionally, one changes everything.
from the outside, the final event looks like luck.
what is invisible is the number of doors the person had been opening for years.
luck is random.
exposure to luck is not.
this is why two people can work equally hard and still drift apart.
one person’s work is mostly consumed by the present.
the other person’s work is gradually building a structure around the future.
they are accumulating things that remain useful:
skills
systems
knowledge
relationships
reputation
distribution
products
capital
none of these guarantees success.
but each one changes what becomes possible next.
and that is the part that matters.
a career is not just a sequence of jobs.
it is a sequence of increasingly constrained or increasingly expanded possibilities.
if every year requires starting from the same position, progress remains difficult.
if every year begins with more knowledge, more credibility, better relationships, better tools, and more options, the next year becomes structurally different from the previous one.
the goal is therefore not to make every hour maximally productive.
that is too local a metric.
the better question is:
what will this work make easier, cheaper, faster, or possible later?
sometimes the answer will be nothing.
that’s unavoidable.
sometimes the answer will be something small.
and occasionally, the answer will be a completely different career.
those occasional outcomes are enough.
because you don’t need every decision to work.
you need enough of them to change the trajectory.
and once the trajectory changes, the work starts looking different too.
not because you suddenly became more disciplined.
not because you found a secret productivity system.
but because the things you built yesterday are finally participating in what happens tomorrow.
]]>why top-k retrieval is the wrong abstraction for agents
for years, rag has been built around a simple assumption.
modern llms can debug code, orchestrate tools, write production software, reason over long execution traces, and iteratively refine plans. yet whenever they need external knowledge, we still route them through a single retrieval api.
results = retriever.search(query, top_k=10)
this looks harmless.
it is also the most opinionated interface in the entire stack.
the retriever decides what the llm is allowed to observe before reasoning even begins.
that assumption made sense when retrieval was the strongest component in the system.
today, reasoning is rapidly becoming the stronger component.
the architecture, however, has barely changed.
rag was never the goal.
it was an optimization.
engineering workspaces are huge. source code, design docs, runbooks, wikis, slack discussions, postmortems and tickets cannot fit into an llm context window.
retrieval solved this by aggressively reducing the search space.
workspace
|
embedding
|
vector search
|
top k chunks
|
llm
for years this was exactly the right tradeoff.
reasoning was expensive.
context windows were small.
vector search was fast.
there was little reason to question the abstraction.
thinking about retrieval as a model hides the more interesting observation.
retrieval is an interface.
every interface determines what information is available to downstream consumers.
the retrieval interface exposes exactly one capability.
given a query, return the best ranked chunks.
it does not expose uncertainty.
it does not expose alternative search paths.
it does not expose near misses.
it does not allow iterative narrowing before information is discarded.
that means the llm never reasons over the corpus.
it reasons over whatever survives retrieval.
this is an architectural constraint rather than a reasoning limitation.
modern rag pipelines repeatedly compress information.
workspace
-> chunking
-> embeddings
-> nearest neighbour search
-> top k
-> llm
every stage removes information.
chunking removes surrounding context.
embeddings remove lexical precision.
ranking discards candidates.
top k discards everything else.
none of these choices are incorrect.
they are practical engineering optimizations.
but every optimization introduces an abstraction boundary.
traditional rag assumes a linear execution model.
question
-> retrieve
-> reason
-> answer
agents rarely operate this way.
they search.
inspect.
form hypotheses.
revise.
verify.
search again.
their execution resembles software debugging more than document retrieval.
this is already visible in coding agents like claude code, codex, aider and openhands.
they spend much of their execution inside terminal tools.
they search.
inspect.
trace.
repeat.
the search process itself becomes part of reasoning.
this is the inversion proposed by the paper.
historically, the retriever owned the search policy.
the llm simply consumed whatever was returned.
direct corpus interaction flips that ownership.
the llm now owns the search policy.
terminal commands become primitive operations rather than complete retrieval solutions.
rg.
grep.
find.
sed.
cat.
head.
tail.
individually they are extremely simple.
collectively they become building blocks for adaptive search.
consider two interfaces.
retrieve(query)
versus
rg
find
grep
sed
cat
python
the first exposes one operation.
the second exposes primitives.
this is reminiscent of the unix philosophy.
small deterministic tools.
simple interfaces.
arbitrary composition.
the intelligence does not live inside grep.
it emerges from how the llm composes these primitives.
an investigation might look like
rg “kv cache”
-> discover paged attention
-> inspect implementation
-> discover continuous batching
-> search again
notice something subtle.
none of these searches attempted to answer the original question.
each search attempted to discover the next search.
the search strategy is synthesized online.
the deepest idea is retrieval interface resolution.
traditional retrieval exposes documents or chunks.
direct interaction exposes symbols, functions, lines, neighbouring context and exact lexical matches.
resolution is about control rather than volume.
higher resolution allows an agent to progressively narrow its investigation instead of repeatedly asking for another ranked list.
the abstraction becomes interactive rather than transactional.
the surprising result is not simply higher benchmark scores.
the interesting observation is that better performance often does not come from surfacing more relevant documents.
it comes from extracting more value after reaching a relevant document.
once the agent reaches useful evidence it performs local exploration.
inspect.
extract entity.
search again.
verify.
continue.
that behaviour is difficult to express through a fixed retrieve api.
none of this means retrieval disappears.
search primitives optimise search depth.
retrievers optimise search breadth.
as corpora become extremely large, exhaustive terminal exploration becomes expensive.
tool calls increase.
latency grows.
cost rises.
there is still a regime where retrieval remains the better engineering tradeoff.
the point is not replacing retrieval.
the point is questioning whether top k retrieval should remain the only interface between reasoning and information.
the contribution is bigger than replacing embeddings with grep.
it reframes retrieval as an interface design problem.
for years we pushed intelligence into retrievers.
agentic systems suggest pushing intelligence into orchestration instead.
once the llm becomes the strongest reasoning component in the stack, perhaps it should own search as well.
Original Paper: Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction
]]>modern ai abstractions are extremely convincing.
from tokenization to datacenter cooling, the entire inference stack is basically an optimization war against physics.
you send text into an api.
a few milliseconds later, coherent language comes back out.
the interface is polished enough that people start imagining the model as some continuously reasoning entity sitting behind the endpoint.
in reality, most production inference systems are closer to numerical factories than digital minds.
underneath every chatbot sits a pipeline that:
the interesting part is not just the math.
it is the chain of engineering constraints that shaped the entire stack.
because every layer in modern llm infrastructure exists to solve a limitation introduced by the previous one:
once you follow the execution flow carefully, most of the “magic” starts collapsing into systems engineering.
when you download a modern language model, most of the size comes from files like:
model.safetensors
despite the branding around ai, this file is fundamentally just structured numerical storage.
inside it are tensors.
a tensor is simply a container for numbers:
machine learning vocabulary makes tensors sound more exotic than they are.
operationally, the file mostly contains weights:
older pytorch checkpoints relied on pickle serialization through .bin files.
pickle could reconstruct arbitrary python objects during loading, which also meant loading a model could potentially execute arbitrary code.
.safetensors exists because model loading eventually needed to become:
so the format became intentionally restrictive:
the important part is this:
nothing inside the file resembles:
the model stores numerical relationships, not human-readable understanding.
models do not understand words.
they only process numbers.
so before inference begins, text must first become numerical identifiers.
this is the tokenizer’s job.
a tokenizer creates a mapping between text fragments and integer ids.
for example:
{
"system": 345,
"observ": 8763,
"ability": 992,
"critical": 2042
}
tokens are not always complete words.
modern tokenizers usually split text into reusable fragments because storing every possible word would create an impractically large vocabulary.
so:
observability
may become:
["observ", "ability"]
this allows related words to reuse components:
all partially overlap.
which means tokenization is solving two problems simultaneously:
the tokenizer therefore shapes:
bad tokenization leaks inefficiency into the entire model pipeline.
this relationship is critical.
the tokenizer determines:
the model then trains against those exact ids.
if:
"system" → 345
then row 345 inside the model’s embedding table gradually learns the statistical representation associated with "system".
changing the tokenizer changes the mapping.
which means the model would retrieve the wrong internal representation entirely.
this is why checkpoints always ship with:
together.
they are components of the same learned system.
before the model itself learns anything, engineers first train the tokenizer.
they feed enormous text corpora into tokenization algorithms:
the tokenizer searches for reusable fragments that maximize efficiency.
suppose the dataset repeatedly contains:
monitor
monitoring
monitored
monitors
the tokenizer may conclude that "monitor" should exist as a reusable token because it appears frequently across multiple contexts.
so it creates:
"monitor" → 5421
later during model training, token 5421 repeatedly appears in relevant contexts, allowing the model to gradually learn useful statistical patterns around it.
this entire process exists because vocabulary size is a hard constraint.
larger vocabularies:
so tokenization becomes an optimization problem before the model even exists.
after tokenization, text becomes integers:
[345, 8763, 992, 2042]
but integers themselves are semantically empty.
345 does not inherently mean "system".
it is simply an index.
the next problem becomes:
how do you convert token ids into representations the model can actually reason over numerically?
this is where embeddings enter the pipeline.
inside the model exists a large embedding table.
conceptually, it behaves like a spreadsheet:
if:
then the embedding table contains:
during inference, if:
"system" → 345
the framework retrieves:
embedding_table[345]
which returns something like:
[-0.12, 0.88, 0.43, ...]
this vector becomes the model’s internal representation of the token.
the important detail is that these dimensions are not manually labeled.
there is no column representing:
the structure emerges statistically during training.
initially these vectors are mostly random numbers.
over billions of training examples, tokens appearing in similar contexts gradually develop related numerical patterns.
the model never learns definitions.
it learns proximity and correlation.
before transformers, sequence models like rnns and lstms processed text sequentially:
this worked for short sequences.
it worked poorly for long ones.
information from earlier tokens gradually degraded as sequences became larger.
which meant relationships between distant words became difficult to preserve.
transformers changed this by introducing attention.
instead of compressing history into a single evolving state, attention allows tokens to directly compare themselves against other relevant tokens in the sequence.
given:
the database server crashed because it ran out of memory
when processing "memory", the model can directly evaluate relevance against:
instead of hoping those relationships survived multiple sequential transformations.
attention mechanisms repeatedly:
this architecture dramatically improved long-range dependency handling, which is one of the main reasons transformers displaced older sequence models so aggressively.
llms generate text one token at a time.
suppose the model generates:
the server crashed because
to predict the next token, the model processes the entire sequence.
after generating another token, it processes the now longer sequence again.
without optimization, generating token 500 would repeatedly recompute large portions of tokens 1–499 again and again.
the redundancy becomes enormous.
this is why autoregressive inference becomes expensive very quickly for long contexts.
during attention, tokens produce internal representations called:
once computed, these representations do not change.
so recalculating them repeatedly during generation is unnecessary.
kv caching stores them directly in gpu memory and reuses them across future generation steps.
instead of recomputing the entire sequence every time, the model only processes the newest token.
this dramatically reduces compute overhead.
the tradeoff is memory pressure.
longer conversations require larger kv caches, which is why:
optimization simply relocates the constraint.
most people assume llm inference is constrained primarily by compute.
in reality, modern inference is often constrained by memory movement.
a 7-billion-parameter model stored in fp32 requires roughly:
7 billion × 4 bytes ≈ 28GB
during inference, large portions of those weights continuously move between:
throughput therefore becomes limited by:
how fast hardware can move data
not necessarily:
how fast it can multiply numbers
this distinction matters because once memory bandwidth becomes dominant, architectural priorities shift completely.
quantization reduces numerical precision to shrink model size.
instead of storing weights as 32-bit floating point values, weights may be compressed into:
this reduces:
while preserving most practical inference quality.
neural networks are statistically tolerant systems.
they usually do not require perfect decimal precision to preserve broader learned relationships.
quantization therefore became operationally necessary because hardware transfer costs eventually dominate inference economics.
once enough gpus participate in training or inference, software stops being the only concern.
modern accelerators consume enormous amounts of power because large-scale matrix operations are physically expensive.
high-density gpu servers routinely draw:
all of that energy eventually becomes heat.
which means modern ai datacenters increasingly optimize:
even machine placement matters.
distributed training constantly transfers tensors between gpus across high-speed interconnects. once clusters become large enough, network topology and interconnect latency materially affect training efficiency.
at sufficient scale, ai infrastructure starts resembling industrial power engineering as much as software engineering.
because every generated token ultimately depends on:
the inference pipeline is mechanically straightforward:
text
→ tokenization
→ token ids
→ embedding lookup
→ vector transformations
→ attention calculations
→ probability prediction
→ next token
→ repeat
everything else in modern ai systems exists to optimize this loop:
which is why production ai engineering increasingly sits at the intersection of:
because eventually every “intelligent” response still has to survive contact with physics.
]]>state, scale, and cached lies.
we argue about tokens when we should be arguing about topology. here is how your architecture actually forces your hand.
most identity writing starts too late.
by the time your engineering team is locked in a meeting room debating the merits of jwts against opaque tokens, the most critical architectural constraints have already been locked in. the servers are distributed. the latency budgets are tight. security is screaming for instant revocation while product is demanding a seamless, invisible login experience.
identity is not a menu of isolated design patterns you pick from. it is a sequence of forced responses to friction. every new layer in your stack exists solely because the simpler model before it buckled under the weight of scale.
if you want to understand why our systems look the way they do, you have to watch them break.
think back to the simplest version of your app. one backend server, one database. a request comes in, the server hashes the provided password, checks it against the database, and serves the payload. the system that knows the truth is the exact same system enforcing it. identity is not a domain here; it is just a sql query.
but you hit your first wall almost immediately. cryptographic password hashing is intentionally slow. running a bcrypt validation on every single http request will melt your cpu. asking a user for multi-factor authentication on every click will ruin your product.
so, you compromise. you decide to perform the expensive mathematical proof exactly once, record the result in memory, and hand the client a random string to represent that successful check.
you just invented the session.
the cookie is not the identity. it is just a pointer to a cached decision. you traded compute cycles for memory space, and it works perfectly… until you need a second server.
traffic spikes. you put ten application instances behind a load balancer. suddenly, a user logs in on node a, but their next api call lands on node b. node b has no idea who they are because the session state is trapped in node a’s memory. continuity shatters.
the obvious engineering fix is to pull that state out of the application and put it into a shared central store like redis. now, any node can verify the session.
but look at what you just did to your architecture. your identity store is now sitting directly in the critical path of every single request. if your api spans multiple regions, latency creeps up as nodes reach across the country to verify a session. if the cache cluster blips, the entire platform goes down. you solved the memory isolation problem, but you created a massive central bottleneck.
in distributed systems, local decisions are cheap, but central coordination is expensive. the next evolution has to preserve trust without making a network call.
if your microservices cannot afford to constantly phone home to a central database, the proof of trust must travel alongside the request itself. you need a way for the auth server to make a statement that any downstream service can verify completely offline.
this is exactly how jwt enters your stack.
you package the user claims, sign them with a private key, and hand the package to the client. when a microservice receives that token, it does not look it up in a database. it simply runs a deterministic mathematical check using a public key. if the math checks out, the token is trusted. you just eliminated the central bottleneck.
but then a real-world scenario hits. it is 2:00 pm, and a rogue employee is terminated. hr deletes their account. but their jwt was issued at 1:00 pm and does not expire until 5:00 pm. the token is cryptographically flawless, but operationally toxic. the ex-employee spends the next three hours pulling internal repositories.
this is the hard lesson of stateless tokens. a signature only proves the token was issued correctly in the past. it guarantees integrity, not freshness. a jwt is essentially a cached lie with an expiration date. you are now stuck managing this gap, balancing agonizingly short token lifetimes against complex refresh flows, just to simulate the instant revocation you used to get for free with sessions.
eventually, your application becomes a platform. a third-party partner wants to build an integration that syncs user data.
historically, users solved this by just giving the partner their raw password. that is a structural disaster. a password is absolute authority. you cannot scope it, you cannot restrict it to “read-only,” and sharing it collapses your entire security perimeter.
you need a valet key. you need a way to grant a limited, highly specific capability without transferring primary credentials.
this forces the adoption of oauth. it fundamentally separates the proof of who you are from what you are allowed to do. the user authenticates with your central server, and the third party receives a token that only allows them to read calendar events for the next hour. you stop sharing identity secrets and start issuing constrained capabilities.
fast forward a few years. your company has grown. you now have an internal wiki, an expense portal, a ci/cd dashboard, and a dozen other tools.
if every single one of those applications maintains its own local user database and login screen, the operational overhead becomes crushing. when an engineer leaves the company, it has to manually hunt down and revoke access across twenty different systems. a single missed account is a critical security breach waiting to happen.
you realize that managing identity per application does not scale. you have to centralize authentication the same way you centralized your logging or your network routing.
this is where sso becomes inevitable. you abstract the authentication layer entirely. the internal apps no longer care about passwords; they just trust the central identity provider. the visible benefit to the user is fewer login prompts. the actual engineering reality is a single, unified control plane for risk.
sessions, stateless tokens, delegated access, and federation are not competing ideologies. they are an evolutionary chain. they solve the distinct consequences of scaling up compute, state, network, and organizational sprawl.
the hard part of identity architecture is not proving who a user is on the first click. the hard part is deciding where the truth lives, how long you are willing to let downstream systems cache it, and exactly how much latency you are willing to pay to revoke it when things go wrong.
]]>ai did not change engineering. it removed the excuses.
i have written systems where a small mistake took hours to trace.
i have also shipped work where most of the code was generated and reviewed only at the boundaries.
output increased, hence the need for judgment increased faster.
that is the shift.
there is a lot of noise about ai writing code.
writing code was never the hard part.
the hard parts are:
example.
a product asks for “real time notifications”.
implementation is easy. websockets. pubsub.
the real questions are:
how many connections per user
what happens when the service restarts
do we guarantee delivery
how do we prevent duplicate events
what is the acceptable delay
ai can write the socket handler.
it cannot define the system contract.
in most production environments, engineers already spend more time thinking, aligning, and debugging than typing.
ai accelerates implementation.
engineering is not implementation.
in real usage, the value is straightforward:
this removes a large amount of mechanical effort.
boilerplate
crud layers
test scaffolding
migrations
basic integrations
example.
you need an internal admin panel for managing users.
routes
forms
validation
api wiring
ai can generate the whole thing in minutes.
but then requirements change.
bulk operations
audit logs
role based access
export for compliance
the first version was easy.
the system is where the work begins.
when the cost of building drops, the amount of software requested increases.
every productivity jump in this industry has done the same thing.
capacity expands. demand expands faster.
ai produces the statistically likely solution.
systems fail because of the unlikely conditions.
this mismatch appears quickly.
structure degrades
without strict direction, generated code accumulates:
example.
three features are added over a month.
each generated separately.
now there are three ways to access the same data.
no single owner.
small changes require touching multiple paths.
complexity grows faster because generation is cheap.
defects move to the edges
the common paths work.
edge cases fail.
example.
a payment service works in testing.
in production:
timeout after charge
retry triggers duplicate payment
user is charged twice
the code was correct locally.
the system behavior was wrong.
someone still needs a correct mental model.
no long term intent
when something breaks, the model optimizes for the local fix.
example.
a test fails due to timing.
the generated fix increases timeout.
tests pass.
latency doubles in production.
local success. system regression.
strong engineers are not limited by typing speed.
they are limited by decision quality.
they carry working models of:
dependencies
failure modes
data movement
operational constraints
business impact
example.
a feature requires joining data from two services.
quick solution: synchronous call.
correct solution: async pipeline.
the difference shows up under load.
ai can write either.
it cannot choose based on future traffic patterns.
when code generation becomes cheap, judgment becomes the scarce resource.
there used to be developers who assembled features without understanding the system.
now there will be developers who generate entire systems without understanding them.
they move fast.
until something non obvious fails.
example.
a search feature is generated using full table scans.
works with 10k records.
fails at 10 million.
fixing it later requires redesign, reindexing, and downtime.
speed becomes baseline.
understanding becomes differentiator.
faster development does not reduce software needs.
it changes expectations.
when building becomes cheap:
one system becomes many variants
defaults become customization
team tools become individual tools
example.
earlier one dashboard served the whole company.
now every team wants:
custom metrics
custom alerts
custom workflows
each version is small.
operational surface area grows quickly.
generated systems still require:
ownership
monitoring
integration
security
cost management
continuous change
building software is the cheapest phase.
operating software is where the cost lives.
the value is moving away from implementation.
the leverage is in:
defining system boundaries
maintaining architectural consistency
setting constraints for generated code
reviewing for long term impact
diagnosing failures under uncertainty
aligning technical decisions with business risk
example.
without guardrails, teams generate services directly hitting the database.
short term speed.
long term coupling.
schema change becomes a company wide incident.
senior impact has never been about writing more code.
it has always been about preventing expensive mistakes.
more code generation means more opportunities for expensive mistakes.
if your value is based on:
syntax knowledge
framework familiarity
writing standard features
you are competing with automation.
if your value is based on:
system design
tradeoff decisions
debugging under ambiguity
long term ownership
your leverage increases.
the role is shifting.
less time producing code.
more time controlling what enters the system.
ai will improve.
larger context
better reasoning
faster generation
but one constraint remains.
when production breaks, someone is accountable.
example.
cpu spikes
latency increases
revenue drops
the question is never:
who wrote the code
the question is:
who understands the system
accountability requires understanding and tools do not carry responsibility.
engineers do.
ai is not removing engineers.
it is removing the value of low judgment work.
when anyone can generate code, the industry stops rewarding people who produce code.
it rewards people who keep systems stable under change.
read that again.
the future belongs to engineers who control complexity.
]]>when preset metrics lie to you
there is a moment every llm engineer quietly dreads.
a pm asks, “so… did the new ranking logic make answers better?”
the dashboard shows:
but when you read real conversations, users are confused. support tickets keep climbing. the “improvement” is just noise.
this post is about that gap. why preset metrics fail. and why custom llm-as-a-judge metrics, built around your product’s real definition of good, are the only way out.
bleu, rouge, exact match, token f1. they come from a simple world.
small test sets. one ground truth answer. short outputs. goal is similarity to reference.
translation or summarization? fine. llm products? no.
we build systems that explain. reason. obey tone, compliance, safety rules. balance correctness, brevity, style.
rouge cannot spot a legal chatbot breaking policy. bleu misses a code assistant suggesting a vuln. exact match ignores if a summary skips the key clause.
metrics give clean numbers. but deeper the product, less those numbers match what users feel.
three patterns.
rag chatbot for internal docs. users need correct, grounded answers. no hallucinations. enough context to trust. no single ground truth wording.
code-review assistant. fewer targeted changes. catches race conditions. says “good as-is” when right. token overlap with reference? meaningless.
support triage. routes right queue. de-escalates. never mislabels legal complaints. error costs vary wildly.
quality here is multi-dimensional. tied to human judgment. not a scalar bleu can fake.
teams end up with dashboards that do not match their gut on samples. metrics answer the wrong question.
natural fix: use a strong llm as proxy human rater.
feed prompt, answer, maybe reference. give eval prompt and rubric. get score. maybe explanation.
unlocks what presets cannot: logic, helpfulness, safety, faithfulness, reasoning quality, format.
first run on real data? magic. flags hallucinations. explains misleading bits. spots “correct but useless”.
busy teams love it. tireless junior reviewer at api speed.
then cracks show.
goodhart first. train against judge, model learns judge-pleasing tricks. verbose fluff. fake confidence. hides uncertainty. product worsens, metric climbs.
vague rubrics. “rate 1-10”. chaos. high variance. drifts with model versions. confuses experiment compares.
judge bias. favors formal verbosity. misses domain facts. encodes training biases.
leakage. judge sees hidden metadata. penalizes user-ok answers. rewards weird conformity.
opacity. “score 0.83”. no tags or reasons. hard to debug. communicate. prioritize.
llm judge without design is fancy bleu.
shift: metrics are product decisions in text.
good ones start from user pain. break quality into failures. attach costs. explicit rubrics. living artifacts.
metric prompt is spec. “10 if fully correct and grounded”. “0 if fabricates”. explain violations.
version them. changelog. review.
rag compliance q&a.
matters: no hallucinated rules. quotes right policy. conservative on uncertainty.
judge reads question, answer, retrieved policy. scores faithfulness, fabrication, alignment. tags safe/unsafe.
code gen assistant.
functional ok. project style. maintainable. no sec bugs.
judge sees code, diff, tests. simulates senior reviewer.
support bot.
resolves alone. escalates with context. no wrong promises. de-escalates.
labels convos. maps to real kpis: resolution time, refunds, nps.
tooling buckets:
preset bundles. easy. rarely aligned.
raw judge wrappers. powerful. bad at versioning, combining, structure.
platforms. scalars per run. weak on failures, ci, drift.
missing:
metrics as versioned objects. names, owners, rubrics.
playgrounds to tweak rubrics. spot disagreements.
composability. faithfulness + helpfulness = product rating.
meta-eval. judge vs human correlation.
opportunity: better ways to express “good”. connect to model behavior.
dataset: reality sample.
llm: hypothesis.
metric: team judgment interface.
vague metric = bad api. brittle = coupled api. good = documented, stable.
llm executes spec. spec is asset.
principles.
start real failures. not synthetics.
name failures: halluc, verbose, under-explain.
rubric explicit. top/mid/zero cases. examples.
metrics as code. git. reviews.
validate vs humans. spot disagrees.
multi-scores. decide balances.
preset metrics fake science. avoid hard questions: what is good here?
custom forces it. tolerable mistakes? style? risk level?
encode that. experiments match gut. team aligns. “improved?” means something.
next feature live. skip rouge comfort.
ask: success feel? failure nightmares? test criteria? judge needs?
tools evolve. leverage is encoded judgment.
llm can judge. values come from you.
]]>a song about seeing someone and realizing you are being seen too
intro
judging you juding me
every stare, a memory
what i see might just be me
glass between us, clarity.
verse 2
two reflections, no referee,
say nothing, still you’re hearing me,
you look once, then carefully,
tension writes the chemistry.
verse 3
judging you, judging me,
same reflection, different frequency,
glass between us, clarity,
what you see is what you be.
verse 4
no confession, no decree,
just this mutual scrutiny,
judging you, judging me,
till the mirror sets us free.
outro
judging you, judging me,
end the trial, set it free,
if the mirror broke tonight,
we’d still know what we see.
this song is about silent moments where two people are aware of each other without speaking. it is not about criticism or judgment in the usual sense. it is about observation, awareness, and recognition.
when i say judging you judging me, i mean the quiet evaluation that happens through eye contact, pauses, and body language. both people are reading each other at the same time. no one is in control and no one is innocent. there is no referee because both sides are equally involved.
the mirror and glass imagery represents distance that still allows clarity. there is a barrier, but it is transparent. you can see the other person clearly, and in doing so, you start seeing yourself just as clearly. the tension in the song comes from that realization.
what i want to convey is that many times we think we are reacting to others, but we are actually confronting parts of ourselves. attraction, curiosity, and discomfort often reveal more about us than about the person we are looking at.
the song does not try to resolve this tension with a confession or a conclusion. instead, it accepts it. the trial ends not because someone wins, but because both sides recognize the truth. even if the mirror breaks, the understanding remains.
this song is about clarity without explanation, connection without pursuit, and self recognition through someone else’s eyes.
]]>and why it will cost you months when you finally do
a patient sync failed silently four hours ago. not a crash. the logs were fine. the api was responding in 45ms. the database was healthy. everything looked green on the dashboards.
but four hours later a clinician pulled up a patient’s medication history and got stale data. made a clinical decision on that. the sync had never happened. the system had no way to tell us.
this is the thing i keep running into. infrastructure observability and domain observability are not the same thing. i spent years thinking they were. they’re not.
early in my career i assumed that if you logged errors and tracked request latency and monitored cpu, you had observability. that was the setup i inherited at my first real job. dashboards everywhere. alerts for everything. we knew when the system was struggling.
we had no idea what the system was actually doing.
i remember needing to understand why prescriptions were being rejected at high rates. went to the logs. found a hundred different rejection messages, all logged differently, some with patient data, some without, some with the drug name, some just saying “validation failed.” nobody had coordinated on how to log this. each developer who touched the code added their own approach.
so i started adding more logging. got more consistent. then someone needed to understand if certain drug interactions were being caught. so i added events. then analytics wanted to track something. so we added event tracking. after a few years the prescription validation code had logging calls and metrics emission and analytics event firing woven through the entire path.
the code worked. but reading it felt like reading a debugging session. it was hard to understand what the actual validation logic was doing because everything was buried under observability calls.
we decided to migrate to a different observability platform. this should have been a tool swap. it turned into a nightmare.
the observability calls were so embedded in the business logic that we couldn’t just swap the client library. we had to touch the prescription validation code. we had to test it carefully. we had to coordinate across teams. it took weeks.
i remember thinking: i’m rewriting critical code to change a logging tool. how is this my problem?
that’s when i realized the issue wasn’t the logging. it was where the logging lived.
after that project i noticed the same pattern everywhere i looked. teams would build a feature, add observability to it, and six months later the business logic and the observability logic were so tangled that you couldn’t change one without being careful about the other.
sometimes it was because someone needed to add a new field to an event and it required understanding how the business logic worked. sometimes it was because the observability wasn’t giving you the context you needed, so people started adding more and more parameters to functions, just to pass context through.
i watched teams decide they needed better visibility into their systems, so they added more instrumentation. the code got more complex. testing got harder. but the visibility didn’t improve. they still couldn’t answer the question they cared about.
what became clear was that infrastructure observability and business observability are operating at different layers.
infrastructure observability tells you if your system is struggling. cpu is high. memory is climbing. requests are slow. outages.
business observability tells you if your system is working. did the prescription get to the pharmacy? did the lab result reach the ordering provider? did the payment actually complete?
most teams build infrastructure observability well. they get alerts for cpu, memory, errors. but business observability requires thinking about what actually matters in your domain. and that has to be deliberate.
i noticed that the teams doing this well weren’t building infrastructure observability differently. they were separating it from the business logic. they created a layer whose job was to translate domain events into whatever observability platform they were using.
the pattern i started seeing was simple. the core business logic reported what happened in domain terms. a prescription was validated. a prescription was rejected due to interaction. a prescription was flagged for manual review.
instead of having the validation code call logging functions and metrics clients and analytics libraries, it reported these observations to a separate object whose sole job was to handle observability.
that observer could log it. could emit it to metrics. could send it to analytics. could store it in a database. the business logic didn’t care. it just reported what happened.
the observer could also handle context. when a request came in, we extracted all the metadata we needed. patient id, provider, facility, request id for tracing. we configured the observer with that context once. the business logic never had to think about context. every observation it made automatically carried all the necessary metadata.
this separation meant that the business logic stayed clean. it was just validation logic. pure. testable in isolation. readable.
it also meant that if we wanted to change how we observed things, we changed the observer, not the business logic.
i expected this to require more code. it doesn’t. it requires less code in the places that matter.
the prescription validation code is smaller. easier to read. easier to test. the observability logic lives somewhere else and is focused on one job: translating domain events into infrastructure events.
the bigger surprise was how much better the observability actually was. because now all the observations were being made from the same place, with the same context, following the same patterns. when i needed to understand which prescriptions were failing, the data was consistent. no more digging through logs that were all formatted differently.
i’ve watched teams try this approach and fail because they didn’t handle context well. they ended up passing patient_id and provider_id and facility_id through every function signature. the code became messier, not cleaner.
the systems that got this right configured the observer once at the boundary. when the request came in, you extracted context. you created an observer pre-configured with that context. you passed that into the business logic. now the logic never had to think about context.
it sounds simple. it changes everything.
i’ve seen the alternative. i’ve lived it. you build a system with observability scattered through it. it works. you add more observability because you need better visibility. the code gets more complex. the problem you’re trying to solve doesn’t actually get solved because the observability is at the wrong layer.
then someone wants to migrate to a different platform. or wants to change how you’re logging. or wants to add a new field to an event. and suddenly you’re touching critical code. taking on risk. spending weeks refactoring something you shouldn’t have to refactor.
i’ve measured this cost. it’s weeks. sometimes months. i’ve watched engineers spend more time refactoring observability code than they spent building the original feature.
what i keep coming back to is that observability is an architectural decision, not a tool decision.
you can pick the best logging library and still get observability wrong if it’s mixed into your business logic. you can pick the worst metrics client and get observability right if you’ve separated the concerns.
the systems i’ve seen age well are the ones where someone early on decided: observability is a design layer. the way we separate persistence from business logic, we separate observability from business logic.
that decision ripples through everything. how you test code. how you reason about code. how you change code. whether you can migrate platforms without rewriting critical logic.
i keep seeing teams solve the wrong problem. they get better monitoring tools. better dashboards. more logging. and still can’t answer the basic questions about their system.
i also keep seeing teams pay months of engineering time to refactor observability that should never have been embedded in the business logic in the first place.
it’s not about picking the right tool. it’s about where the logic lives.
the next time you’re adding observability to a critical path, ask: is this business logic or is this observability logic? if they’re mixed, separate them. not later. now.
because that separation is what lets you change your system later. that separation is what makes your observability actually useful. and that separation is what keeps you from paying months to fix something that should have been designed right the first time.
]]>building your own auth control plane: a pragmatic analysis before you buy
there’s a moment in every growing company when someone suggests using a managed authentication platform. it’s a reasonable suggestion. these platforms exist. they’re stable. they have funding. they have teams of people thinking about security.
and then you realize you’re going to need to customize almost everything about it.
before you even look at a platform, sit down with your architecture and ask a specific question: what percentage of my auth problem is “i need to sign tokens” versus “i need to decide who gets tokens and what they can do with them?”
the first part is 15 percent of the work. it’s also the part that every platform does. rsa signatures, jwt encoding, key rotation. this is solved. genuinely solved. you don’t need to think about it.
the second part is 85 percent of the work. and that’s where managed platforms start getting interesting because they’re not actually built for the shape of your problem.
think about what you need to decide:
these aren’t authentication questions. they’re policy questions. and every organization has different answers.
here’s what actually happens when you integrate a managed auth platform into a microservice architecture.
your identity provider issues a token with a specific format. your services need to validate it. you have two choices:
option one: every service calls the platform’s validation endpoint on every request. this works until you do the math. you’ve got fifty services, each handling a thousand requests per second. that’s fifty thousand validation calls per second going to someone else’s infrastructure. your auth latency is now their network latency plus their database latency. you also now have a very specific dependency: if their validation endpoint is slow, your entire system is slow.
option two: you cache the validation decision locally. now you’re managing cache invalidation. you’re managing staleness. you’re hoping that when a token gets revoked, the revocation propagates before someone uses a cached validation result. you’re also managing the consistency model. if a token gets revoked and a service has a stale cache entry, what happens? this is where the operational overhead starts compounding.
neither option is wrong, but both have costs. costs that a platform doesn’t really talk about when they’re trying to sign you up.
then there’s the feature gap. you want to issue tokens with different ttls based on the client’s tier. the platform has one ttl setting. so you write code to adjust tokens after the fact. you want to track which client issued how many tokens. the platform logs this somewhere, but getting the data out requires hitting their api or exporting everything. you want to add a scope-based rate limit. they have rate limiting, but it’s per user, not per scope.
what you’re building is a shadow auth system. you’re running their auth in parallel with custom code that makes it do what you need. and now you’ve got two systems to maintain instead of one.
let’s think about what changes when you decide to build the token policy layer yourself.
first, the schema becomes your design problem. you’re not fitting your needs into their token structure. you’re designing a token structure that fits your needs. need custom claims? add them. need tenant information? add it. need a way to encode the service tier that issued the token? you control that.
second, validation becomes your operational responsibility. but that also means you control it completely. you can validate tokens in-process using your service’s public key. zero network hops. zero cache inconsistency problems. your validation latency is bounded by your crypto library, which is typically microseconds.
third, the audit trail becomes actually queryable. you design the schema. you decide what gets logged. you can ask questions like “how many tokens did client x issue yesterday?” without exporting anything. you can ask “which services called which services?” without reconstructing it from logs. this matters more than you think when you need to investigate a security incident.
fourth, revocation becomes something you control. want to revoke a token in under a second? you can. want to revoke all tokens for a client immediately? you can. want a revocation that cascades to related tokens? you can implement that. managed platforms give you revocation, but it’s revocation designed for their general case, not your specific case.
here’s a scenario that every company with multiple auth systems eventually faces.
you’ve got users logging in via your enterprise identity provider. you’ve also got internal services that need to act on behalf of users. you’ve got api clients that are third-party integrations. they all need different tokens, or at least tokens with different claims.
the naive approach is to pass the external token all the way through your stack. every service validates it. every service extracts claims from it. but now your internal contract depends on the external provider’s token format. when you want to switch providers, you’re touching every service. when you want to add an internal claim, you can’t, because the token format is locked.
the better approach is token exchange. your user logs in with the external provider and gets a token. they send that token to your auth service. your auth service verifies it with the external provider, extracts what you care about, and mints a new token in your internal format.
now your services never see external tokens. they only see internal tokens. which means you can change identity providers without touching your services. you can add custom claims. you can add scopes. you can add whatever internal structure makes sense.
this isn’t a novel idea. it’s literally part of the oauth spec. but most managed auth platforms don’t think about this pattern because they’re trying to be the identity provider, not a layer on top of identity providers.
when you own this layer, you can implement it in a weekend. you need to verify a jwt with an external provider’s jwks endpoint, extract claims, and mint your own token. that’s maybe two hundred lines of code. the complexity is in the operational semantics, not the implementation.
if you’re going to do this, think about what the service actually needs to do.
issuing tokens: a client calls with credentials, you verify them, you issue a token. that’s one endpoint.
validating tokens: a service calls with a token, you verify the signature and check if it’s revoked. that’s one endpoint.
exchanging tokens: an external token comes in, you verify it with the external provider, you issue an internal token. that’s one endpoint.
refreshing tokens: a client calls with a refresh token, you issue a new access token. that’s one endpoint.
revoking tokens: a client calls saying this token should be invalid. you record it. that’s one endpoint.
that’s the core api. everything else is supporting infrastructure.
the database is simple. a clients table for who can get tokens. a tokens table for metadata about issued tokens. a revocation table for what’s been revoked. an audit table for compliance. indices on the obvious fields. maybe a rate limit tracking table if you want per-minute limits.
the token format is straightforward. an issuer claim. a subject claim identifying the client or user. a scope list if you care about scopes. an expiry. optionally custom claims for context. a jwt id for revocation tracking.
none of this is complex. none of this requires specialized knowledge. what it requires is being clear about what you need and building exactly that, nothing more.
when you build this yourself, you’re committing to understanding how it works and being on call for when it breaks. this is worth thinking about.
but here’s the thing: you’re going to be on call for auth anyway. whether you own it or not, you need to understand it well enough to debug it when it’s broken. and when something breaks in a managed platform, you’re debugging without visibility. you’re sending support tickets. you’re hoping the platform team understands your use case.
when you own it, you’re debugging your own code. which is generally faster.
the operational burden is real, but it’s probably less than you think. a service with one job tends to be stable. it’s not doing complex business logic. it’s signing tokens and checking revocation lists. the failure modes are pretty limited. bad credentials. database down. network partition.
the tooling you need is standard. async http server. database. crypto library. structured logging. metrics. all of this exists and is boring. deploy it on your infrastructure like any other service.
you’ll know you should do this when you find yourself in one of these situations:
you’re using a managed platform and you discover that adding a custom claim requires redeploying all your services that rely on the claim structure.
you’re using a managed platform and you need to change how tokens are issued for a specific tier of clients, but the platform’s configuration doesn’t support tier-specific behavior.
you’re using a managed platform and you want to switch identity providers next quarter, but that’s going to require rewriting integration code in ten different services.
you’re using a managed platform and you realize that your audit and compliance team needs query access to token decisions that aren’t exposed through the api.
you’re using a managed platform and you calculate that the cost of validation calls plus the operational overhead of caching plus the cost of custom code you’re writing anyway is more than the cost of just owning it.
any one of these is a sign that the platform’s generality is working against you. at that point, you’re not getting leverage from managed auth. you’re fighting it.
if you decide to do it, don’t overthink the implementation. the goal isn’t to build a platform. the goal is to build a service that solves your specific problem.
start with the core four endpoints: issue, validate, revoke, refresh. get those working. get them tested. deploy them.
add token exchange when you need it. it’s one more endpoint. it adds complexity around external provider integration, but that complexity is localized.
add rate limiting when you need it. it’s tracking counts in time windows. standard stuff.
add audit logging when you need it. it’s writing documents to a collection. standard stuff.
add custom policies when you need them. if you want different ttls for different tiers, that’s logic in the issue endpoint. if you want custom revocation cascading, that’s logic in the revoke endpoint.
build it incrementally. use boring technology. use a language you know well. use a database you understand. use standard libraries for crypto. the whole point is to be able to understand every line and every decision.
when you decide to build this instead of using a platform, you’re not betting that you can do it better than the platform team. you’re betting that you can do it better for your specific constraints.
you’re betting that understanding your own auth system is worth the engineering time. that having full visibility into token decisions is worth maintaining a service. that being able to iterate on auth policy without platform limitations is worth the operational overhead.
these are reasonable bets if you’re at a certain scale and certain level of architectural sophistication. they’re not reasonable bets if you’re small or if your auth needs are simple.
but if you’re reading this because you’re already frustrated with a platform, if you’re already writing custom code to work around platform limitations, if you’re already maintaining integration code that’s basically a shadow auth system, then you’re not really betting on building your own anymore. you’re already maintaining your own. you’re just doing it in a way that’s split between their code and your code.
at that point, consolidation is probably the right move.
here’s what makes this work in practice: once you own the token validation, you own the validation latency. validation happens locally. it’s not a network call. it’s not a cache hit or miss. it’s your service’s public key and your service’s crypto library.
this changes what’s possible architecturally. services can be strict about validation because validation is fast. you can add validation to endpoints that wouldn’t normally have it because there’s no performance penalty. you can iterate on token structure because you only need to update your service and the places that consume tokens.
it’s not flashy. it’s not the kind of thing that makes for impressive architecture diagrams. but it’s the kind of thing that makes systems feel responsive and makes teams feel like they’re not fighting their infrastructure.
the decision to build versus buy is ultimately about honesty.
be honest about how much of the platform you actually use. be honest about how much code you’re writing to work around platform limitations. be honest about what your compliance and operational teams actually need. be honest about whether the platform’s defaults align with your architecture or work against it.
if you’re honest about those things, the decision usually becomes pretty clear.
and if you do decide to build it, remember that you’re not building an identity platform. you’re building a token policy service. keep it small. keep it focused. keep it yours.
the platforms aren’t going anywhere. if you decide this isn’t worth it, you can always switch later. but at least you’ll have made the decision with your eyes open, understanding the actual shape of your problem instead of fitting your problem into someone else’s shape.
]]>