Businesses cannot truly have deep integration with AI tools until they are capable of being honest about who they are

Just about every company I have ever worked at or with says it wants to “move fast” and “embrace change”

‍

Most of them also had aggressively gatekept sign-off processes, executive vibes driving team activity and is actively engaged in creating more meetings and more paperwork rather than less. By just about every metric they are trying to move slower.

‍

They say they want results but then they measure activity and inputs instead of outcomes. They reject, underfund or outright neglect research in favour of big swings based on internal opinions and court politics. Business change and products, by the time they’ve been fully funded, paid for and delivered, have not been anywhere near a customer at any point. A truly incredible way to find out what you delivered does not resemble what your customer wants.

‍

They say they want to be “a great place to work” while doing governance-by-spreadsheet and deciding that they “have to shrink headcount” ensuring that people get managed out regardless of performance.

‍

They put "force for good" on their about page and in founder interviews, and then their website is dripping with manipulative design and they’re bringing in Palantir to replace their customer service functions.

‍

It can be tempting to ascribe this entirely to hypocrisy and wishful thinking. Bad leaders making bad decisions, cynical comms, values that were bought from a branding agency that’s “good at that sort of thing”, all that shit. There might be a bit of that stuff at work here, but I think that ascribing everything to a lack of sincerity is something of a red herring. Plenty of the executives I've sat across the room from at least claim to believe the words on their walls, and realistically it doesn't matter if they believe them or not because what actually changes things in the world around them is what they do. If those values fold under pressure then they are vibes. Values that flex in response to economics are not worth much because they don’t really tell you anything useful about the person who holds them. Values should inform you about how somebody will act in specific situations. However, we have seen that the moment those values are actually tested, when treating staff well requires investment, when getting honest results delays a funding round or other action, almost everyone folds to the incentive that was really in charge all along. The economics. But it’s considered poor taste to put on your /values page how much you love making money, and how you want to make more money than anyone else, because money can be spent on cool things that you enjoy.

‍

A better explanation of why this keeps happening, and how we can understand it better, comes from a British cybernetician who spent a lot of time in the 1970s trying to wire up the Chilean economy, and it’s a useful way of thinking about why we get the same outcomes time and time again from actors who keep promising us that they are different.

‍

‍

POSIWID

Stafford Beer's dictum was blunt: the purpose of a system is what it does. He used the phrase in The Heart of Enterprise in 1979 and kept using it for the rest of his life, telling an audience at the University of Valladolid in 2001 that it "stands for bald fact, which makes a better starting point in seeking understanding than the familiar attributions of good intention." Elsewhere he put it more bluntly. There is "no point in claiming that the purpose of a system is to do what it constantly fails to do."

‍

He's right. Well, I think he's right. I'm going to spend the next few paragraphs explaining why. Beer is saying that a system's purpose cannot be derived accurately from its mission statement or its founder's intent or the strategy deck. Purpose is an observed property. You infer it from outputs. If your hiring process consistently produces a monoculture, then producing a monoculture is what your hiring process is for. If your release process consistently takes eleven weeks, your organisation is built to ship quarterly, regardless of what your OKRs say about velocity.

‍

I acknowledge that this sounds like a semantic trick until we start to use it as a diagnostic.

‍

Take any outcome your business reliably produces that nobody claims to want, high churn, burned-out staff, endless roadmap slippage, random and scattergun experimentation that doesn’t drive change, analytics that are busted and tell you nothing, business intelligence that keeps conflicting with itself and ask: what would this organisation look like if that outcome were the actual goal?

‍

Usually the answer is: exactly how it looks right now. The incentives, the reporting lines, the metrics, the motivations of your practitioners, how performance is measured and understood at your organisation, and the meeting cadence all make perfect sense once you stop believing in a stated purpose and start confronting the revealed one.

‍

Business structures as they are documented and planned vs observed structures as they exist in reality

‍

Why your words and your system diverge

The gap between what organisations say and what they do is structural, and it has actually been researched and measured.

‍

Chris Argyris and Donald Schön spent decades documenting the difference between espoused theory and theory-in-use. Essentially, what people say governs their behaviour versus what actually governs their behaviour.

‍

Their finding, which holds up depressingly well, was that the two routinely diverge and that people are largely blind to divergence in themselves, and also that pointing it out often triggers defensiveness rather than learning. Organisations tend to institutionalise this. Theory is saved for the annual report; theory-in-use lives in the bonus structure and the outcomes.

‍

The empirical version is pretty clear too. Donald Sull and colleagues at MIT analysed 1.2 million Glassdoor reviews against the published values of roughly 700 large companies and found no correlation at all between the values a company puts on its website and how employees rate it on those same values. The correlation coefficient hovered around zero so not just weak but zero. The values statement and the lived culture are, statistically speaking, completely independent variables.

‍

So what does govern behaviour, if not what the business says it cares most about? Incentives, mostly, and here Beer's thinking pairs neatly with Goodhart's law which is that when a measure becomes a target, it ceases to be a good measure. People do not optimise for outcomes. They optimise for whatever proxy the organisation pays, promotes and praises on, and the proxy always drifts from the outcome.

‍

Image courtesy of https://www.flickr.com/photos/jeepersmedia/13946047640/

‍

Wells Fargo is often used as the canonical case. The bank in its values statement and all its marketing stressed the importance of customer relationships but it paid its reps on cross-sold products, and it used a crude counter where a dormant chequing account scored the same as a mortgage. Staff responded entirely rationally to the actual incentive and opened millions of accounts customers never asked for, earning the bank a $100 million fine from the CFPB in 2016, a huge debt of trust it might never actually repay, and a star position as an example in this article. Apply POSIWID to that and I would say that the purpose of Wells Fargo's sales system was to manufacture account openings. That is what it did reliably at scale for years. The mission statement was irrelevant and independent of the behaviour.

‍

Beer actually had a structural explanation for why leaders can't see this and fix it. His Viable System Model is built on Ashby's law of requisite variety. A controller can only control a situation if it can match the situation's complexity. The lived reality of a business with thousands of daily decisions, workarounds, compromises etc contains so much more variety than any executive team can realistically absorb so executives manage a compressed model of the company instead. They manage dashboards, RAG statuses, meetings, town hall Q&As with pre-vetted questions. Every single thing leading them further from the source and abstracting away reality..Every layer of the hierarchy distorts the reality and each in a flattering direction because the messenger's incentives punish bad news and reward flattery.

‍

By the time that reality reaches the board it has been laundered into a kind of obtuse strategy language and that leaves the people responsible for setting the direction to instead just be adjusting the copy, styling and font choice again on a company brochure.

‍

Then one of two things usually happens. Either the brochure gets edited again and again but is entirely unpegged from reality and so nothing meaningfully changes, or, the board recognises that the outcomes that it’s actually beholden to (churn rate, profitability) are slipping despite all their interventions and they switch into a kind of panicked and short-termist vibes-based diktat where they attack the issue they believe to be the core problem, with an entirely unproven but different methodology, with constantly weaving and shifting aims living entirely undocumented in their minds which both cannot possibly be realised by anyone else at the company and also is very unlikely to work. Neither of these methods are effective.

‍

That is the underlying machinery of institutional self-deception. There is no villain required. The incentives set the behaviour, the behaviour diverges from stated values, the reporting structure filters out the evidence of that divergence, and the leadership sincerely believes an account of the company that the company's own outputs contradict. If you are wondering why there is a “say-do” gap then I am afraid I have some bad news. The purpose of a system is what it does and it's what the system is built to produce.

‍

Humans are REALLY perceptive too. Remember the Glassdoor vs Values study mentioned earlier? Most staff are capable of reading the gap perfectly and they calibrate their effort to the real incentives while performing a belief in the stated ones. This is especially exhausting and corrosive the more downward pressure is placed upon the staff to perform the ritual of belief. Strategies that have been built on the documented model keep on failing in ways that surprise only the board and surprises nobody on the shop floor. The organisation loses any ability to learn, because learning would require admitting the aforementioned theory-in-use exists and rules.

‍

This isn't how language models actually work, they're actually just graphing the next most likely character or outcome based on a huge dataset of training data, whereas a machine like this that "does X when Y" is deterministic, but this big machine that goes clunk, whirr etc is more interesting to look at and think about so here it is for the metaphor

‍

Now hand that to a machine that follows your instructions to the letter and never doubts you

Before AI this was a kind of management curiosity but as businesses remove humans from the loop in favour of automation it radically becomes a quite urgent commercial problem.

‍

MIT's NANDA initiative reported in 2025 that 95% of enterprise generative AI pilots deliver no measurable P&L impact. Standard explanations are tooling, data quality, skills etc as well as an obfuscation on what “AI” actually is (because it covers a whole range of technologies from harnesses, to LLMs to agents to workflows).

‍

All of those things are real. But I think a much deeper reason is what I have been writing about above. Organisations are briefing these systems with their espoused theory, and the espoused theory is entirely fictional.

‍

Let’s think about what happens when a company “deploys AI”. Someone in that business writes a prompt and either they or the system itself provides some kind of context. That context might be a policy document, a process description, a system instruction. All of that context is drawn from the official account of how the business works. The process which is entirely fictional and is based on a vision of how people wish the business and reality to be, and not what it is.

‍

It gives the documented process and not the workaround everyone actually uses*. The stated approval chain, not the complaint to the manager or personal relationship with a PM that really unblocks things. It contains the published values, ones with a correlation coefficient of zero.

‍

A human employee handed that same documentation does something remarkable that is not priced in: they ignore it. Within a month, your new hire learns which rules are real and which are ceremonial. They learn, but never vocalise because vocalising is taboo, who actually decides things, who to flatter, who to leave out of the loop, what the metric definitions mean in practice versus what they mean on paper. Humans read between the lines because they're embedded into that whole informal system. The whole organisation ticks along on this unrecognised, undocumented, interpretive labour.

‍

The machine acknowledges none of it. A language model believes what you tell it, completely and literally. AI researchers have called the failure mode specification gaming.

‍

The system satisfies a literal objective you wrote down and cannot accurately interpret the intent behind it, and it does so with total commitment. Anthropic's own research found models will generalise from small gaming behaviours to larger ones when the specification rewards it. You are effectively getting Goodhart's law with the safety off. A human employee gaming a metric at least knows they're gaming it. Your AI model is unaware of the game, unaware of the context and insulated from any and all outcomes (what will you do? Fire them? Take away their safety? Take away their livelihood? Put Claude on a PIP?)

‍

So any company that says it wants speed but is actually structured for extreme caution will deploy an AI agent briefed on the official process, and the agent will execute that process faithfully, faster. And it will fail for all the reasons a human would if they followed that process. A human will either need to intervene in the workflow to actually produce the outcome you wanted each time, or, you will need to rewrite the process. Now you might be thinking “Amazing! AI will help us rewrite our processes to be more efficient” but in doing that you expose yourself to risk at scale and don’t actually tackle the underlying system that created that caution.

‍

Anyway, congratulations, you have automated your bureaucracy. A company that says it values customer relationships but pays on conversion will point AI at the funnel and get a tireless, literal-minded optimiser of the proxy metric that was already distorting the business, rejected at delivery because a human still recognises that the implicit metric was profit at any cost rather than building customer relationships. Forbes' tech council published an editorial on this subject. An AI agent built on a flawed process doesn't fix the problem, it scales the dysfunction.

‍

In Beer's terms, AI is a variety amplifier of enormous power, and you will choose what it amplifies. If you feed it the theory-in-use, the system as it actually operates but you might need to confront some uncomfortable truths about yourself, your business, how it operates and what it does, but on a purely functional level it can uplift your capability. Feed it your espoused theory and it will widen the gap between the story and the reality, at machine speed, with no human to correct the course.

‍

This is why I'd argue the say-do gap is the single biggest predictor of whether a business gets anything out of AI. Not your model choice, not your budget or appetite for tokens, not your talent.

‍

Honesty.

‍

The 5% of pilots that succeeded in the MIT data are, I'd bet because I do not have the data, disproportionately in back-office processes where the documented workflow and the real workflow are nearly the same thing, invoice matching, contract extraction, places where the story and the reality match and the failures all grow where the fiction is at its thickest.

‍

Hey did you know that a stock photography model is paid roughly £30 an hour for their time and recieves no royalties for stock photographs? This guy gets to be the face of "guy who doesn't know stuff" in a load of decks and articles like this one and he gets absolutely nothing for it and yet everywhere he goes people are looking at him like "I think I have seen that guy before". Crazy to think about.

‍

Can we do anything about it?

‍

A practical move here costs almost nothing but your pride. Run a well structured POSIWID audit before you write a single prompt. It’s also the scientific method we’d use for an experimentation hypothesis but applied to this problem.

‍

Take each system you plan to point AI at and describe its purpose purely from its outputs, as a hostile observer would. It is absolutely imperative here that you leave your marketing spin and PR at home for this one. You CANNOT say "our support function exists to delight customers". You have to be realistic about what it does. If something like "our support function exists to close tickets in under four minutes, which is what we pay our bonuses on, which is why customers often get cut off mid-problem." then that’s what you write down. Crucially this kind of honesty should not be punished because it will be unpublishable. That document, not the process wiki, is the real system specification, and is the only briefing an AI can act on without recreating existing failures.

‍

Then decide, deliberately, which system you want to automate. The story or the reality.

‍

If it's the story, you have a LOT of redesign work to do first. Your incentives, metrics, sign-off structures and feasibility all need work before the machine can get involved. It is a slow and unglamorous fix but it's the actual work. If you want to reproduce the outcomes you currently get then at least brief the machine truthfully and let it optimise something real.

‍

What you cannot do is what most of the 95% did: hand a machine that behaves entirely literally a flattering fiction and expect it to infer the truth as an employee would. It won't do this. It will take your word for it. The purpose of a system is what it does, and what your system does will include everything your machines were told.

‍

Here, good practice dictates that I advertise the business. So entirely in a non-sequitur that's what I'm doing. Do you want your website to perform better by having better journeys and functionality that's much closer to what your customer wants? Do you want somebody who can build better products alongside you? That's me. I do that. Please hire me to do that at your company.

* A good anecdote here would be every time I’ve taken over an experimentation programme from an agency that promises its data-driven and supplies models and processes and documentation around what experiment to run, where and why, but then what actually shipped was a bunch of internal pet projects, best practice tweaks and highly visual changes because their stakeholders were really responsive to those. The documentation never writes “if your client is unhappy they’ll leave so if we have to make exceptions to our process to satisfy the client we will do it every time” so no automation would consider it. I sympathise with those agencies because even with the best of intentions, they came face-to-face with dysfunction and the true nature of the business. Which doesn't make for a good case study at all.

‍