Enterprise AI spending is growing 40–60% year-on-year. GPU compute, LLM API contracts, vector databases, ML platform tooling, and the engineering labor required to build and operate AI systems together constitute the fastest-growing category in enterprise technology — and, in most organizations, the least-governed one. The bill that nobody approved is arriving at the CFO’s desk.
The six-layer cost taxonomy
Most enterprises track AI cost at the GPU compute level and stop there. The full picture has six layers, and the layers most often missed are the most strategically dangerous.
Layer 1 — GPU compute for training and inference. The visible layer; typically 30–45% of total AI spend. Layer 2 — API contracts with model providers (OpenAI, Anthropic, others); 15–25%. Layer 3 — data infrastructure: vector databases, feature stores, ETL pipelines feeding model training and serving; 8–15% but growing fast. Layer 4 — ML tooling: MLOps platforms, model registries, observability, evaluation frameworks; 5–10%. Layer 5 — labor: ML engineers, data scientists, platform engineers, the highest unit cost in the entire taxonomy; often 20–30% when fully loaded. Layer 6 — shadow AI: spend that bypasses governance entirely — SaaS-embedded AI features, BU-led pilot subscriptions, individual API keys on personal cards. Typically 20–30% of the total and invisible to traditional cost governance.
The four questions boards are asking
Total cost?
What is the annual cost of our AI program, all layers included?
Cost per use case?
What does each AI use case actually cost us per inference, per month?
Where is the ROI?
Which initiatives are producing measurable business value, and which aren’t?
Allocation framework?
How are we deciding which initiatives get GPU capacity?
For most enterprises, the honest answers are: somewhere between $3–8M; we haven’t tracked it; we’re still measuring; no formal framework. That set of answers is what triggers board alarm. CFOs do not fundamentally object to AI investment — they object to material spend categories without governance. Uncontrolled spend plus no ROI visibility plus exponential growth equals material risk by any reasonable definition.
The architecture decision that matters most
The single biggest cost decision in enterprise AI is deployment architecture. The cost spread between proprietary API at scale, open-source self-hosted, and fine-tuned hybrid can exceed $1.2M annually for the same use case at the same throughput. The decision pivots on four variables: call volume (proprietary API economics break above ~10M calls/month for most models), latency requirements (self-hosted wins on tail latency at scale), data sensitivity (some categories of data preclude third-party API processing entirely), and the team’s ML operating capability (self-hosted requires real ops investment).
The economically wrong choice is rarely catastrophic. The unexamined choice almost always is. Most enterprises today default to proprietary API because it is fastest to ship, then discover at scale that the unit economics no longer work — and that re-architecting after the fact is far more expensive than choosing correctly the first time.
The 90-day governance baseline
Get ahead of the question before the board asks it. Within 90 days a credible governance baseline includes: a six-layer AI cost taxonomy with each layer allocated and tagged; a cost-per-use-case baseline for the top five initiatives; shadow AI surfaced through expense data, SSO logs, and network signals; and a board package that shows total cost, breakdown by use case, ROI evidence for the most mature initiatives, and the governance controls now in place.
The reframe matters. This is not a cost-restriction conversation; it is a cost-governance opportunity. CIOs who bring AI cost data to the board before the board demands it become the trusted voice on AI strategy. CIOs who wait to be asked are responding, not leading — and the next conversation is rarely on their terms.
Key takeaways
- Six layers of AI cost — most enterprises track only the GPU layer. The other five matter just as much.
- Shadow AI is 20–30% of total AI spend in most organizations and is invisible to traditional cost governance.
- Architecture choice produces $1M+ cost spreads for the same use case. The decision deserves a real business case.
- Boards are asking four specific questions. Build the answers in 90 days; don’t wait to be asked.
- The reframe is governance, not restriction. The CIO’s job is to make AI investment defensible — not to slow it down.