OpenAI isn’t only making models more capable. They’re meaningfully becoming more efficient at running them – and that should get every business leader’s attention.
Internal financials reported by The Information point to a sharp improvement in OpenAI’s “compute margin” (what’s left after the cost to run models for paying users): ~70% in October, up from ~52% late last year and ~35% in January 2024.
Here’s the takeaway: efficiency is now part of the AI product strategy – not an afterthought.
Why This Matters (Even If You're Not Building Models)
If a company operating at OpenAI’s scale is pushing hard on runtime efficiency, it’s because compute is the constraint. The article underscores that server availability is a major limiter, and even includes a blunt quote from Sam Altman: OpenAI is “compute constrained,” suggesting that significantly more compute could translate into significantly more revenue.
For SMBs, mid-market firms, and enterprises, that should translate into a clear question: Are we using AI in a way that improves outcomes… or just increases spend?
Because AI is different from traditional software: every single request has a runtime cost. And while OpenAI’s margins improved, the article notes these margins still don’t look like classic software economics where adding users costs almost nothing.
What's Driving Efficiency Gains (And What You Can Learn From It)
The article points to multiple contributing factors, including:
- Lower costs to rent compute over time
- Model tweaks to run more efficiently
- Pricing and packaging changes (including a higher priced tier for some customers)
You may not control all those levers directly; but you do control how you architect AI across your environments and how you measure success.
What SMBs, Mid Market, and Enterprises Should Evaluate Right Now
The organizations that win with AI over the next 12–24 months won’t be the ones that “use AI the most.” They’ll be the ones that run AI in the right place, with the right model, at the right cost, with the right controls.
And “right place” now means multiple runtime environments:
- Cloud across the three hyperscalers: AWS, Microsoft Azure, Google Cloud
- IoT / Edge environments where latency, bandwidth, privacy, and resiliency matter
Below is a practical way to think about it by business segment.
SMBs: Keep It Simple. Make It Measurable.
SMBs don’t need sprawling architecture. They need repeatable efficiency.
Focus on:
- A small number of high-frequency workflows (support, sales ops, finance ops, internal helpdesk)
- Fast time-to-value with real metrics: time saved, cycle time reduced, cost per completed task
- Right-sized models (don’t default to the largest model if a smaller one hits the quality bar)
- Guardrails so usage doesn’t quietly become a runaway line item
Runtime reality: Start in the cloud. Add edge where connectivity, latency, or data locality forces the issue.
Mid Market: Standardize So You Can Scale Without Chaos
Mid‑market teams often do dozens of pilots and then struggle to industrialize them.
Prioritize:
- A repeatable delivery pattern (security, data access, testing, deployment, monitoring)
- Model routing (small/fast for routine tasks, larger models for complex tasks)
- Operational governance (evaluation harnesses, versioning, approvals, rollback)
- FinOps for AI (visibility and accountability for spend vs. value)
Runtime reality: Cloud-first is common. Edge becomes strategic in manufacturing, retail, healthcare, logistics, and field operations.
Enterprises: Manage AI Like A Runtime Portfolio
Enterprises aren’t choosing a model. They’re managing:
- multiple models
- multiple business units
- multiple data sensitivity tiers
- multiple regions and compliance requirements
- multiple cloud platforms
- and increasingly… edge fleets
What “good” looks like:
- A model portfolio strategy (who can use what, for which tasks, under which policies)
- Policy-driven routing based on cost, latency, data sensitivity, and jurisdiction
- Observability at scale (quality, drift, performance, security, cost)
- A hybrid execution design: cloud where it fits, edge where it must
Runtime reality: If you’re not designing for hybrid, you’ll end up there anyway – just without standards and controls.
The Efficiency First AI Checklist (Cloud + Edge)
If you want a fast, practical evaluation framework, use these five lenses:
- Right model, right job
- Can a smaller model handle most requests?
- Are larger models reserved for the work that truly requires them?
- Right runtime placement
- Cloud for elasticity and centralized governance
- Edge for low latency, offline resiliency, bandwidth savings, and local data control
- Right architecture patterns
- Caching, batching, retrieval augmentation, re-use
- Async where real-time isn’t required
- Right infrastructure utilization
- Are you paying for idle capacity?
- Are accelerators utilized effectively?
- Right operating model
- Monitoring and SLOs (latency, errors, quality regressions)
- Cost visibility tied to outcomes, not just usage
This is the same mindset implied by OpenAI’s efficiency push: not one magic change – a disciplined set of decisions across models, infrastructure, and operations.
How MILL5 Helps: Strategy. Build. Operate.
If you want AI to drive real efficiency – across AWS, Azure, Google Cloud, and IoT/Edge – MILL5 can help you move from experimentation to operational advantage.
Strategy
We help you define:
- the right use cases (prioritized by measurable ROI)
- the right runtime plan (cloud vs edge vs hybrid)
- a model portfolio approach (fit-for-purpose + routing + guardrails)
- governance that supports speed and control
Build
We help you engineer:
- production-grade AI/ML solutions across hyperscalers
- architectures that reduce runtime cost and improve performance
- secure integrations into the workflows and systems your teams already use
- evaluation/testing pipelines so quality doesn’t drift silently
Operate
We help you sustain:
- observability (quality + drift + latency + cost)
- FinOps discipline for AI spend and utilization
- ongoing tuning and optimization as models, pricing, and workloads evolve
If OpenAI proves that efficiency is a competitive advantage, it’s time to evaluate how you run AI – across cloud and edge – and make sure every workload is engineered for outcomes, cost control, and scale. MILL5 can help with Strategy, Build, and Operate.
Contact MILL5 today for your complimentary strategy session.
