Building an AI Centre of Excellence — What Actually Works

Every enterprise we work with has an AI Centre of Excellence on the org chart, planned for one, or recovering from one that didn't work. The CoE is the structural answer to a real question: how does a large organisation build AI capability without duplicating effort across every team? The shapes that actually compound value differ from the consultancy-recommended default.

This piece is a practitioner view of what an AI CoE should actually do, what shapes work and don't, and how to set one up so it serves the organisation rather than becoming its own end.

What CoEs typically try to do

The standard CoE mandate covers some combination of:

• Strategy — set the organisation's AI strategy, prioritise investments
• Standards — define standards for AI development, deployment, and operation
• Governance — review and approve AI initiatives
• Platform — provide shared infrastructure for AI workloads
• Expertise — provide deep AI expertise to other teams
• Innovation — explore new technologies, build pilots
• Training — upskill the broader organisation
• Vendor management — manage relationships with AI providers

Few CoEs do all of these well. Most do some of them. The ones that compound value have a clear sense of which they do and which they don't.

The shapes that work

Platform-led CoE

The CoE operates the AI platform. Other teams build their workloads on the platform. The CoE's value is in the leverage of shared infrastructure: shared model access with negotiated pricing, shared observability, shared evaluation tooling, shared compliance audit, shared governance.

What this looks like:

• A platform team that builds and operates the AI infrastructure
• Self-service for line-of-business teams to deploy workloads
• Standards encoded in the platform (you can't deploy without observability; you can't deploy without governance review; you can't deploy without budget controls)
• Expertise available to support teams using the platform

The pattern works because the CoE adds value continuously, not just at project kickoff. Teams use the platform every day; the value of the shared layer compounds.

Expertise hub

The CoE concentrates deep AI expertise. Other teams have business and domain expertise; the CoE pairs with them on initiatives that need AI engineering depth.

What this looks like:

• A senior AI engineering team
• Engagement model with line-of-business teams (paired delivery, advisory, training)
• Pattern documentation and shared knowledge base
• Light-touch governance focused on technical quality, not approval gates

The pattern works for organisations where AI initiatives are concentrated in a few teams; spreading the expertise thinly across many independent initiatives works less well.

Governance light + platform

A hybrid. Governance is a thin layer; the heavy work happens in the platform. The CoE writes the policies and runs the reviews; the platform enforces the policies automatically.

This is the shape that has scaled most successfully in the enterprises we have seen. The governance burden on individual teams is low because the platform handles most of it; the policies are real because the platform enforces them.

The shapes that don't work

Governance-only CoE

The CoE writes policies and runs reviews. It doesn't operate infrastructure or provide expertise. Other teams build their own AI capability and submit to CoE review.

The pattern doesn't work because the CoE has no operating leverage. Teams either route around the CoE (often successfully, given the speed of AI capability evolution) or comply slowly. The CoE becomes a delay rather than an accelerator.

Innovation-only CoE

The CoE builds pilots. Lots of pilots. None of them ship to production. The CoE becomes a showcase for what could be done, not what is done.

The pattern doesn't work because innovation without production is theatre. The CoE produces credible demos and the operational organisation continues to do its work without AI. The CoE's value is illegible.

Shadow IT CoE

The CoE operates outside the normal IT governance. It procures models, runs compute, builds applications, all in parallel with the normal IT operating model. Over time, the CoE accumulates infrastructure that nobody is monitoring, compliance gaps that nobody is auditing, and bills that nobody is challenging.

The pattern doesn't work because parallel infrastructure is exposure. The CoE's freedom from constraint becomes the organisation's risk.

The right scope at the right stage

The CoE's appropriate scope evolves with the organisation's AI maturity:

Early stage — no shipped workloads

The CoE focuses on:

• A first end-to-end production deployment
• Establishing patterns from real engagement
• Defining the platform shape that other teams will use
• Engaging governance partners as part of the work

What it doesn't do yet:

• Comprehensive standards (no production experience to base them on)
• Broad training (no specific patterns to teach)
• Strategy documents disconnected from delivery

Middle stage — several shipped workloads

The CoE focuses on:

• Operating the platform that other teams build on
• Codifying patterns from delivery experience
• Providing expert support to line-of-business teams
• Building reusable components

It begins to do:

• Standards based on shipped experience
• Training based on real patterns
• Governance review at scale

Mature stage — broad adoption

The CoE focuses on:

• Platform engineering as a first-class function
• Strategic capability building (fine-tuning pipelines, foundation models if relevant)
• Cross-team optimisation
• Influencing the broader organisation's AI strategy

It does less:

• Direct delivery (line-of-business teams now have their own AI engineers)
• Manual governance (the platform automates it)

The CoE's role evolves. A CoE that stays in the early-stage shape after years of organisational maturity becomes a bottleneck. A CoE that pretends to be at mature stage when the organisation is at early stage produces frustration.

The staffing pattern

CoEs that work have a recognisable staffing shape:

• Senior AI engineers who have shipped production systems. The CoE's authority is technical; it depends on having people who actually know what they're doing.
• Platform engineers who operate the shared infrastructure. The platform is real engineering; it needs real engineers.
• A product manager focused on the platform's roadmap. Without one, the CoE drifts toward whatever the most senior engineer finds interesting.
• A governance lead who partners with security, compliance, legal. Without one, the governance work doesn't happen reliably.
• Architects who pair with line-of-business teams on their initiatives.

What is conspicuously not on the list: a large strategy team, a layer of project managers, a roster of consultants. These accumulate in CoEs that have lost their delivery focus.

What we keep seeing

Recurring patterns in enterprise AI CoEs:

Platform-led CoEs scale. Others don't. The ones with a real platform that line-of-business teams actually use accumulate compounding value. The others fight for relevance.

Governance enforced by the platform works. Governance enforced by reviews doesn't. Teams comply with what the platform requires; they route around what reviews require if the reviews are slow.

The shipped workload is the credential. A CoE that has shipped real workloads has credibility with the rest of the organisation. A CoE that has only produced documents and pilots doesn't.

Executive sponsorship is necessary but not sufficient. Sponsorship gets the CoE created; only delivered value keeps it relevant.

The expertise hub model works where workloads are concentrated. It struggles where the organisation has many small AI initiatives across many teams; the expertise can't cover all of them.

What we recommend

For organisations standing up or restructuring an AI CoE in 2024:

1. Start with platform-led ambition. Build the shared infrastructure as the CoE's primary value.
2. Embed governance in the platform. Manual reviews don't scale.
3. Hire senior AI engineers who have shipped. Credibility comes from competence.
4. Define the engagement model with line-of-business teams. Paired delivery, advisory, or training — pick the shape.
5. Avoid governance-only or innovation-only mandates. Both produce CoEs that don't accumulate value.
6. Engage governance partners (security, compliance, legal) as collaborators, not approval gates.
7. Re-assess the CoE's scope as the organisation matures. The right shape evolves.

The AI CoE is one of the highest-leverage organisational investments available in enterprise AI adoption. The investment compounds when the CoE has a real platform, real engineering, and real engagement with the rest of the organisation. The investment evaporates when the CoE produces documents and pilots without operating leverage. The shape determines the outcome.