Measuring a community's success requires knowing what stage the community is at, and why you are measuring in the first place. Everything else follows from those two answers.
Efios first published on this subject in 2004, as a short Spanish-language paper written for sponsors who were being asked to fund communities of practice and wanted to know what they would get back. The core argument has held up well: the three parties around a community — the sponsor, the participants, and the organisation employing them — each define success differently, and a measurement approach that ignores that asymmetry will satisfy none of them.
What has not held up is everything around it. The measurement literature moved on substantially after 2011. Collaboration moved onto platforms that emit telemetry no one had in 2004. Network analysis became something you can do without a research budget. And privacy law now sets real limits on what you are permitted to measure about identifiable people at work.
This page is the current version of that argument: the theory it rests on, an analytics model that survives sponsor scrutiny, and how we apply it in multi-party railway programmes, where the knowledge that matters is spread across an infrastructure manager, several operators and a supply chain that does not share an employer.
Measurement arguments go wrong early when "community of practice" is used loosely. The term carries a specific meaning, and the measurement implications follow directly from it.
Lave and Wenger (1991) described learning as participation in a practice rather than transfer of content, with newcomers moving from the periphery toward the centre as they become competent. The measurable object is therefore participation and trajectory, not consumption.
Wenger (1998) characterised a community of practice by mutual engagement, a joint enterprise, and a shared repertoire. A group missing any of the three is a team, a distribution list, or an audience — and should be measured as one.
Wenger, McDermott and Snyder (2002) set out a progression from potential through coalescing, maturing and stewardship to transformation. Each stage makes different evidence meaningful.
Wenger-Trayner and colleagues (2011) reframed the problem: rather than one number, value is traced through cycles — immediate (the interaction itself), potential (knowledge capital), applied (changed practice), realised (performance effect), and reframing (changed definition of success). Later work extends the set toward strategic and enabling value.
Nonaka and Takeuchi (1995) distinguished tacit from explicit knowledge and described how organisations convert between them. Communities are disproportionately good at the socialisation step — the one that leaves the least documentary trace.
Cross and Parker (2004) showed that informal networks explain organisational performance in ways org charts do not, and that the useful interventions are structural: connect the disconnected, relieve the overloaded broker, close the gap between two clusters.
Three layers, in order. Skipping to the third is what produces dashboards nobody trusts; stopping at the first is what produces dashboards nobody uses.
Contributions, active members, response latency, thread depth, repeat participation. This layer is cheap, automatic on any modern platform, and answers exactly one question: is the community alive? Treat it as a health check, not as evidence of value. Rising post counts with flat application is a well-documented failure mode, not a success.
The one activity metric consistently worth watching is response latency to a first-time asker. It predicts whether newcomers return, and it degrades before anything else does.
Organisational network analysis turns interaction data into structure: how connected the community is, whether it depends on a handful of brokers, whether sub-groups have fused or stayed separate, who is isolated, and whether the core is renewing or ageing. These are the measurements that suggest an intervention rather than merely a verdict.
Useful indicators: network density and its trend; the share of connections passing through the top three brokers (a concentration risk, not an achievement); reciprocity; the count of members with no ties outside their own unit; and time-to-first-tie for new members.
Run the value-creation cycles as the reporting spine. For each cycle, hold one or two indicators and at least one documented case, and make the link between them explicit: a story with no indicator is an anecdote, an indicator with no story cannot be defended when challenged.
Collect the cases continuously rather than in an annual scramble, and record them in a fixed structure — situation, what the community contributed, what changed, what it was worth, who can confirm it. Structured narrative is data. Free-form testimonials are not.
The same community warrants different evidence at different stages. This is the practical form of the 2004 argument, updated to the value-creation cycles.
The 2004 paper closed on a line worth keeping: measurement is about the people in and around the community — the technology for measuring the data is merely an enabler. Two decades of better tooling has not made that less true. What the tooling changed is the cost of doing it properly.
Interaction data that once required a survey now arrives by default from the collaboration platform. Network analysis that once needed a specialist is now a modest piece of work. The binding constraint moved from data availability to interpretation, consent and the discipline to collect evidence of applied value while it is still fresh.
The part that did not change: sponsors fund communities on the strength of a credible story supported by defensible numbers, and neither half works alone.
We work where knowledge has to cross organisational boundaries under safety and delivery pressure — an infrastructure manager, several operators, and a supply chain with no shared employer. Communities of practice are one of the few mechanisms that function in that setting, and they are consistently the first thing cut when nobody can say what they produce.
Agree the sponsor's real question, the stage-appropriate evidence, and the privacy position at the point the community is chartered — not at the first budget review, when the baseline is already unrecoverable.
A defensible starting position: who is connected to whom across the programme, where knowledge is stuck behind a single broker, and which groups have no path between them. Repeatable, and aggregated to protect individuals. This is part of our Frontier Services practice.
A light structure for capturing applied and realised value as it happens, with named confirmers, so the annual justification is an assembly job rather than an archaeology project.
Community value expressed in the terms the programme already reports — rework, defect escape into later test phases, avoided duplication between parties, lead time on issue resolution — rather than in a parallel vocabulary nobody is accountable for.
Some groups are not communities of practice and should not be funded as such. Establishing that early is a legitimate and frequently valuable outcome of the first measurement pass.
The measurement approach is documented and transferred to the community's own coordinators. If it only works while we are present, it does not work.
If you need to justify a community of practice to a sponsor — or decide whether it is worth continuing — we are glad to discuss the approach directly.