Ownership metrics beat McCabe complexity at predicting…
0 reactions
Ownership metrics beat McCabe complexity at predicting bugs: 6-month study across Django, FastAPI, Pydantic
I'm working on an open source codebase intelligence tool. One layer of it scores every file 1-10 using 15 deterministic biomarkers. No LLM. AST parsing via tree-sitter plus git history.
Wanted to know if the scores actually mean anything. So I ran a time-travel experiment.
Setup
Scored every file at time T, then counted bug-fix commits over the following 6 months. Three repos: FastAPI (104 files), Pydantic (216 files), Django (542 files). 862 files total.
The biomarkers fall into four buckets:
\- Structural (7): brain\_method, nested\_complexity, bumpy\_road, complex\_method, large\_method, complex\_conditional, primitive\_obsession
\- Duplication (1): dry\_violation (Rabin-Karp rolling hash over tree-sitter tokens, survives variable renames)
\- Test coverage (2): untested\_hotspot, coverage\_gap
\- Organizational (5): developer\_congestion, knowledge\_loss, hidden\_coupling, function\_hotspot, code\_age\_volatility
What I found
On Django: Spearman ρ = -0.34 (p < 0.0001). Precision@20 = 70%, meaning 14 of the 20 worst-scoring files had real bugs in the next 6 months.
The two strongest single predictors were both process signals, not structural ones.
\- untested\_hotspot (Cliff's delta = 0.67): files that change a lot but have no test coverage
\- developer\_congestion (Cliff's delta = 0.78 on Django): too many authors touching the same file in a short window
McCabe complexity and nesting depth ranked lower than both.
The weird one
knowledge\_loss went negative. Files where original authors had left the project had fewer bugs. My read: stable legacy code that nobody touches doesn't break. The metric captures something real (absent knowledge) but the effect gets swamped by the fact that those files are also cold.
I'm still thinking about how to fix this. Probably need to gate it on recent change frequency.
The honest part
Controlling for file size drops the overall correlation from \~0.3 to \~0.1. Bigger files carry more complexity, more churn, and more bugs. File size is a confound in basically every code health study.
CodeScene published a study claiming 15x more defects in unhealthy code but never reported this confound. I didn't want to make the same mistake.
The composite score still adds predictive value on top of file size alone, but I want to be clear that size is doing a lot of the heavy lifting.
Has anyone else seen ownership/process metrics outperform structural complexity in practice? I never see teams optimising for it
Repo is open source if anyone wants to poke at the methodology or run it on their own codebase.
#technology
source
I'm working on an open source codebase intelligence tool. One layer of it scores every file 1-10 using 15 deterministic biomarkers. No LLM. AST parsing via tree-sitter plus git history.
Wanted to know if the scores actually mean anything. So I ran a time-travel experiment.
Setup
Scored every file at time T, then counted bug-fix commits over the following 6 months. Three repos: FastAPI (104 files), Pydantic (216 files), Django (542 files). 862 files total.
The biomarkers fall into four buckets:
\- Structural (7): brain\_method, nested\_complexity, bumpy\_road, complex\_method, large\_method, complex\_conditional, primitive\_obsession
\- Duplication (1): dry\_violation (Rabin-Karp rolling hash over tree-sitter tokens, survives variable renames)
\- Test coverage (2): untested\_hotspot, coverage\_gap
\- Organizational (5): developer\_congestion, knowledge\_loss, hidden\_coupling, function\_hotspot, code\_age\_volatility
What I found
On Django: Spearman ρ = -0.34 (p < 0.0001). Precision@20 = 70%, meaning 14 of the 20 worst-scoring files had real bugs in the next 6 months.
The two strongest single predictors were both process signals, not structural ones.
\- untested\_hotspot (Cliff's delta = 0.67): files that change a lot but have no test coverage
\- developer\_congestion (Cliff's delta = 0.78 on Django): too many authors touching the same file in a short window
McCabe complexity and nesting depth ranked lower than both.
The weird one
knowledge\_loss went negative. Files where original authors had left the project had fewer bugs. My read: stable legacy code that nobody touches doesn't break. The metric captures something real (absent knowledge) but the effect gets swamped by the fact that those files are also cold.
I'm still thinking about how to fix this. Probably need to gate it on recent change frequency.
The honest part
Controlling for file size drops the overall correlation from \~0.3 to \~0.1. Bigger files carry more complexity, more churn, and more bugs. File size is a confound in basically every code health study.
CodeScene published a study claiming 15x more defects in unhealthy code but never reported this confound. I didn't want to make the same mistake.
The composite score still adds predictive value on top of file size alone, but I want to be clear that size is doing a lot of the heavy lifting.
Has anyone else seen ownership/process metrics outperform structural complexity in practice? I never see teams optimising for it
Repo is open source if anyone wants to poke at the methodology or run it on their own codebase.
#technology
source
| 3,000 mlx | |
| 4 | |
| 0 |