4,031
123
0
165
0
67,103
0 reactions
A 99.5% district name-match still left 13.6% of my India map blank
I've been joining Indian district-level survey data to district boundaries, and I want to
flag something that cost me a day, in case it saves someone else one.
Indian district data is published keyed by district *name*, not by LGD code, so you're
stuck with name matching. Raw, I got 85.9% of rows onto a boundary. Building an alias table
by hand - the renames, the transliteration variants, the districts that split - took that to
99.5%, which felt finished.
The map still had holes in it. About an eighth of the country.
The problem was that I was reading the match rate as *rows that found a boundary*, which
was essentially 1.0. The number that decides whether a choropleth looks finished is the
reverse join: *boundaries that found a row*. Measured that way, 99 of 728 districts were
blank, and none of it was a name problem:
\- 54 districts had no row in the source file at all
\- 45 had a row whose value was the literal string "NaN"
And the pattern wasn't scattered, which is the tell. It was Telangana (27 of 33), Delhi
(11 of 11), and the whole of Sikkim, Puducherry, Goa, the Andamans and Dadra & Nagar
Haveli. The index I was mapping is computed for rural areas only. It was never going to
exist for the urban union territories. No amount of fuzzy matching was going to recover
data that was never collected - and if I'd pushed the fuzzy matching harder to close the
gap, I'd have started inventing matches in exactly the region where Indian district names
genuinely repeat across states.
Two things I'd do differently from the start:
Measure boundary-side coverage, always, and put that number next to the map rather than the
row match rate. They diverge exactly when it matters.
Treat "NaN" as null at the type-sniffing stage. The publisher writes the literal string, so
a column of otherwise clean numbers gets typed as text - in that file, 70 of 79 columns.
It's a common habit and it'll bite you again.
The map linked below is the one I ended up with: Mission Antyodaya instead, which is a
rural village survey that covers 680 of 728 districts (93.4%) with zero no-value rows. The
48 that still don't colour are genuinely urban and no dataset on that portal will fix it.
Shading is the share of surveyed villages in each district with no tap water - a ratio, not
a count, because the raw village count just redraws district size.
Live map, no sign-up: https://app.vizzie.org/#example=india-village-infrastructure
Data: Mission Antyodaya via India Data Portal (ODC-BY-1.0)
Disclosure: the tool it's drawn in is mine — I'm building it solo and the examples are
public. Happy to talk about the joining side either way; that's the part I'm still getting
wrong.
Does anyone here have a district alias table they trust, or a workflow that keys on LGD
codes before it falls back to names? That's the piece I'd most like to stop hand-building.
#education
source
I've been joining Indian district-level survey data to district boundaries, and I want to
flag something that cost me a day, in case it saves someone else one.
Indian district data is published keyed by district *name*, not by LGD code, so you're
stuck with name matching. Raw, I got 85.9% of rows onto a boundary. Building an alias table
by hand - the renames, the transliteration variants, the districts that split - took that to
99.5%, which felt finished.
The map still had holes in it. About an eighth of the country.
The problem was that I was reading the match rate as *rows that found a boundary*, which
was essentially 1.0. The number that decides whether a choropleth looks finished is the
reverse join: *boundaries that found a row*. Measured that way, 99 of 728 districts were
blank, and none of it was a name problem:
\- 54 districts had no row in the source file at all
\- 45 had a row whose value was the literal string "NaN"
And the pattern wasn't scattered, which is the tell. It was Telangana (27 of 33), Delhi
(11 of 11), and the whole of Sikkim, Puducherry, Goa, the Andamans and Dadra & Nagar
Haveli. The index I was mapping is computed for rural areas only. It was never going to
exist for the urban union territories. No amount of fuzzy matching was going to recover
data that was never collected - and if I'd pushed the fuzzy matching harder to close the
gap, I'd have started inventing matches in exactly the region where Indian district names
genuinely repeat across states.
Two things I'd do differently from the start:
Measure boundary-side coverage, always, and put that number next to the map rather than the
row match rate. They diverge exactly when it matters.
Treat "NaN" as null at the type-sniffing stage. The publisher writes the literal string, so
a column of otherwise clean numbers gets typed as text - in that file, 70 of 79 columns.
It's a common habit and it'll bite you again.
The map linked below is the one I ended up with: Mission Antyodaya instead, which is a
rural village survey that covers 680 of 728 districts (93.4%) with zero no-value rows. The
48 that still don't colour are genuinely urban and no dataset on that portal will fix it.
Shading is the share of surveyed villages in each district with no tap water - a ratio, not
a count, because the raw village count just redraws district size.
Live map, no sign-up: https://app.vizzie.org/#example=india-village-infrastructure
Data: Mission Antyodaya via India Data Portal (ODC-BY-1.0)
Disclosure: the tool it's drawn in is mine — I'm building it solo and the examples are
public. Happy to talk about the joining side either way; that's the part I'm still getting
wrong.
Does anyone here have a district alias table they trust, or a workflow that keys on LGD
codes before it falls back to names? That's the piece I'd most like to stop hand-building.
#education
source
