DATA DREAMING

India is poised to make it easier for the public to access official data, setting in motion a long overdue overhaul of its statistical systems. EPISODE #69

Apr 18, 2022

Dear Reader,

A very Happy Monday to you.

Next month (or around that) India will formally unveil the National Data and Analytics Platform (NDAP), the platform from Niti Aayog to promote a “data-driven” national discourse. The beta version is up and running and a select few have been giving it a go to share feedback with the developers.

So this week I will try and unpack this initiative. On the face of it yes the topic is boring. But trust me it is something that is very relevant to all of us. So please stay the course.

The cover picture for this week is something I stumbled upon while doing a warm-up walk around the tennis courts of a sports facility in Delhi.

A big shoutout to Kapil, Aashish, Yugainder, Abhijit, Yog, Gautam and Vandana for your informed responses, kind appreciation and amplification for last week’s column. Gratitude also to all those who responded on Twitter and Linkedin. Reader participation and amplification is key to growing this newsletter community. And, many thanks to readers who hit the like button😊.

BEGINNINGS

Little over two years ago the National Institution for Transforming India or the Niti Aayog (the institution which replaced the erstwhile Planning Commission) declared its intent to create the National Data and Analytics Platform (NDAP) to promote a “data-driven” national discourse.

The project’s ambition was summed up as follows in a press release:

“The platform will for the first time merge and standardize government datasets from across sectors, time periods, and geographies, bringing valuable data that can be easily used by policy-makers, bureaucrats, researchers, innovators, data scientists, journalists, and citizens.
NDAP will combine publicly available datasets from central and state ministries, and over time will include data from surveys conducted by government agencies.”

In a month from now the NDAP platform will be launched. It is a very important moment in the economic history of India with a government arm willing to process and showcase official data to the public. Too good to be true, right?

The business of access to official statistics is a prickly subject. All government’s are loath to allow easy access, primarily because information is power.

And it is no different in India. Unfortunately the situation here is a little more dire. After a dream start immediately after Independence, the official statistics effort lost steam rapidly. Worse the neglect also undermined some of the processes resulting in a gradual erosion of credibility of the statistical system and the data they generated.

It is in this backdrop the NDAP comes as a breath of fresh air: holding out the promise of unpacking official data for the general public by reinventing the UX (user experience).

NDAP

Accordingly a Project Management Unit (PMU) was set up within Niti Aayog with fiscal support from Omidyar Network India. The technical effort undertaken by the PMU was led by IDinsight along with Development Data Lab (DDL).

Predictably it raised the hackles in sections of the government, especially those who had proprietary claims on official statistics. The fact that D-day has come means the sceptics were contained.

I will try and unpack what NDAP will do for the public. It will take already published data—it will keep adding to this set—clean it up, standardise the format so that they are comparable. Thereafter this will be put out on the platform for anyone to access.

You would appreciate this effort if you have at some stage tried to access a data set from the government—pre or post-Internet era. For some reason (suspect it is a hand-me-down from the colonial era and something which the ever secretive bureaucracy has guarded zealously since) the data is always put out as a pdf document (in the pre-Internet era it was in a physical document).

Today there is technology to scrape this information and convert it into say an excel file. But till recently the only option was to re-key the data; imagine doing this for information gathered by the Census or the various sample surveys. This is classic bureaucratic play: slow them down.

This is the step that the NDAP is seeking to resolve. They are also throwing in some embellishments which will allow a user to do a basic visualisation. The latter is key to overcoming the phobia about big data sets. In fact data science is evolving into a fine art. Check out the sample below from this must follow twitter handle doing outstanding data visualisation by telling the India story through numbers.

India in Pixels @indiainpixels

Listen to the sound of money transacted via UPI from October 2016 to March 2022. 🔉 The music is generated from the data - through a process called data sonification.

In short, what the NDAP is fixing is a section of the front office. An important step indeed. But, in my view, not sufficient in fixing the larger problem with data access.

Cup Half-Full

Herein lies the rub. Yes, the government is willing to loosen its grip and enable easy access to some sets of official data. Part of the problem could be that the NDAP is housed in the Niti Aayog and the ministries overseeing the data will seek official clearance to go the full distance.

Worse, NDAP does not address the flaws surrounding data collection and processing that have emerged over the years and left the official statistical system vulnerable. Institutional knowledge within government indicates that the only honest effort dates back decades when Prof. Prasanta Chandra Mahalanobis took charge of the process.

Not only did he set up the Indian Statistical Institute, an institution devoted to the research, teaching and application of statistics, natural sciences and social sciences, Mahalanobis was instrumental in creating the architecture for the official statistical system. The fact that ISI was set up in 1931, should hint at the vision of the erudite professor. I am sharing a link which uses multimedia to summarise the compelling life story of Prof Mahalanobis.

The work he undertook to study Indian agriculture later became the basis of the sample surveys that are rolled out by the National Sample Survey Office (NSSO). In short Prof Mahalanobis was central to India creating the architecture for its official data and scripting the science to capture these numbers. This included the setting up of the Central Statistical Organisation (renamed later as the Central Statistical Office) to coordinate all data collection.

Later Prof Mahalanobis also assumed the role of the key policy planner for Independent India’s first government by joining as a member of the erstwhile Planning Commission. The centralised planning that was initiated was powered by the data crunching generated by the country’s official statistical machinery. With the benefit of hindsight we may quibble about India’s economic choice, but at that moment it was visionary.

Why?

But this is not a story about the father of Indian statistics. Instead it is about the import of his vision for India to own a statistical system built on sound principles. And about the failure to see the mission to its logical conclusion.

The cost of this was brought home very starkly when the covid-19 pandemic struck two years ago. Overnight the government (including states) were at sea on the status of the medical infrastructure (whatever little that existed). It was a mad scramble to pull together data. Imagine the challenge faced by authorities in charting the next steps to mitigate the fallout of pandemic at a time the extreme versatile and ferocious virus was constantly moving the goalposts.

Similarly, economic management in India has always been based on prudent guess work; on a wing and prayer as it were. Every year the finance minister’s budget is presented based on dated statistics—since most official data comes with a lag. In the pandemic era, the assumptions underlying the annual Budget were undone forcing the government to go back to the drawing board; similarly the Ukraine conflict has undermined some of the key assumptions of this year’s Budget.

Now, you can never plan for a once in a century pandemic or a conflict. But if the country had access to reliable data the fight back would have been that much better. And this fallout is not just restricted to the government. Companies plan their investments based on official growth projections; now if these are flawed then the consequences will be devastating for a company.

The capture of everyday statistics like inflation, growth, jobs and so on are of vital importance that impact all of us. Their sanctity therefore needs to be above reproach and collected on sound principles. Recent controversies around the consumption survey data sets suggest that this is not the case.

It can be safely surmised then that India’s statistical systems are in desperate need of an overhaul. What the NDAP is attempting is but a baby-step. In fact most ministries now feature dashboards capturing and sharing a summary of the latest data sets. While this is welcome, it is largely cosmetic in nature.

It is only when the back office is overhauled—including the setting up of an independent statistical authority—will India have a world-class statistical system that can withstand public scrutiny. This trust quotient is non-negotiable. Hopefully the best is yet to come.

Capital Calculus 2.0

Discussion about this post