Statistics from digital traces: how is big data changing official statistics?

The Babylonian scribe wrote on a clay tablet, Graunt collected church bills, Gallup sent interviewers from house to house. Today every card payment, every phone signal and every online purchase leaves a trace. The new era of statistics is beginning with these traces.

What is "big data"? These are huge volumes of data that accumulate on their own, collected not for statistics but for other purposes: records of payment systems, mobile communications data, online shop prices, satellite images.
The UN decision. At its 45th session in March 2014 (decision 45/110), the UN Statistical Commission approved the establishment of a global working group on big data for official statistics. The group's first meeting took place in Beijing in October 2014.
The principles do not change. The Commission required the group's mandate to be linked to the Fundamental Principles of Official Statistics. These principles were adopted by the UN Statistical Commission in 1994, and in 2014 they were endorsed by the UN General Assembly in resolution 68/261. The group's documents state that these principles apply unconditionally even when private data is used.
Challenges. How to access private companies' data, how to protect privacy, whether the data reflects the whole population: these questions are still being discussed.
The lesson of 1936 is relevant again. Large volume is no guarantee of accuracy. If data comes only from smartphone owners, those without smartphones remain "invisible": just like the poll drawn from the telephone directory.
New and old methods together. That is why statistical offices are testing big data not as a replacement for traditional surveys and censuses but as a source that complements them and speeds them up.

Example (prepared by the author, hypothetical figures). The statistics department of the city of "Bahor" monitors the price of bread.

Method	Coverage	When the result is ready	Strength	Weakness
A price collector goes round the shops	40 shops	At the end of the month	Accurate and verified	Slow, small coverage
Cash register (receipt) data	1 200 points of sale	Every day	Fast and broad	Market stalls without cash registers are left out

If both methods show that bread has risen in price from 4 000 soum to 4 200 soum, the growth rate is: 4 200 ÷ 4 000 × 100% = 105%, that is, a 5% increase. If the two sources confirm each other, confidence in the figure grows. If there is a difference, the statistician looks for the reason: it is exactly this check that distinguishes statistics from mere "big numbers".

The 1980s, the 2000s and today (the general picture, the author's summary).

Aspect	1980s	2000s	Today
Data collection	A paper questionnaire, the interviewer goes from house to house	Questionnaires entered on a computer, electronic reports	Online surveys, registers, cash register and mobile data
Computing tool	Large (mainframe) computers, calculators	Personal computers and statistical software	Cloud computing and big data tools
How quickly results are ready	Months, sometimes years	Weeks, months	Some indicators within days
Main risk	Slowness and manual input errors	Inconsistency between databases	Unrepresentative data and privacy

Statistics from digital traces: how is big data changing official statistics?

See also

Sources