Have We Learned Anything in Nearly 100 Years? #Statistics #BigData #IrishWater via @ad_greenway

In a short story by Andrew Greenway on data and digital, he writes about the work of Geoffrey Drage nearly 100 years ago. In December 1916, Drage presented a paper to the Royal Statistical Society on "The Reorganisation of Official Statistics and  a Central Statistical Office" (available in Jstor). This paper was published in the Journal of the Royal Statistical Society (Vol. 80, No. 1, pp. 31-64) in 1917. 

In 1916 there was no Central Statistics Office in the UK (Ireland of course was part of the UK until 1921), and when I read some of the key findings of Drage's paper I wondered have we learned anything about data and statistics since then. With more news this week from Irish Water of new databases, plus the lack of existing data when the company was set up, and the difficulties they are having with errors - I thought I was reading a list of things wrong with Irish Water (and other bodies), when I read the following findings by Drage:

In 1916, Drage diagnosed six problems with government statistics as follows:

1. Lack of cooperation between the different departments
2. The absence of any central or general supervision of national statistics as a whole
3. Publications (such as the Parliamentary Blue Book) serving conflicting purposes, confusing users like administrative and departmental officers
4. The inclusion of departmental reports of quantities of matter for the purpose of showing how much work is done in a year
5. The fact that compulsory powers are too few and seldom applied
6. Defective supervision in the collection of statistics and the employment, especially for census work, of ill-paid, uneducated, and therefore uninterested persons in the collection.

With the exception of the point #6 above, the others make for interesting reading (though in #6 above you could make a case for front-line staff today being under-valued. It wasn't until 1941 that a Central Statistics Office was set up in the UK when Churchill got fed up with bad data during World War II. The CSO in Ireland was set up in 1949 (we didn't need a war).

It seems to me that problems at the bewildered Irish Water Company with new databases, PPS numbers, data protection, and joining the dots seems to have learned nothing in nearly 100 years. Expect lots more data analysis mess to come!