Approaches to Data Quality

New approaches continue to emerge among national and international statistical offices on macroeconomic data quality, but a discernible consensus is forming around a multidimensional concept of data quality. Papers and reports on data quality assessments are described and referenced below. Additional examples are welcome.  

The Organization for Economic Co-operation and Development (OECD) created the Short-Term Economic Statistics (STES) Timeliness Framework, which is a tool to assist statistical organizations to improve the timeliness of their short-term economic statistics. The STES Timeliness Framework is a structured collection of documentation on a range of methodological and operational good practices currently used by National Statistical Organizations for improving timeliness, reducing costs or improving accuracy in the production of short-term economic statistics. This framework is presented as a user friendly website at www.oecd.org/std/research/timeliness, where both summary and detailed documentation on methods can be accessed through the main reference table. The current documentation referenced within the framework comes from 18 different countries and will be continuously updated. Guidelines for the submission of documentation, including those written in national language, can be found at: http://www.oecd.org/dataoecd/13/6/33630498.pdf.

The evaluation of the Swiss Federal Statistical Office (FSO) and of the Swiss statistical system, by Ivan P. Fellegi and Jacob Ryten, constitutes the first known example in which the management of a national statistical office voluntarily requested counterparts from another country to conduct a review. The resulting report, "A Peer Review of the Swiss Statistical System (2000)," identifies and describes the strengths and weaknesses of the Swiss statistical system and draws up proposals and recommendations for improvement. Fellegi and Ryten conducted a large number of interviews, both of insiders and outsiders of the system, centered around the following three general questions: how adaptable is the system in adjusting to evolving needs; how effective is the system in meeting existing client needs; and how credible is the system in terms of quality and objectivity. To answer these questions, the reviewers assessed the solidity of the legal and institutional environment, the trustworthiness of the quality of the FSO's products, the masse de manoeuvre (i.e., budget, personnel, access to authority) at its disposal, and the adequacy of the instruments developed by the FSO to carry out its mandate.

W. de Vries, in his paper "How Are We Doing? Performance Indicators for National Statistical Systems, (1998)" proposes a system approach to evaluating the performance of national statistical offices (NSOs) and takes the view that there is a high correlation between the quality of a statistical system and the quality of its products. De Vries uses the United Nations Fundamental Principles of Official Statistics as a general framework to assess the performance of NSOs, provides a brief explanation of each principle, and raises several operational questions related to each principle.

In the paper "Quality Work and Conflicting Quality Objectives, (1998)" T. Holt and T. Jones underscore the multifaceted aspect of the concept of data quality and describe the various facets, identified as accuracy, relevance, coherence and consistency, continuity, timeliness, accessibility, and revisability. The authors highlight some of the conflicts that arise between different facets of data quality such as those between consistency and timeliness and underscore that trade-offs must be made.

The chapter on "Data Quality" in the Australian Bureau of Statistics (ABS) publication entitled Balance of Payments and International Investment Position, Australia: Concepts, Sources, and Methods,(1998) is an example of data quality assessment undertaken by a data producer. The chapter lists ABS dimensions of quality in statistics as accuracy, revisability, timeliness, relevance, comprehensiveness, and accessibility. The concept of each dimension is briefly developed and the quality of data for the 1998 Australian balance of payments and international investment position is assessed against these dimensions.

E. Elvers and B. Rosén, in their chapter on"Quality Concept for Official Statistics (1997)," published in the Encyclopedia of Statistical Sciences, define quality of statistics by referring to how well statistics meet user's needs and expectations for statistical information, once disseminated. The authors suggest that to allow users to assess the quality of the statistics they utilize, producers of official statistics provide neutral, descriptive information about all aspects of statistics that affect users' views on how well the statistics might meet their needs and expectations. They suggest that this information be organized by main quality components, identified as contents, accuracy, timeliness, coherence (especially comparability), availability, and clarity. The authors provide definitions for the main quality components and their subcomponents. They highlight that, although there is wide agreement among the statistical community on what the subcomponents should be, there is no universal consensus on how to group them under the main quality components.