Abstract

10.1 The environment in which statistical agencies operate is changing. New opportunities to access and interrogate big data are becoming available, increasing the potential to provide new insights and possibilities for the compilation of consumer price indices (CPIs). The statistical landscape is becoming more complex, expectations of decision makers are growing, and national statistical offices (NSOs) are being challenged to deliver the best possible statistical program in more efficient and innovative ways.

Introduction

10.1 The environment in which statistical agencies operate is changing. New opportunities to access and interrogate big data are becoming available, increasing the potential to provide new insights and possibilities for the compilation of consumer price indices (CPIs). The statistical landscape is becoming more complex, expectations of decision makers are growing, and national statistical offices (NSOs) are being challenged to deliver the best possible statistical program in more efficient and innovative ways.

10.2 The launch of barcode scanner technology during the 1970s, and its growth in the twentieth century, enabled retailers to capture detailed information on transactions at the point of sale. Scanner data are high in volume and contain information about individual transactions, including date, quantities and values, and detailed product characteristics. The data provided to NSOs are typically aggregated across consumers and across time (week or month). It is a rich data source that can potentially be used to improve the accuracy of the CPI and reduce the costs of physically collecting data; however, it may increase the costs for data analysis and processing. Also, scanner data give opportunities for improved quality in other aspects, such as publishing more detail and new data products.

10.3 This chapter discusses the opportunities and challenges presented when using scanner data to compile the CPI and aims to provide guidance to NSOs on different practices for the treatment of scanner data. It outlines several practical considerations regarding the acquisition of scanner data sets, the assessment and preparation of the data, and implementation issues. New methods that have been developed to construct price indices from scanner data, so-called multilateral methods, are presented. This chapter concludes with a discussion of the assessment of the new methods and the empirical results, communication with users and stakeholders, and publication and dissemination of the “new” price indices.

10.4 Given the dynamic nature of using scanner data for CPI compilation, this chapter provides an overview of country practices. Work on the treatment of scanner data continues, especially with regard to developing elementary aggregate formulas appropriate for compiling an index using scanner data. The treatment of scanner data has been included on the CPI research agenda for further discussion and development.

Practical Considerations

Introduction

10.5 The availability of scanner data provides opportunities to improve the CPI. Scanner data sets typically contain complete coverage of items sold by a retailer at all their locations and include both quantities sold and revenue received by the retailer for these items. This information has the potential to: improve the accuracy of the prices used to compile the CPI by calculating unit values for homogeneous products (see paragraphs 10.26–10.61); improve the samples of items priced, with the potential to use a census of items for the product categories and outlets covered by the scanner data, for example, the product categories in the data provided by a particular supermarket chain; and use quantity or revenue information to weight items according to their economic importance. Scanner data will typically not cover the entire universe that is in scope of the CPI. For example, in most countries, scanner data do not cover services, rents, automobiles, restaurants, or cafes. In addition, this information may only be available for large retail chains but not for small independent stores or other types of outlets.

10.6 While scanner data sets present opportunities to improve the accuracy of the CPI, there are also challenges that need to be addressed before NSOs can use scanner data to compile the CPI.

Obtaining Scanner Data Sets

10.7 Scanner data have existed for several decades and their value in the compilation of official statistics has become more and more evident over time. One challenge faced by NSOs is obtaining the scanner data sets. Two main options are available. NSOs may seek the supply of scanner data sets directly from retail businesses or from third-party data providers. Both options present benefits and challenges.

10.8 Several NSOs have successfully negotiated the supply of scanner data directly from retail businesses and used these data in the compilation of their CPI. Direct collection of data sets from the retail business by NSOs has potential benefits. These include the ability to negotiate:

  • The supply of the data set at no (or minimal) cost

  • The scope of items included in the data set

  • The level of item aggregation to ensure homogeneous information

  • The temporal coverage and detail (day, week, or month)

  • An agreed timetable for the supply of the data set to meet CPI processing requirements

  • A contact officer within the retail business who is familiar with the data set to answer NSO data queries

10.9 Negotiating the supply of scanner data sets directly with retail businesses presents challenges as well. The primary challenge is that the bilateral negotiation of scanner data sets concerns data that may be regarded as confidential because they contain information on turnover and quantities at items level. Another factor is the legal and institutional setting that governs the relationship between NSOs and retailers. In some countries, it may be necessary that the (statistical) law stipulates which data are to be supplied, whereas in other countries a verbal agreement between the parties is sufficient. Experiences in countries using scanner data suggest these negotiations will take at least six months to complete. The negotiations relate to a wide range of topics: from information technology (IT) systems and reporting formats to confidentiality concerns. An agreement reached between the NSO and retail business is typically formalized in a memorandum of understanding (or similar) which documents the roles and obligations of each party and aims to ensure an ongoing supply of scanner data to the NSO according to the agreed timetable.

10.10 An alternate approach to obtaining scanner data sets directly from retail businesses is to obtain these data sets from intermediaries or market research companies. Market research companies possess scanner data sets obtained by some NSOs for CPI assessment and compilation purposes (Krsinich 2015). Market research companies have no legal obligation to provide scanner data; however, for a fee they may be willing to provide older scanner data that would allow the NSO to explore and become more familiar with the data. A better understanding of these data may clarify the requirements before starting negotiations with retailers or market research companies. The primary benefit of this approach is the ability to negotiate the supply of multiple data sets relating to a diverse set of products with a single or small number of data providers.

10.11 The experience of NSOs using scanner data sets to compile their CPI suggests that obtaining data sets directly from retail businesses is preferred. However, obtaining scanner data sets from market research companies is beneficial in those cases where securing data sets directly from retail businesses is not possible or resources are not available to negotiate bilateral data supply agreements. Accessing scanner data from market research companies, in most cases, requires resources to purchase these data.

Assessing and Preparing Scanner Data for Use

10.12 If the NSO is successful in securing access to scanner data sets, these data sets should then be turned into information that can be effectively and efficiently used to compile the CPI. NSOs need to overcome several challenges to achieve these objectives.

Developing an IT System

10.13 Scanner data are by their very nature big data. The size of the fles depends on the characteristics of the underlying data. For instance, fles with daily data by outlet will be more voluminous than fles with weekly data aggregated at the retail chain level. The use of this information by the NSO requires an IT/computing system that can acquire, store, and process the large scanner data sets if the information is to be used to compile the CPI. The IT system needs to be able to acquire and process data sets that have different classification structures, formats, and contents. This is because retail businesses (and third-party data providers) normally develop unique systems for their own internal reporting purposes. These data sets can be used to minimize the burden on retailers and enhance the timely delivery of data to the NSO. IT system development requires human and financial resources. Several NSOs have documented the challenges presented by the need to develop an IT system. The solution is dependent on individual NSO circumstances. NSO resources will be required for an IT system if the NSO is to utilize scanner data to compile the CPI, irrespective of the data provider.

10.14 Given these large investments, it is important for the NSOs to gain experience with test data (for example, from market research companies) and engage with other NSOs with experience. The system needs to be designed to last some time, while at the same time being able to adapt to evolving methodological developments and being scalable to cater for the large volume of data received from an increasing number of data providers. It is recommended that, regardless of the methods that are implemented to produce the CPI, the system remains easily adaptable to newer calculation methods as they evolve.

Classifying Scanner Data

10.15 Scanner data sets generally include product classifications that are unique to the individual retailer. The NSO will most likely receive data sets that contain different product classifications which need to be mapped to a single CPI classification. The classification of scanner data sets may require significant NSO resources. The largest investment of resources is needed when the data sets are first received by the NSO. However, there is a need for ongoing classification resources as new products enter the data set.

10.16 Classifying scanner data items to the CPI classification has been addressed by NSOs in various ways, many of them using the classification of the individual retailer. Such classifications provide important information and can be very useful if they are at the same (or more) level of detail than the lowest level of the Classification of Individual Consumption According to Purpose (COICOP) If the correspondence is 1:1 or n:1 (retailer:COICOP), then scanner data can be mapped automatically. In other cases, scanner data either need to be classified by the NSO or the data are excluded. From time to time, the retailer may also change its classification. The IT system and the classification process should be designed to be flexible so that changes in retailer classifications can be handled in a timely manner.

10.17 NSOs have attempted to find a solution given their circumstances. Some countries have classified scanner data items to the CPI classification by purchasing market research metadata (Müller 2010). One European NSO uses the most detailed classification provided by the retailers and then checks if the mapping is correct and makes appropriate changes if required (van der Grient and de Haan 2010). Some NSOs have, for various reasons, undertaken the entire classification of scanner data items to their CPI classification internally (Howard and others 2015).

10.18 Several NSOs have been exploring the use of machine learning algorithms for classifying scanner data (see Van Loon and Roels 2018). These methods use an input data set either of prelabeled items (supervised learning) or of unlabeled items (unsupervised learning) to predict the correct taxonomy label for each item. The resulting model can then be used to classify new data sets. Machine learning methods are particularly promising where there is a mismatch between the product classifications used by retailers and the classification used for CPI compilation. As with all classification methods, ongoing maintenance is required to ensure that items with new features that have not previously been identified are classified appropriately.

10.19 The challenge of classifying scanner data items to the CPI classification primarily arises when scanner data sets have been secured directly from retail businesses. Obtaining scanner data sets from market research companies may enable the NSO to negotiate the supply of scanner data that have already been classified according to the CPI classification. This is viewed by some NSOs as a particular advantage of obtaining scanner data from market research companies.

10.20 The reliability of the classification system needs to be continuously monitored. Errors that are made at this stage will be reflected in the resulting subindices that may then be compiled based on wrongly classified items.

Quality Assurance of the Scanner Data Sets

10.21 Compared with traditional price collection in outlets, scanner data sets are a new data source to compile the CPI. As is the case with any change in data source, the compilers of statistical series should undertake a range of checks to ensure the new data source provides the foundation from which to produce fit-for-purpose statistics.

10.22 The checks should become routine and performed automatically each production run. Because scanner data sets are new, it is important that NSOs gain some experience with them before using these data in production. The experience gained will facilitate setting the values for the checks.

10.23 These scanner data checks can be classified as either global checks or detailed checks. Global checks occur when the data enter the production process and are part of the acceptance procedure. Detailed checks typically occur toward the end of the production process.

10.24 Global checks relate to broad quality measures generally applied at the time the NSO receives the data set. These checks ensure the data set is broadly consistent with data sets received by the NSO from the same data provider in previous periods. The checks may relate to the format of the data set, the total number of items within the data set, and the total revenue by outlet. These global checks should highlight significant errors with the data set.

10.25 Detailed checks are generally applied at the variety or item group level. These checks aim to highlight significant changes in the quantities sold, revenue, and the prices of the items within the data set. These detailed checks have traditionally been referred to as micro-editing of price data. Unexpected changes in the development of prices, turnover, or quantities will trigger these checks. Processing scanner data means processing much larger sets of data and it may require a different approach than processing traditionally collected data.

10.26 Both the global and detailed checks should be automated to generate reports for analysis by NSO staff. These checks may require contact with the data provider, as well as comparing the data with alternate price information sources (for example, flyers and online prices). The final compiled indices should be reviewed and validated to ensure plausibility.

Implementation—From Confrontation to New Methods

The Benefits and Challenges of Using Scanner Data

10.27 The use of information contained in scanner data sets to compile the CPI can represent a significant change to the data collection practices and the price index methods traditionally employed by NSOs. This suggests that these changes need to be carefully managed, both with regard to the impact on the statistical program as well as communication with users, key stakeholders, and staff. NSO staff need to understand how to manage these data because scanner data sets are so much larger than the traditional CPI data sets. Traditionally, the influence of each price could be traced, and often needed to be visible. In dealing with scanner data, such attention to detail may not be feasible, and these data may require a more top-down approach. Communication with users and key stakeholders is important. It is important for data users to fully understand how scanner data are used in the compilation process. This enhances transparency and user confidence.

10.28 Scanner data potentially enhance the accuracy of the CPI in several ways and provide significantly more data at lower cost. The scanner data sets can be used to (1) compare and validate price data; (2) replace field-collected prices (including a better treatment of sales, promotions, and discounts); (3) expand pricing samples; (4) expand the period over which prices are collected; (5) weight items at the lowest levels of the CPI to reflect their economic importance; and (6) implement new improved index calculation methods and enable process automation. Each of these enhancements is explained in the following text.

Using Scanner Data Sets for Data Validation and Quality Assurance

10.29 The availability of scanner data provides NSOs with the opportunity to validate or quality assure the data used to construct the CPI. Scanner data sets contain variety quantities sold and revenue received by the retailer for these varieties for some period of time, usually a week or a month. This information enables NSOs to calculate a price for an individual variety by dividing a variety’s revenue by the quantity sold. This price, referred to as a unit value, represents the average price experienced by consumers over a period of time. Note, however, that revenue data may not align perfectly with the purpose and concept of the national CPI because it may include expenditure by nonresident households, businesses, or even government (Fenwick 2014).

10.30 For a homogeneous item, the unit value more accurately reflects prices paid by consumers over the whole period than point-in-time pricing (Balk 1998). Unit values contain discounts and the effects of these discounts on the quantity of varieties sold. The period for which unit values are calculated is important with regard to the accuracy of the unit value. Diewert and others (2016) argue that unit value prices used for constructing the CPI should be for the same period as the index to be constructed, rather than for a subperiod.

10.31 It is acknowledged that NSOs may use a subperiod of the reference period due to data supply timeliness and publication deadlines. The bias and variance this introduces can be assessed by comparing indices compiled using a subperiod of data with indices compiled using the full reference period (Krsinich 2015).

10.32 Price analysts can compare the prices collected in the field to those calculated from the scanner data sets. This analysis provides insight into any biases introduced to the CPI from point-in-time pricing compared with unit values. An analysis of the variety’s revenue and quantities sold can be used by the NSO price analysts to highlight where CPI samples could be improved.

Using Scanner Data Sets to Replace Field-Collected Prices

10.33 In most countries, the majority of the prices used to compile the CPI are collected by visits to sampled retail businesses. These visits are made by NSO field officers who observe point-in-time prices as well as discuss discounts, special offers, and volume-selling items with the respondent. The field officers record this information during the visit, often in handheld electronic devices. The regular visits to outlets enable the NSO field officers to actively monitor market developments and observe quality change.

10.34 Replacing field-collected prices by prices (unit values) from scanner data generally results in NSO resource savings, because NSO field officers are no longer required to visit businesses where prices were collected. The potential for NSO resource savings is influenced by the size of the field officer reductions and the increase in resources required by the NSO to manage and process scanner data sets.

10.35 Unit values should relate to a single homogeneous variety whose specification remains constant over time because changes in the composition of varieties sold and the quality of these varieties should not be reflected as price changes. These requirements present some challenges when replacing field-collected prices with information from scanner data sets. Negotiation between the NSO and data provider is needed to ensure access to data at the appropriate level of item aggregation (or disaggregation) required to support the production of unit values for use in compiling the CPI. The direct supply of product characteristics can facilitate the classification of items. Such information, if available, could then be used to perform explicit quality adjustment.

10.36 Several NSOs have experience in producing unit value data from scanner data sets. At the most detailed level, items in scanner data sets are typically identified by barcode or the corresponding Global Trade Item Number (GTIN) or its subvariants, the Universal Product Code, and the European Article Number. While standardized identifiers such as GTIN allow for the tracking of items across different retailers, they may be too detailed, differentiating varieties by characteristics, such as packaging, which are considered irrelevant to consumers (Dalen 2017). Item churn will then be overestimated and there is a potential problem of relaunches which may impede the calculation of the CPI. For example, when using GTIN as the item identifier, the price change of a homogeneous variety whose GTIN changes at the same time will not be measured. In some countries, the use of stock keeping unit (SKU) rather than GTIN has proven to be successful (Howard and others 2015).

10.37 An essential part of price measurement is accounting for quality change and the introduction of new items. This has been achieved by NSOs when field officers visit retail outlets with the aim of measuring price change for identical or equivalent items in successive periods and identifying new items. As the characteristics of varieties are altered, the NSO field officers collect descriptive information that enables the effects of a change in quality to be separated from the price change, so that the CPI measures only pure price change.

10.38 Accounting for quality change is particularly challenging when using scanner data. Scanner data sets tend to exhibit a high level of churn in the varieties available from month to month. There are new models (and versions of models) of items becoming available in the market and old models dropping out of the market as they become obsolete. Calculating quality-adjusted prices is therefore difficult.

Using Scanner Data Sets to Update Pricing Samples

10.39 The collection of point-in-time prices by NSO field price collectors visiting retail outlets is resource-intensive. A census of items cannot practically be priced each period resulting in the need for some sort of sampling approach. For example, sampled products may be selected for inclusion in the CPI basket by NSO field price collectors who discuss with the respondent which items are volume sellers or examine the shelf space of the products and make judgments about their relative importance. NSO field staff then aim to select a representative basket of items for pricing. This is a purposive sampling approach.

10.40 Nonprobability or purposive sampling has traditionally been used because sampling frames for items purchased were not available and detailed quantity or revenue data to measure the economic importance of the items was lacking (see Chapter 4 for further information on sampling). Nonprobability or purposive sampling can lead to biases when the selected items are not representative of the product population.

10.41 This traditional approach to sampling can be replaced by more scientific sampling methods due to the availability of scanner data. Since scanner data typically is a census of products, scanner data sets can be used as a sampling frame for updating pricing samples. A pricing sample is usually two-dimensional; it is a combination of a sample of outlets and a sample of items/product varieties. If all the stores from a retail chain are covered, the scanner data set can be used as a sampling frame for both the outlet and item dimension (see also Chapters 4 and 5). For example, a two-stage sampling approach could be adopted by first selecting outlets and then selecting items within the sampled outlets.

10.42 Revenue shares for each product (or product/outlet combination) can be used to determine the significance of each product within a product group. Products are then selected for inclusion in the CPI “basket” based on revenue share either through sampling proportional to revenue or cutoff sampling (de Haan and others 1999).

10.43 Over time, however, products in the sample can lose relevance or even cease to exist. In these situations, a replacement product is needed to maintain the relevance of the sample. Relevance tests can be used to highlight those items in the samples that have become unsuitable, and also highlight and rank suitable items as replacements.

10.44 The main principle behind these relevance tests is that the products should have a stable revenue share (that is, consistent revenue share compared to other products), within the CPI product group. These product groups are referred to as the elementary aggregate. The stable revenue share is important as items can have large sales when introduced into the market due to novelty or introductory sales prices, have insignificant revenue thereafter, and hence not be representative of the broader market.

10.45 To mitigate these problems, possible replacement products’ revenue must have been stable and significant for a specified period (for example, three to six months) before they can be considered for inclusion in the price samples. CPI analysts can then manually review all items which are fagged for replacement and select items from a list ranked according to average monthly revenue share over the previous six months.

10.46 Many food and household items will have varieties of the same base item which have similar if not identical price evolution. A specific brand of canned tuna, for example, is available in many favors and CPI compilers will be aware that prices for the different favors from the same brand will behave similarly, going on sale at the same time and changing price at the same time. Having a single favor in the sample will hence represent the price movement for a much more significant portion of the market than that single favor’s revenues would suggest.

10.47 The sampling process used to ensure product samples are representative is usually manually driven, requiring the CPI analysts to select a replacement from this ranked list of potential products that pass certain eligibility criteria. This scanner data sampling approach requires additional CPI analyst resources which, ideally, are offset by the reduced field collection resources.

Using Scanner Data to Update Index Structures and to Apply Weights

10.48 Variety samples have traditionally been small. When the additional CPI analyst resources are indeed offset by the reduced field collection resources, the NSO could decide to expand the variety samples without changing the price index formula at the elementary aggregate level or the sampling procedure.

10.49 It may be worthwhile, however, to reconsider the index structure and the sampling procedure when the NSO obtains scanner data directly from retail chains. Traditionally, an elementary aggregate index is compiled from prices collected at outlets that belong to different retail chains (or independent stores). When the NSO wants to use much more price information from a retail chain than before, it seems preferable to treat elementary aggregate1-chain combinations as separate strata in the index compilation process.

10.50 If the NSO chooses to use the classification system provided by the retailer, it will be necessary to expand the index structure below the elementary aggregate level with separate elementary price indices for each retail chain. This raises several issues. The first issue is whether the stores belonging to the chain in question should be viewed as separate outlets. If this is the case, unit values for the sampled items should be calculated at the store level. On the other hand, when the service levels are similar across stores belonging to a retail chain, it may be useful to calculate unit values at the chain level (Ivancic and Fox 2013). In this case, the retail chain scanner data directly represent all the stores of that chain. When using chain-level data, one must ensure that each retail chain is weighted in the final index. Some retail chains operate different types of stores with different assortments of products and price levels. In this case, a stratification can be introduced that distinguishes between the types of stores belonging to a retail chain. Some NSOs do not have a choice if they receive scanner data at the chain level.

10.51 A regional disaggregation of the scanner data may be needed if regional CPIs are compiled. The chains, stores, products, and prices included in the resulting indices should be representative for the respective region(s). Alternatively, scanner data can be directly exploited at the national level. It has then to be determined whether the resulting scanner data price indices, which are representative for the country as a whole, can also be used to compile the regional CPIs.

10.52 The next issue to be considered by the NSO is to what extent existing sampling procedures should be improved. Suppose the NSO formerly used sampling of items proportional to revenue from the scanner data. This procedure can also be used to sample items from chain-specific elementary price indices, where items are either defined (and unit values calculated) at the outlet level or the chain level. If the NSO wants to significantly increase the sample sizes to make use of a substantial part of the price information contained in the scanner data set, sampling procedures need to be reconsidered.

10.53 Another issue is how to integrate the chain-specific elementary price indices from scanner data with price information from other sources. Because these elementary price indices are different from the elementary price indices in the traditional index structure, the scanner data price indices have to be aggregated up to a level—perhaps the lowest level of product aggregation the NSO publishes price indices—where they can be combined with price indices from other sources. If a retailer-specific classification is used, it must be as detailed as the lowest level of product aggregation that the NSO publishes. If not, the data need to be reclassified accordingly. In other words, two aggregation steps are required: aggregation of the chain-specific elementary price indices up to some higher-level product category, and aggregation of the resulting scanner data indices with price indices at that level pertaining to other retail chains and independent stores.

10.54 The revenue data provides the opportunity for NSOs to weight price indices more frequently using more timely data. This can be achieved in various ways, depending on the availability to the NSO of scanner data for multiple chains. It is suggested that the weights to combine the price indices from scanner data be updated annually, using product revenue data from the previous 12 months. Combining the scanner data indices with the price indices compiled from other sources requires expenditure data for the latter indices, which may be difficult to come by or estimate.

10.55 With the household budget survey, detailed data by item (or item/outlet combination) are not available. The majority of NSOs therefore apply unweighted price index methods at the lowest levels of the CPI: the prices or price changes of the items sampled from an elementary aggregate are combined without explicitly weighting the items according to their economic importance. In most cases, the Jevons index formula is used by NSOs.

10.56 Scanner data sets contain revenue data at the most detailed variety level. These data can be used to sample varieties proportional to their revenue, as mentioned previously, but this raises a few issues. The inclusion probabilities serve as implicit weights. That is, the elementary price index will be an implicitly weighted index, and the inclusion probabilities should correspond with the target/population index aimed at (Balk 2005). Moreover, the revenue distribution within an item category as observed in scanner data is often highly skewed. Consequently, sampling proportional to revenue is likely to select some high-revenue varieties with a probability of one, and the high-revenue items in this “self-selecting” subsample should be explicitly weighted— without explicit or implicit2 weighting of these items the (unweighted) sample-based Jevons index cannot be an unbiased estimator of a weighted geometric target index.

10.57 It is preferable to reflect the items’ economic importance explicitly via a weighted index number formula rather than implicitly via the inclusion probabilities in an unweighted index.

Using Scanner Data Sets to Implement New CPI Compilation Methods

10.58 The approaches outlined previously enable the NSO to continue using sample-based methods to compile their CPI. Improvements to the accuracy of the CPI will be achieved because the prices (unit values) are more representative of those actually paid by consumers. Also, the varieties sampled reflect volume sellers, and the weights used to produce aggregate measures of price change are based on more timely information and can be updated more frequently.

10.59 The major challenge faced by NSOs implementing these approaches relates to the increase in resources needed. Maintaining a sample-based approach, especially when the variety samples are extended, requires significant manual intervention, primarily because variety turnover can be large. When one European country first introduced scanner data for supermarkets into the CPI, a Lowe index was used (Schut and others 2002). The idea was to mimic traditional methods and processes on a sample of about 10,000 items (barcodes) from each supermarket chain. This approach was very demanding with regard to the manual selection of items to replace disappearing items and with regard to quality adjustments when deemed necessary.

10.60 Ideally, acknowledging practical constraints, an NSO would use all the available information in scanner data sets rather than taking samples. Manually processing a census of varieties from scanner data sets is prohibitively expensive, however, and cannot be undertaken to meet the CPI production timeframes. Automating CPI compilation processes is required.

10.61 Also, when using a census of varieties, not a sample, a weighted index number formula is preferred. Again, variety turnover poses a significant problem. To maximize the number of matches in the data, chaining at high frequency will be needed. This can lead to a significant drift in the index. Multilateral price index number methods, which are drift-free by construction, are currently considered a suitable method to handle a census of items and varieties from scanner data; however, work continues on how best to apply multilateral methods to compile elementary aggregates using scanner data. The scanner data offer many opportunities for new research and developments.

Multilateral Price Index Methods

Introduction

10.62 Scanner data can be implemented in the CPI using traditional sample-based methods. Prices formerly observed by price collectors visiting outlets can be replaced by unit values from scanner data without changing the sampling design or the price index number formula used. If the NSO decides to use all the available data rather than taking samples, the preferred approach, multilateral price index number methods are most suitable. Multilateral methods were originally developed to compare price levels across countries, but they can be adapted to price comparisons over time. These methods are particularly useful for scanner data, where item turnover is often high and promotional sales occur frequently.

10.63 The most important multilateral price index number methods are described in the following text; for a comprehensive discussion, see Chapter 7 of Consumer Price Index Theory. After defining the variety, a short overview of traditional bilateral price indices and chaining is provided.

Defining the Variety

10.64 Before applying any index calculation method, the individual variety to be priced must be defined. The basic principle is to compare like with like and to track the price of the same variety over time. The article code level usually represents the most detailed level of homogeneity in the data. In addition to this product dimension, one must also consider the outlet and the time dimensions. Often, the same article sold at different moments of time, in the same or similar outlets can be considered to define a variety which is sufficiently homogeneous so that an average transaction price (unit value) can be calculated for the variety.

10.65 In some cases, the article codes such as GTINs are stable and long-lived. Some countries have access to retailers’ product codes, for instance, an internal SKU code that is already more aggregated than the GTIN code. However, in other cases, these article codes level may be too detailed for price index calculation. In some product categories, such as clothing and footwear, the article codes frequently appear and disappear making it difficult to match them across time and therefore price changes are not adequately measured. The different strategies to cope with the article code changes are described in the paragraphs that follow.3

10.66 One approach is to group together different individual articles with similar characteristics. NSOs can choose to define the variety more broadly or more narrowly. It is important to create groupings so that consumers are more or less indifferent between the different individual articles within these groupings. Calculating unit values at this level not only makes it possible to capture substitution effects between comparable articles but also facilitates the inclusion of new articles entering the market. NSOs face tradeoffs in balancing these groupings. If a grouping is too broad, this can lead to unit value bias (and high volatility) as the individual articles are not strictly comparable. On the other hand, defining the grouping too narrowly may lead to a lack of matching between outgoing and new or returning articles. Decisions made at this stage can significantly impact the price indices that are eventually obtained. This could be especially relevant for technology products, especially models with high turnover (see Chapter 6 for more details). If feasible, the sensitivity of the definitions of the groupings on the results should be tested.

10.67 The practical construction of these groupings can be challenging. NSOs require information about article characteristics, including brand and size, as well as internal classification codes used by retailers. Some retailers may only provide characteristics in a specific text string while others may have several different variables that describe the characteristics of the different articles. Characteristics gathered in one single variable (text string) require some form of text mining to make it useful for classification. Not all characteristics are equally important and affect the price in the same degree. The clustering of individual articles should be defined by the most important price-determining characteristics.

10.68 Another approach would be to impute the prices for the new and disappearing articles in the periods when they are not available. Prices could, for instance, be imputed with the help of a hedonic function. Instead of using the characteristics of the articles to form the groupings, this information is now used to estimate missing prices. However, such a strategy is only appropriate for index number formulas, such as the Törnqvist or the Gini, Eltetö, Köves, and Szulc (GEKS)-Törnqvist, which are responsive to imputations for missing prices.

Bilateral Price Indices and Chaining

10.69 Suppose first that the set of varieties sold is fixed over time (that is, that the price statistician is dealing with a static universe). This fixed set of items is denoted by S and its size by N. The sample period consists of (T +1) time periods t = 0,...,T. The prices (unit values) of item i e S (i = 1,...,N) in periods 0 and t (t = 1,...,T) are denoted by pi0andpit;qi0andqit are the corresponding quantities sold. The aim is to construct price indices that compare period 0, the starting period of the time series, with each period t.

10.70 In the situation with no expenditure information, Chapter 8 of this Manual recommends the use of the Jevons price index, the unweighted geometric mean of price relatives:

IJ0:t=ΠiS(pitpi0)1n=ΠiS(pit)1nΠiS(pi0)1n(10.1)

10.71 The NSO traditionally draws a sample of items from the entire universe S to reduce CPI production cost. Without access to scanner data, S is unknown and a detailed sampling frame is lacking. Most CPI samples have therefore been drawn purposively, with the risk of introducing bias in the index.

10.72 Since scanner data contains expenditure information for a census of items, the construction of superlative price indices is possible on the entire set S. The focus here is on the Törnqvist rather than the Fisher index or other superlative index formulas. While the Fisher and Törnqvist produce very similar results, the Törnqvist index allows for simpler expressions. The Törnqvist price index is given by

IT0:t=ΠiS(pitpi0)si0+sit2(10.2)

where si0=pi0/ΣiSpi0qi0andsit=pitqit/ΣiSpitqit denote the expenditure shares in periods 0 and t.

10.73 In a dynamic universe, there are new and disappearing items so that not all items can be matched over time. The sets of items in periods t (t = 0,...,T) are denoted by St with size Nt. To maximize the number of matches in the data, chaining matched-model superlative price indices seems useful, for example, chaining period-on-period Törnqvist price indices:

ITt1:t=ΠiSMt1:t(pitpit1)si,Mt1+si,Mt2(10.3)

where SiMt1andSiMt are the expenditure shares in the two periods with respect to the set SiMt1,t=St1St of matched items that are available in both period t - 1 and period t.

10.74 However, empirical work showed that high-frequency chaining of weighted price indices, including superlative price indices, can lead to strong chain drift. In case of promotional sales with reduced prices, the quantities purchased often increase substantially. But when the prices return to their original level, the quantities purchased of storable goods may not return to their “normal” level. This type of asymmetric behavior can cause chain drift in superlative price indices, which is typically downward. Ivancic (2007), using market research scanner data on goods sold in supermarkets, found a downward drift in chained Fisher price indices (see also Ivancic and others 2009, 2011). Drift in chained matched-model superlative price indices has been documented for durable goods as well. Here, the drift is likely due to seasonal fluctuations in prices and quantities. De Haan and Krsinich (2014), using scanner data, found a downward drift in chained Törnqvist price indices for consumer electronics goods sold. Silver and Heravi (2005) presented evidence of downward bias in chained Fisher indices using scanner data on televisions.

10.75 Table 10.1 shows a numerical example of downward drift in the chained Törnqvist price index. There are two items and nine periods distinguished. The “regular” prices of items 1 and 2 are 3.00 and 4.00, respectively, but the price of item 1 is temporarily reduced in periods 3 and 7, while the price of item 2 is reduced in periods 2 and 6. Notice that in the last period (period 9) the prices and quantities are exactly the same as those in the first period. Nevertheless, the period-on-period chained Törnqvist price index ends up at 78.18, thus measuring a price decline of almost 22 percent. For the multilateral indices, the index equals one (as will the direct Törnqvist). This downward drift for the period-on-period chained Törnqvist stems from the fact that because the quantities do not immediately return to their “normal” level after the discount, the price change from the normal price to the reduced price has a bigger weight than the following price change from the reduced price back to the normal price.

Table 10.1

A Numerical Example of Chain Drift

10.76 One way to avoid chain drift due to promotional sales for storable goods would be not to weight the items and to construct a time series by chaining period-on-period matched-model Jevons price indices:

IJt1:t=ΠiSMt1,t(pitpit1)1NMt1,t(10.4)

where NMt1,t is the number of matched items between periods t - 1 and t. This is not to say that the use of the chained matched-model Jevons index is without problems. For instance, clearance sales can put downward pressure on the index. In order to mitigate this problem, a dump filter can be implemented. The filter removes an item if both the price and the quantities sold of an item fall sharply. A downward drift may also arise for fashion goods, such as clothing, that exit the sample at low clearance prices and never return. Clothing requires special treatment as noted in Chapter 11.

10.77 The lack of weighting is problematic as well. Item expenditures are usually highly skewed, and so the many low-expenditure items will be given the same weight as the few high-expenditure items. A crude form of implicit weighting can be attained by simply excluding low-expenditure items (that is, by giving them an inclusion probability of zero), for example, using a threshold based on the items’ average expenditure shares in adjacent months. This approach, sometimes referred to as the “dynamic approach” (see Eurostat 2017), has been implemented by several European countries (for example, see van der Grient and de Haan 2010, 2011). This method reduces the risk of chain drift because weights are used implicitly for the sampling of items, but not explicitly in the index calculations. This method has the advantage that it relies on the usual bilateral index methods while at the same time making the best use of the information contained in the scanner data sets. The method is therefore easy to explain to users. Yet, this is not an optimal situation. A more advanced solution would be to explicitly weight the items and construct weighted multilateral price indices.

Multilateral Methods

10.78 Multilateral methods produce transitive price indices. For price comparisons across time, this means the indices are independent of the choice of base period, can be written in chained form, and are therefore free from chain drift. Multilateral methods have in common that price indices are constructed simultaneously for the entire sample period.

10.79 Two types of multilateral methods can be defined. The first type starts from matched-model price comparisons between any pair of time periods across the entire sample period and then “transitivizes” this set of bilateral price indices. The best-known method is GEKS (Eltetö and Köves 1964; Gini 1931; Szulc 1964). An alternate method is based on spanning trees (Hill 1999a, 1999b), where a spanning tree is a supplier of paths between countries. For a certain spanning tree, the bilateral indices are chain linked to construct price comparisons between any pair of countries or, adapted to our context, time periods. It is not clear, however, what the theoretical and practical advantages are over the easier-to-construct GEKS indices. The second type of multilateral method attains transitivity in another way, which will be explained in the following text, and includes the Geary– Khamis method (Geary 1958; Khamis 1972) and the Country Product Dummy method (Summers 1973).

GEKS Method

10.80 The GEKS index between period 0 and period t is calculated as the geometric average of the ratios of the matched-model bilateral price indices I0,j and I0,k, constructed with the same index number formula, where each period l is taken as the base. Provided that the bilateral indices satisfy the time reversal test, the GEKS index can be written as (de Haan and Van der Grient 2011; Ivancic and others 2011):

IGEKS0:t=Πl=0T(I0,lIt,l)1T+1=Πl=0T(I0,l×Il,t)1T+1(10.5)

10.81 The time reversal test requires that when the base period and the comparison period are reversed, the result should be equal to the reciprocal of the original index. In its standard form, the GEKS method uses bilateral Fisher indices, which satisfy the test, but other choices are possible, including bilateral Törnqvist indices. GEKS–Törnqvist indices are also referred to as Caves, Christensen, and Diewert indices.

Geary–Khamis Method

10.82 The Geary–Khamis (GK) method, when applied to comparisons over time, gives rise to the following price index:

IGK0:t=ΣiStpitqitΣiStpiqitΣiS0pi0qi0ΣiS0piqi0=[ΣiStsit(pitpi)1]1[ΣiS0si0(pi0pi)1]1(10.6)

10.83 The numerator of equation 10.6 is a price index (using period t quantities) with “reference prices” pi that are fixed across the sample period. The index should be equal to one in the starting period 0, so it will be necessary to normalize the index by dividing by its period 0 value, which is the denominator of equation 10.6. The reference prices are given by

pi=ΣτSiqiτ(piτIGKτ0,tau)ΣτSiqiτ(10.7)

where St is the set of time periods in which item i is actually sold and for which prices are available. Equation 10.7 shows that pi equals a weighted arithmetic average of the deflated observed prices, with each period’s share in the total number of sales of the item across the entire sample period serving as weights.

10.84 Since the GK index acts as the deflator in equation 10.7, equations 10.6 and 10.7 define a system of equations which must be solved simultaneously. This can be done iteratively (which may be simpler to implement), but there are other ways to solve the system (Balk 2008).

Time Product Dummy Method

10.85 This is a regression-based approach. Assuming n different items are observed in the entire sample period 0,...,T (most of which will typically not be sold in every time period), the time product dummy (TPD) regression model for the pooled data is

lnpit=α+Σt=1TδtDit+Σt=1N1γiDi+εit,(10.8)

where Di is a dummy variable that has the value of one if the observation relates to item i and zero otherwise, and Dit is a dummy variable with the value one if the observation relates to period t and zero otherwise; dummies for item n and period 0 are excluded to identify the model.

10.86 Diewert (2005) proposed to estimate model (10.8) by Weighted Least Squares regression with the items’ expenditure shares in each period serving as weights. Exponentiating the estimated time dummy parameter δ^t yields the TPD index between periods 0 and t; ITPD0,t=exp(δ^t). The weighted TPD method can be written as a system of equations that is similar to the geometric GK-style system defined by (10.6) and (10.7), and thus, the TPD can be solved iteratively as well as via direct regression methods (Rao 2005):

ITPD0:t=ΠiSt(pitexp(γ^i))sitΠiS0(pi0exp(γ^i))si0(10.9)
exp(γ^i)=ΠτSi(piτITPD0,t)siτΣτSisiτ(10.10)

10.87 Equation 10.10 shows that the exponentiated item fixed effect estimates γ^i, or reference prices, are equal to the expenditure-share weighted geometric averages of the deflated prices with the TPD index serving as deflator. Both GK and TPD explicitly arrive at reference prices. In the case of GK, this means that the index is consistent with national accounts, as it is additively decomposable. TPD, being a geometric index, is not.

10.88 Notice that the TPD index (10.9) can be viewed as a normalized geometric Paasche index with imputed period 0 prices based on the reference prices (10.10). Similarly, the GK index (10.6) can be viewed as a normalized (ordinary) Paasche index with imputed period 0 prices based on the reference prices (10.7).

Lack of Matching and Quality Adjustment

Implicit Quality Adjustment

10.89 Like GEKS, GK and TPD are matched-model methods in the sense that items with a single observation in the entire sample period do not affect the index. Items contribute to aggregate price change only when price relatives can be calculated from the prices observed in both periods compared, unless information on characteristics would be available to perform explicit quality adjustments. One implication of the matched-model method is that items introduced in the most recent period T are ignored.

10.90 Implicit quality adjustment can also be illustrated by using an alternate interpretation of the GK index. Dividing the value index of the product category by the ratio of “quality-adjusted quantities” defines a quality-adjusted unit value index (de Haan 2004, 2015):

IQAUV0:t=ΣiStpitqitΣiS0pi0qi0ΣiS0λi/bqitΣiS0λi/bqi0=ΣiStpitqitΣiStλi/bqitΣiS0pi0qi0ΣiS0λi/bqi0=[ΣiStSit(pitλi/b)1]1[ΣiS0Si0(pi0λi/b)1]1(10.11)

10.91 If λi/b = 1 for all i, equation 10.11 simplifies to the ordinary unit value index. The quality-adjustment factors λi/b aim to express the quantities purchased of each item i with regard to quantities of an arbitrary item b. The ratios pit/λi/bandpi0/λi/b in the second expression of (10.11) are quality-adjusted prices. In the static universe case (with no new and disappearing items), if the quality-adjustment factor of an item corresponds to its base price, the quality-adjusted unit value index turns into the Paasche price index. Von Auer (2014) argued that many conventional price indices can be viewed as, what he called, a generalized unit value index.

10.92 A comparison of equations 10.6 and 10.11 shows that the GK index can be viewed as a quality-adjusted unit value index where the quality-adjustment factors are measured by the reference prices in equation 10.7. Similarly, the TPD index in 10.9 can be viewed as its geometric counterpart where the quality-adjustment factors are measured by the reference prices (10.10). Whether the reference prices in the GK and TPD indices properly reflect quality differences is likely to depend on the market circumstances.

Explicit Quality Adjustment

10.93 Data on item characteristics permitting, explicit quality adjustment is preferred, in particular through hedonic regression methods. A useful starting point is the multilateral time dummy hedonic (TDH) model:

lnpit=α+Σt=1TδtDit+Σk=1Kβkzik+ϵit(10.12)

where zik is the quantity of characteristic k (k = 1,...,K) for item i. Notice that, as pointed out by Aizcorbe and others (2003) and Krsinich (2016), the TPD model (10.8) arises from the TDH model (10.12) by replacing the hedonic effects exp(Σk=1Kβkzik) by item-specific fixed effects exp(γi). Similar to the estimation of the TPD model, it is assumed that equation 10.12 is estimated by expenditure-share weighted regression on the pooled data of all time periods t = 0,...,T.

10.94 The resulting weighted TDH index, ITDH0,t=exp(δ^t), can be expressed in a similar way as the TPD index (10.9), with the estimated hedonic effects exp(Σk=1Kβkzik) instead of the estimated item fixed effects exp(γi) now acting as reference prices. The formula exp(Σk=1Kβk(zikzbk)) can also be used to estimate the quality-adjustment factors γi/b in equation 10.11. The resulting explicitly quality-adjusted unit value index—or “hedonic variant” of the GK index—is expected to be very close to the TDH index (de Haan and Krsinich 2018).

10.95 De Haan and others (2016) compared the TPD and TDH methods. They argued that the TPD model suffers from overfitting because it has too many parameters and “distorts the regression residuals towards zero.” Under certain pricing strategies of retailers, such as price skimming (new items) and dumping (old items), the TPD index can be quite different from the TDH index. Similarly, the GK index can be quite different from its hedonic counterpart, the quality-adjusted unit value index.

10.96 If relaunches of homogeneous items with different barcodes or SKUs are a major cause of a low matching rate, then defining items by their characteristics rather than barcode or SKU could be an option (Chessa 2016). However, scanner data sets provided by retailers typically include rather broad item descriptions, from which it may only be possible to extract a few characteristics, such as package size and brand name. In that case, the prices, calculated as unit values across all the barcodes of SKUs that belong to the various “groups,” can suffer from unit value bias.

10.97 Unlike TPD and GK, GEKS does not aim at implicitly adjusting for quality change. A potential advantage of GEKS over TPD and GK, however, is that the “missing prices” of the unmatched new and disappearing items can be imputed. It is therefore possible to estimate explicitly quality-adjusted GEKS indices by replacing the bilateral matched-model Törnqvist price indices by bilateral hedonic imputation Törnqvist indices, for example, as proposed by de Haan and Krsinich (2014) or de Haan (2019). This means there is no need to define the items by their characteristics; barcode or SKU will suffice to distinguish between items. The hedonic imputations for the unmatched items adjust for quality changes and also deal with the relaunch problem. De Haan (2019) suggests using the same characteristics information in the hedonic regressions that would be used to define the “groups” for dealing with relaunches in the TPD and GK methods.

10.98 Missing information on important characteristics makes the use of hedonic quality adjustment, or explicit quality adjustment in general, problematic as this can lead to an omitted variables bias. Also, as mentioned previously, it may give rise to unit value bias in the “group approach.” A few NSOs have been exploring the use of web scraping to observe quality characteristics from retailers’ or manufacturers’ websites to enrich scanner data sets. Scanner data obtained from a market research company may already contain detailed information on item characteristics. One NSO, for example, produces quality-adjusted GEKS price indices for consumer electronics products based on scanner data sets from one market research company that include many item characteristics (Krsinich 2015). All these methods are rather data-intensive as they require prices, turnover, and price-determining characteristics at a detailed level.

Revisions in Multilateral Indices

10.99 When new data become available, previously estimated multilateral indices change when new data are processed. This is problematic as the CPI is not revisable. Also, as time passes, recent price movements will be increasingly affected by prices and price changes in the distant past. In the context of the GEKS index, this is known as a loss of characteristicity. Different methods have been proposed to extend a multilateral time series without revising published index numbers (and mitigate the loss of characteristicity). The methods can be characterized by a number of choices: window adjustment (rolling window or extending window), the link period, and the index in the link period (Chessa 2019).

10.100 Rolling window methods estimate multilateral indices on a rolling window with fixed length, say T + 1, which is shifted forward each period. Table 10.2 illustrates a rolling time window of 13 periods. The multilateral index compiled in period 13 covers periods 1–13. The multilateral index compiled in period 14 covers periods 2–14 and so on. The results of the latest window are then linked onto a previously calculated index. For example, movement splice links the most recent period-to-period index change onto the index of the previous period (that is the latest published index). In Table 10.2, the movement splice index of period 14 is obtained by linking onto period 13 the price change between periods 13 and 14 derived from the corresponding multilateral index.

Table 10.2

Movement Splice Linking with Rolling Window of 13 Months

The splicing starts in period 14 (shown in bold). The published indices for periods 1–13 are obtained at the first compilation round. The published index in period 14 is obtained by applying the change between period 13 and period 14 indices of the second compilation round to the published index of period 13 (103 8 × 104.6/103 3 = 105.1). The published index in period 15 is obtained by applying the change between period 14 and period 15 indices of the third compilation round to the published index of period 14(105 1 × 104 1/104 4 = 104 8).

10.101 An alternative to movement splice is a window splice, which links the full period index change onto the latest calculated index of T periods ago. The movement splice was proposed by Ivancic and others (2011) for the GEKS method, and the window splice was proposed by Krsinich (2016) for the TPD method. However, each splicing method can be combined with any multilateral method. These splicing methods link index changes onto a single link period. Diewert and Fox (2017) proposed a mean splice by taking the geometric mean of the price indices obtained from using every possible link period. This makes the result independent of the choice of link period.

10.102 Chessa (2019) pointed out that there are in fact two main options for splicing methods (apart from movement splicing, which has the index published in the previous period as the only link option). Successive window shifts generate a sequence of recalculated or “revised” indices alongside the initially published index in the same period. Both the recalculated and published indices are candidates for the index on which a new index series can be linked. In empirical research, and in applications by two NSOs, the first option has been applied. The second option (that is, linking onto the published index numbers) has been recently suggested by Chessa (2019). Linking onto the published index numbers has advantages. For example, year-on-year rates (inflation figures) calculated from shifted windows will also be the published figures if the link period corresponds to the period which is 12 months ago. This increases the transparency of splicing methods. Moreover, each year-on-year rate is derived from a transitive index series and is in that sense free from chain drift. This is not the case when splicing on the latest recalculated index which can therefore still lead to some drift (see Chessa 2019).

10.103 The choice of window length is a point of concern. Ivancic and others (2011) advocated a 13-month (or five-quarter) window as this is the shortest window that can deal with strongly seasonal goods. However, recent research suggests that indices for strongly seasonal items can be significantly improved with a 25-month (or nine-quarter) window (Chessa 2020). It is possible to construct weighted GEKS indices which consider the reliability of the bilateral price indices (Rao 2001b). Melser (2018) proposed a weighted GEKS method where the weights depend on the degree of matching of the items, for example, with regard to expenditure shares. Here, the choice of window length is less important since bilateral indices with a lower degree of matching will be down weighted.

10.104 The annually chained direct extension method (Chessa 2016) constructs multilateral index series of, say, 13 months, starting in, for example, December and ending in December of the next year, and chain links them in December of each year to obtain a long-term time series. The length of the estimation window for the short-term indices is extended each month—the index for January in the short-term series is estimated on two months of data (which is a bilateral rather than multilateral comparison), and so forth, until in December 13 months of data is used.

10.105 A potential weakness of the direct extension method is that the price indices for the first couple of months of each year are based on sparse data and expected to be volatile. Also, December acts as the short-term index reference period and is given special importance. If, for some reason, December is an unusual month, the results may be adversely affected. To mitigate these problems, Lamboray (2017) suggested combining the annually chained direct extension method with a rolling window approach.

Implementation of Multilateral Methods

Assessing Multilateral Methods

10.106 The implementation of new data sources and methods in any statistical series requires careful consideration of the statistical impacts as well as the benefits and costs. Only a handful of NSOs have implemented multilateral price indices in the CPI.

10.107 A suggested criteria to assess multilateral methods should consider a broad concept of statistical quality. A possible framework could include seven dimensions of statistical quality:

  • Institutional environment—pertains to the institutional and organizational context in which a statistical producer operates

  • Relevance—pertains to how well a statistic meets user needs

  • Timeliness—pertains to how quickly and frequently the statistic is published

  • Accuracy—pertains to how well a statistic measures the desired concept

  • Coherence—pertains to how consistent the statistic is with sources of related information

  • Interpretability—pertains to the information available to provide insight into the statistic

  • Accessibility—pertains to ease of access to the statistic

10.108 Of note, multilateral methods are more complicated than standard bilateral indices and present communication challenges for NSOs. High value should be placed on transparency to explain the statistics published, and describing and justifying the methods used. This is of critical importance for the trust in the published CPI. Two aspects of interpretability need consideration: first, to what extent the methods themselves are easy for index practitioners and users to understand; and second, whether it is easy to understand the price movements each index produces, especially which products have the greatest influence on these movements and why.

10.109 This framework can be used by an NSO to determine the benefits and challenges of using multilateral index methods considering country-specific circumstances.

10.110 Multilateral index methods to compile the CPI can also be assessed from a theoretical perspective. The assessment can use approaches previously applied to bilateral and spatial indices. The bilateral price indices are assessed both from axiomatic/test approaches (Chapter 3 of Consumer Price Index Theory) and economic approaches (Chapters 4 and 5 of Consumer Price Index Theory). Similar approaches to assessing multilateral indices in a spatial context have been developed and presented in several papers, especially Diewert (1999b) and Balk (2001).

10.111 A description of the theoretical assessments of multilateral price index methods in the present temporal context using the axiomatic/test and economic theory approaches is available (ABS 2016a; Zhang and others 2019). This assessment can be used as a basis for NSOs to undertake similar assessments in their local context. A comprehensive discussion of the various multilateral methods using the economic approach to index number theory can be found in Chapter 7 of Consumer Price Index Theory; see also Diewert and Fox (2017). The most important result is that GEKS deals appropriately with substitution effects whereas GK and TPD are appropriate only under restrictive assumptions about consumer preferences. However, in practice, GK and TPD generate very similar results. Another NSO developed a generic processing method based on the GK (see Chessa 2016). In a first step, the method was only implemented for mobile phones before being applied to more products and retailers in the following years.

10.112 In addition, expert peer review of the proposed multilateral methods may be appropriate in circumstances where CPI users may expect NSOs to demonstrate broader endorsement of the proposed changes.

Calculating Indices

Operational Choices

10.113 The matched-model property of (nonhedonic) multilateral indices implies that without any manual intervention, the results depend on the choice of item identifier. For example, when using the barcode as item identifier, the price change of a homogeneous item whose barcode changes at the same time—a “relaunch”—will not be measured. As mentioned earlier, the use of SKU codes mitigates the problem since SKU generally consists of multiple barcodes for similar items and is more stable than barcode. Nevertheless, even SKU may be too detailed.

10.114 If a relatively small number of observable attributes with discrete values suffice to define homogeneous items, items could be defined by cross-classifying the sets of categorical variables for each attribute and prices calculated as unit values across all the barcodes/SKUs. Most likely there will still be new and disappearing items (cells) across the sample period. To maximize the degree of matching without introducing chain drift, a multilateral method could be applied (Chessa 2016). A potential issue is that the available characteristics information may be limited, especially when the characteristics are extracted from variety descriptions in scanner data, which are often rather broad. In this case, unit value bias is likely to arise. Also, if the characteristics information is deemed sufficient, it may be better to construct hedonic indices.

10.115 While taking a (cutoff) sample that ignores items, however defined, with small expenditure shares would in many cases not significantly affect the results, it is not necessary when using a weighted multilateral method. Most of the issues discussed earlier in this chapter, such as the choice of calculating unit values at the store or at the chain level, and the need to have an index structure that facilitates the use of scanner data, apply here as well.

Producing Empirical Results

10.116 The purpose of producing empirical results of multilateral methods is twofold: examining the performance of various methods in local contexts as well as demonstrating to CPI users the likely impacts of moving from current CPI data sources and methods to new approaches. Ideally, these multilateral methods should be examined against each other, as well as in comparison to the official CPI. These comparisons should be undertaken at the lowest level of the published CPI and at various aggregation levels, including the all-items CPI. For some empirical evidence on multilateral price index methods, see Chapter 5 of ABS (2016a), Chapter 3 of ABS (2017), and Chessa and others (2017).

10.117 Several insights can be obtained from producing empirical results. Often these insights further reinforce the theoretical arguments for utilizing multilateral index methods to compile the CPI. This may include the impact of using contemporaneous information for weighting purposes that capture consumer behavior, including substitution, over time. The empirical results should be communicated to CPI users and stakeholders.

Communicating with Users and Stakeholders

10.118 The use of scanner data to compile the CPI potentially represents quite a significant change to the data sources and methods employed by NSOs. These changes need to be carefully communicated to CPI users and stakeholders. A suggested set of activities includes:

  • Publishing information papers that outline the proposed new methods and data sources

  • Conducting face-to-face meetings with key stakeholders (for example, central banks, treasury, and finance ministries) and other interested parties, including members of the public

  • Using media releases and briefing of economic journalists to help inform the public of proposed changes

  • Encouraging stakeholders and the public to provide submissions to the NSO for consideration

  • Engaging with leading academics both to review the proposed changes and to encourage their support

10.119 Following this consultation, which could take a couple of years, the NSO should publish a position paper that both responds to the topics raised as part of the consultation process and articulates how the NSO will proceed with the use of scanner data to compile the CPI, including the rationale and empirical results that support this approach. The position paper should clearly state the data sources and methods to be employed and provide a timetable for the implementation of changes.

Publication and Dissemination

10.120 Following the publication of the position paper, it is suggested the NSO compile the CPI using both current and new data sources and methods in parallel for a period of approximately six months. This transition period allows the NSO to refine processes and procedures to compile the CPI using the new methods, as well as to compare the empirical results of the two approaches. This transition period is often the first opportunity for the NSO to use the new data sources and methods in real time following the CPI processing and publication timetable. It is at the NSO’s discretion whether the results of the parallel processing are made public.

10.121 The first period for which the new data sources and methods are used to compile the CPI should be announced well in advance and should include detailed metadata for the media and other key data users. This will ensure that the methodological changes implemented to the CPI are well understood. Further information relating to the CPI dissemination can be found in Chapter 14 of this Manual.

1

The elementary aggregate refers to the lowest level for which expenditure weights are available. Because scanner data contains detailed and timely quantity information, the elementary price indices are stratum indices that are aggregated to obtain the higher-level indices in the CPI classification structure.

2

Implicit weights result when samples are selected using probabilities proportional to size sampling methods.

3

A similar problem can also be encountered with web-scraped data (see Chapter 5, Annex 5.6).

Author: Brian Graf
  • Abe, N., and K. Shinozaki. 2018. “Compilation of Experimental Price Indexes Using Big Data and Machine Learning: A Comparative Analysis and Validity Verification.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 7–9. https://www.unece.org.

    • Search Google Scholar
    • Export Citation
  • Abraham, K. G., J. S. Greenlees, and B. R. Moulton. 1998. “Working to Improve the Consumer Price Index.” Journal of Economic Perspectives 12 (1): 2736.

    • Search Google Scholar
    • Export Citation
  • Aczél, J. 1987. A Short Course on Functional Equations. Dordrecht: Reidel Publishing Co.

  • Advisory Commission to Study the Consumer Price Index. 1995. Toward a More Accurate Measure of the Cost of Living. Interim Report to the Senate Finance Committee, September 15. Washington, DC.

    • Search Google Scholar
    • Export Citation
  • Aizcorbe, A. M., and P. C. Jackman. 1993. “The Commodity Substitution Effect in CPI Data, 1982–91.” Monthly Labor Review 116 (12): 2533.

    • Search Google Scholar
    • Export Citation
  • Aizcorbe, A., C. Corrado, and M. Doms. 2001. Constructing Price and Quantity Indexes for High Technology Goods. Industrial Output Section, Division of Research and Statistics. Washington, DC: Board of Governors of the Federal Reserve System.

    • Search Google Scholar
    • Export Citation
  • Aizcorbe, A., C. Corrado, and M. Doms. 2003. “When Do Matched-Model and Hedonic Techniques Yield Similar Price Measures?Working Paper 2003–14, Federal Reserve Bank of San Francisco, San Francisco, CA.

    • Search Google Scholar
    • Export Citation
  • Alterman, W. F., W. E. Diewert, and R. C. Feenstra. 1999. International Trade Price Indexes and Seasonal Commodities. Washington, DC: Bureau of Labor Statistics.

    • Search Google Scholar
    • Export Citation
  • Anderson, R. G., B. E. Jones, and T. Nesmith. 1997. “Building New Monetary Services Indexes: Concepts, Data and Methods.” Federal Reserve Bank of St. Louis Review 79 (1): 5383.

    • Search Google Scholar
    • Export Citation
  • Arguea, N. M., C. Haseo, and G. A. Taylor. 1994. “Estimating Consumer Preferences Using Market Data: An Application to U.S. Automobile Demand.” Journal of Applied Econometrics 9: 118.

    • Search Google Scholar
    • Export Citation
  • Armknecht, P. A. 1996. “Improving the Efficiency of the U.S. CPI.” Working Paper No. 96/103, International Monetary Fund, Washington, DC.

    • Search Google Scholar
    • Export Citation
  • Armknecht, P. A. 2016. “Fixed Basket Methods for Compiling Consumer Price Indexes.” Meeting of the Group of Experts on Consumer Price Indices, Geneva, Switzerland, May 2–4, 2016. http://www.unece.org/index.php.

    • Search Google Scholar
    • Export Citation
  • Armknecht, P. A., and D. Weyback. 1989. “Adjustments for Quality Change in the U.S. Consumer Price Index.” Journal of Official Statistics 5 (2): 10723.

    • Search Google Scholar
    • Export Citation
  • Armknecht, P. A., and F. Maitland-Smith. 1999. “Price Imputation and Other Techniques for Dealing with Missing Observations, Seasonality and Quality Change in Price Indices.” Working Paper No. 99/78, International Monetary Fund, Washington, DC. http://0-www-imf-org.library.svsu.edu/external/pubs/ft/wp/1999/wp9978.pdf

    • Search Google Scholar
    • Export Citation
  • Armknecht, P. A., and M. Silver. 2014. “Post-Laspeyres: The Case for a New Formula for Compiling Consumer Price Indexes.” The Review of Income and Wealth 60 (2): 22544.

    • Search Google Scholar
    • Export Citation
  • Armknecht, P. A., W. F. Lane, and K. J. Stewart. (1997) 2003. “New Products and the U.S. Consumer Price Index.” In Scanner Data and Price Indexes, NBER Studies in Income and Wealth, edited by R. C. Feenstra and M. D. Shapiro. Chicago, IL: University of Chicago Press, pp. 37591.

    • Search Google Scholar
    • Export Citation
  • Arrow, K. J., H. B. Chenery, B. S. Minhas, and R. M. Solow. 1961. “Capital–Labor Substitution and Economic Efficiency.” Review of Economics and Statistics 63: 22550.

    • Search Google Scholar
    • Export Citation
  • Astin, J. 1999. “The European Union Harmonized Indices of Consumer Prices (HICP).” in Proceedings of the Ottawa Group Fifth Meeting, Reykjavik, Iceland, August 25–27, edited by R. Gudnason and T. Gylfadottir. Reykjavík: Statistics Iceland; also published in Statistical Journal of the United Nations ECE16: 12335. http://www.statcan.ca/secure/english/ottawagroup/.

    • Search Google Scholar
    • Export Citation
  • Astin, J. 2001. “New Uses of PPPs within the European Union.” Paper presented at the Joint World Bank–OECD Seminar on Purchasing Power Parities: Recent Advances in Methods and Applications, Washington, DC, January 30–February 2.

    • Search Google Scholar
    • Export Citation
  • Aten, B., R. Summers, and A. Heston. 2001. “An Explanation of Stability in Country Price Structures: Implications for Spatial– Temporal Comparisons.” Paper presented at the Joint World Bank–OECD Seminar on Purchasing Power Parities: Recent Advances in Methods and Applications, Washington, DC, January 30–February 2.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 1997. “An Analytical Framework for Price Indexes in Australia.” Information Paper, Catalogue No. 6421.0. http://www.abs.gov.au.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 2003. “Australian Consumer Price Index: Concepts, Sources and Methods.” Catalogue No. 6461.0. http://www.abs.gov.au/.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 2009. “Consumer Price Index: Concepts, Sources and Methods.” Catalogue No. 6461.0, ABS, Canberra. http://www.abs.gov.au/ausstats/abs@.nsf/mf/6461.0.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 2013. “Feature Article: The Use of Transactions Data to Compile the Australian Consumer Price Index.” Catalogue No. 6401.0, ABS, Canberra. http://www.abs.gov.au/.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 2015. “Enhancing the Australian CPI: A Roadmap.” Catalogue No. 6401.0.60.001, ABS, Canberra. http://www.abs.gov.au/.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 2016a. “Making Greater Use of Transactions Data to Compile the Consumer Price Index.” Information Paper 6401.0.60.003, ABS, Canberra. http://www.abs.gov.au/.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 2016b. “Measuring Price Change of Attached Dwellings in the CPI.” Feature Article, ABS, Canberra. http://www.abs.gov.au/.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 2017a. “An Implementation Plan to Maximise the Use of Transactions Data in the CPI.” Information Paper 6401.0.60.004, ABS, Canberra. http://www.abs.gov.au/.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 2017b. “Introduction of the 17th Series Australian Consumer Price Index.” Information Paper 6470.0.55.001, ABS, Canberra. http://www.abs.gov.au/AUS-STATS/abs@.nsf/DetailsPage/6470.0.55.0012017.

    • Search Google Scholar
    • Export Citation
  • Bäckström, Peter, and Muhanad Sammar. 2012. “The Use of Superlative Index Links in the Swedish CPI.” Paper presented at the meeting of the Group of Experts on CPI, Geneva. https://www.unece.org/index.php.

    • Search Google Scholar
    • Export Citation
  • Bailey, M. J., R. F. Muth, and H. O. Nourse. 1963. “A Regression Method for Real Estate Price Construction.” Journal of the American Statistical Association 58: 93342.

    • Search Google Scholar
    • Export Citation
  • Baker, D. 1998. “Does the CPI Overstate Inflation? An Analysis of the Boskin Commission Report.” In Getting Prices Right, edited by D. Baker. Washington, DC: Economic Policy Institute, pp. 79155.

    • Search Google Scholar
    • Export Citation
  • Baldwin, A. 1990. “Seasonal Baskets in Consumer Price Indexes.” Journal of Official Statistics 6 (3): 25173.

  • Baldwin, A., A. Nakamura, and M. Prudhomme. 2010. “An Empirical Analysis of the Different Concepts for Owned Accommodation in the Canadian CPI: The Case of Ottawa, 1996–2005.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 10–12. https://www.unece.org.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 1980a. “Seasonal Products in Agriculture and Horticulture and Methods for Computing Price Indices.” Statistical Studies No. 24. The Hague: Netherlands Central Bureau of Statistics.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 1980b. “Seasonal Commodities and the Construction of Annual and Monthly Price Indexes.” Statistische Hefte 21: 1106.

  • Balk, B. M. 1980c. “A Method for Constructing Price Indices for Seasonal Commodities.” The Journal of the Royal Statistical Society Series A 143: 6875.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 1981. “A Simple Method for Constructing Price Indices for Seasonal Commodities.” Statistische Hefte 22: 728.

  • Balk, B. M. 1983. “Does There Exist a Relation between Inflation and Relative Price Change Variability? The Effect of the Aggregation Level.” Economic Letters 13: 173180.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 1985. “A Simple Characterization of Fisher’s Price Index.” Statistische Hefte 26: 5963.

  • Balk, B. M. 1989a. “Changing Consumer Preferences and the Cost of Living Index: Theory and Nonparametric Expressions.” Zeitschrift für Nationalökonomie 50 (2): 15769.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 1989b. “On Calculating the Precision of Consumer Price Indices.” Contributed Papers 47th Session of the ISI. Paris: International Statistical Institute.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 1990. “On Calculating Cost-of-Living Index Numbers for Arbitrary Income Levels.” Econometrica 58 (1): 7592.

  • Balk, B. M. 1994. “On the First Step in the Calculation of a Consumer Price Index.” Paper presented at First Meeting of the International Working Group on Price Indices, Ottawa, October 31–November 4. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 1995. “Axiomatic Price Index Theory: A Survey.” International Statistical Review 63: 6993.

  • Balk, B. M. 1996a. “A Comparison of Then Methods for Multilateral International Price and Volume Comparisons.” Journal of Official Statistics 12: 199222.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 1996b. “Consistency in Aggregation and Stuvel Indices.” The Review of Income and Wealth 42: 35363.

  • Balk, B. M. 1998a. Industrial Price, Quantity and Productivity Indices. Boston, MA: Kluwer Academic Publishers.

  • Balk, B. M. 1998b. “On the Use of Unit Value Indices as Consumer Price Subindices.” Paper presented at the Fourth Meeting of the International Working Group on Price Indices, Washington, DC, April 22–24. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 2000a. “Divisia Price and Quantity Indexes 75 Years After.” Draft Paper, Department of Statistical Methods, Statistics Netherlands, Voorburg.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 2000b. “On Curing the CPI’s Substitution and New Goods Bias.” Research Paper 0005, Department of Statistical Methods, Statistics Netherlands, Voorburg.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 2001. Aggregation Methods in International Comparisons: What Have We Learned? Report Series Research in Management ERS-2001–41-MKT, Erasmus Research Institute of Management. Rotterdam: Erasmus University.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 2005. “Price Indexes for Elementary Aggregates: The Sampling Approach.” Journal of Official Statistics 21: 67599.

  • Balk, B. M. 2008. Price and Quantity Index Numbers. Cambridge: University Press.

  • Balk, B. M. 2008. Price and Quantity Index Numbers: Models for Measuring Aggregate Change and Difference. New York: Cambridge University Press.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 2010. “Lowe and Cobb-Douglas Consumer Price Indexes and Their Substitution Bias.” Jahrbücher für Nation-alökonomie und Statistik (The Journal of Economics and Statistics) 230 (6): 72640.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M., and H. M. P. Kersten. 1986. “On the Precision of Consumer Price Indices Caused by the Sampling Variability of Budget Surveys.” Journal of Economic and Social Measurement 14: 1935.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M., and W. E. Diewert. 2001. “A Characterization of the Törnqvist Price Index.” Economics Letters 73: 27981.

  • Ball, A., and D. Fenwick. 2004. “Costing Mobile Telephone Calls: The Use of Constrained User Profiles.” Paper presented at the Eighth Meeting of the International Working Group on Price Indices, Helsinki, August 23–25. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Bartik, T. J. 1988. “Measuring the Benefits of Land Improvements in Hedonic Models.” Land Economics 64 (2): 17283.

  • Bas, C. 2018. “Different Approaches and Methods for Strong Seasonal Items in the CPI.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 7–9. https://www.unece.org.

    • Search Google Scholar
    • Export Citation
  • Bascher, J., and T. Lacroix. 1999. “Dishwashers and PCs in the French CPI: Hedonic Modeling, from Design to Practice.” Paper presented at the Fifth Meeting of the International Working Group on Price Indices, Reykjavik, August 25–27. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Baxter, M. (ed.). 1998. The Retail Prices Index. Technical Manual. London: Office for National Statistics, UK.

  • Bean, L. H., and O. C. Stine. 1924. “Four Types of Index Numbers of Farm Prices.” Journal of the American Statistical Association 19: 305.

    • Search Google Scholar
    • Export Citation
  • Becker, G. S. 1965. “A Theory of the Allocation of Time.” Economic Journal 75: 493517.

  • Beck-MacNeil, R. 2018. “Pricing the Internet—The Internet Access Services Component of the Canadian CPI.” Statistics Canada. Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 7–9. https://www.unece.org.

    • Search Google Scholar
    • Export Citation
  • Beidelman, C. 1973. Valuation of Used Capital Assets. Sarasota, FL: American Accounting Association.

  • Beidelman, C. 1976. “Economic Depreciation in a Capital Goods Industry.” National Tax Journal 29: 37990.

  • Beisteiner, A. 2008. “Optimal Allocation of Prices in Practice— Using the Neumann Formula as Tool.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 8–9. https://www.unece.org.

    • Search Google Scholar
    • Export Citation
  • Bentley, A., and F. Krsinich. 2017. “Towards a Big Data CPI for New Zealand.” Paper presented at the Fifteen Meeting of the International Working Group on Price Indices, Eltville am Rhein, May 10–12. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Bernardini, A., C. De Vitiis, A. Guandalini, F. Inglese, and M. D. Terribili. 2016. “Measuring Inflation Through Different Sampling Designs Implemented on Scanner Data.” Paper presented at Meeting of the Group of Experts on Consumer Price Indices, Geneva, Switzerland, May 2–4.

    • Search Google Scholar
    • Export Citation
  • Berndt, E. R. 1991. The Practice of Econometrics: Classic and Contemporary. Reading, MA: Addison-Wesley.

  • Berndt, E. R., D. Ling, and M. K. Kyle. 2003. “The Long Shadow of Patent Expiration: Generic Entry and Rx to OTC Switches.” In Scanner Data and Price Indexes, NBER Studies in Income and Wealth, edited by M. Shapiro and R. C. Feenstra. Chicago, IL: University of Chicago Press, pp. 22973.

    • Search Google Scholar
    • Export Citation
  • Berndt, E. R., L. T. Bui, D. H. Lucking-Reiley, and G. L. Urban. 1997. “The Roles of Marketing, Product Quality and Price Competition in the Growth and Composition of the U.S. Anti-Ulcer Drug Industry.” In The Economics of New Goods, NBER Studies in Income and Wealth, edited by T. Bresnahan and R. J. Gordon. Chicago and London: University of Chicago Press, pp. 23277.

    • Search Google Scholar
    • Export Citation
  • Berndt, E. R., Z. Griliches, and N. J. Rappaport. 1995. “Econometric Estimates of Price Indexes for Personal Computers in the 1990s.” Journal of Econometrics 68: 24368.

    • Search Google Scholar
    • Export Citation
  • Berry, S., J. Levinsohn, and A. Pakes. 1995. “Automobile Prices in Market Equilibrium.” Econometrica 63 (4): 841–90; also published as NBER Working Paper No. W4264, July 1996. http://www.nber.org.

    • Search Google Scholar
    • Export Citation
  • Beuerlein, I. 2001. “The German Consumer Price Index for Telecommunication Services: A User Profile Approach for Mobile Technology and Internet Access.” Paper presented at the Sixth Meeting of the International Working Group on Price Indices, Canberra, April 2–6. http://www.ottawagroup.org/.

    • Search Google Scholar
    • Export Citation
  • Bhardwaj, H., T. Flower, P. Lee, and M. Mayhew. 2017. Research Indices Using Web Scraped Price Data. August 2017 Update, UK Office for National Statistics, London.

    • Search Google Scholar
    • Export Citation
  • Bird, D., R. Breton, C. Payne, and A. Restieaux. 2014. Initial Report on Experiences with Scanner Data in ONS. Report, Office for National Statistics, Newport, UK.

    • Search Google Scholar
    • Export Citation
  • Bode, B., and van J. Dalén. 2001. “Quality-Corrected Price Indexes of New Passenger Cars in the Netherlands, 1990–1999.” Paper presented at the Sixth Meeting of the International Working Group on Price Indices, Canberra, April 2–6. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Boettcher, I. 2015. “Automatic Data Collection on the Internet (Web Scraping).” Statistic Austria. Paper presented at the Fourteen Meeting of the International Working Group on Price Indices, Tokyo, May 20–22. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Böhm-Bawerk, E. V. 1891. The Positive Theory of Capital. Translated from the original German edition of 1888 by W. Smart. New York: G. E. Stechert.

    • Search Google Scholar
    • Export Citation
  • Bortkiewicz, L. V. 1923. “Zweck und Struktur einer Preisin-dexzahl.” Nordisk Statistisk Tidsskrift 2: 369408.

  • Boskin, M. J. (Chair), E. R. Dullberger, R. J. Gordon, Z. Griliches, and D. W. Jorgenson. 1996. Final Report of the Commission to Study the Consumer Price Index, U.S. Senate, Committee on Finance. Washington, DC: U.S. Government Printing Office.

    • Search Google Scholar
    • Export Citation
  • Boskin, M. J. (Chair), E. R. Dullberger, R. J. Gordon, Z. Griliches, and D. W. Jorgenson. 1998. “Consumer Prices in the Consumer Price Index and the Cost of Living.” Journal of Economic Perspectives 12 (1): 326.

    • Search Google Scholar
    • Export Citation
  • Böttcher, I., and S. Sergeev. 2014. “Austrian Scanner Data Project—Report Multipurpose Consumer Price Statistics: The Use of Scanner Data.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 26–8. https://www.unece.org.

    • Search Google Scholar
    • Export Citation
  • Bowley, A. L. 1899. “Wages, Nominal and Real.” In Dictionary of Political Economy, Volume 3, edited by R. H. I. Palgrave. London: Macmillan, pp. 64051.

    • Search Google Scholar
    • Export Citation
  • Bowley, A. L. 1901. Elements of Statistics. Westminster: Orchard House.

  • Bowley, A. L. 1919. “The Measurement of Changes in the Cost of Living.” Journal of the Royal Statistical Society 82: 34361.

  • Bradley, R. 1995. “The Use of Scanner Data as a Means to Reduce the Mean Squared Error in the CPI.” Working paper, U.S. Department of Labor, Bureau of Labor Statistics, Washington, DC.

    • Search Google Scholar
    • Export Citation
  • Bradley, R. 2007. “Analytical Bias Reduction for Small Samples in the U.S. Consumer Price Index.” Journal of Business & Economic Statistics 25: 33746.

    • Search Google Scholar
    • Export Citation
  • Bradley, R., B. Cook, S. E. Leaver, and B. R. Moulton. 1997. “An Overview of Research on Potential Uses of Scanner Data in the U.S. CPI.” Paper presented at the Third Meeting of the International Working Group on Price Indices, Voorburg, April 16–18. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Braithwait, S. D. 1980. “The Substitution Bias of the Laspeyres Price Index: An Analysis Using Estimated Cost-of-Living Indexes.” American Economic Review 70 (1): 6477.

    • Search Google Scholar
    • Export Citation
  • Brand, M., C. Vermeulen, D. Fischbach, and Y. Carpy. 2017. “Swiss Residential Property Price Index: The Use of Geolocal-ised Information for Quality Adjustment in Location.” Paper presented at the Fifteenth Meeting of the International Working Group on Price Indices, Eltville, May 10–12. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Bresnahan, T. F. 1997. “Comment.” In The Economics of New Goods, NBER Studies in Income and Wealth, edited by T. F. Bresnahan and R. J. Gordon. Chicago, IL: University of Chicago Press, pp. 23747.

    • Search Google Scholar
    • Export Citation
  • Breton, R., G. Clews, L. Metcalfe, N. Milliken, C. Payne, J. Winton, and A. Woods. 2016. “Research Indices Using Web Scraped Data.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 2–4. https://www.unece.org.

    • Search Google Scholar
    • Export Citation
  • Brown, C., and A. Stockburger. 2006. “Item Replacement and Quality Change in Apparel Price Indexes.” Monthly Labor Review (December): 3545. https://www.bls.gov/opub/mlr/2006/12/art3full.pdf.

    • Search Google Scholar
    • Export Citation
  • Burnett-Isaacs, K. 2015. “Using Hedonics to Create Land and Structure Indexes for the Ottawa Condominium Market.” Paper presented at the Fourteen Meeting of the International Working Group on Price Indices, Tokyo, May 20–22. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Cage, R., J. S. Greenlees, and P. Jackman. 2003. “Introducing the Chained Consumer Price Index.” In International Working Group on Price Indices (Ottawa Group): Proceedings of the Seventh Meeting, edited by Thierry Lacroix. Paris: INSEE, pp. 21346.

    • Search Google Scholar
    • Export Citation
  • Canning, J. B. 1929. The Economics of Accountancy. New York: The Ronald Press Co.

  • Carli, G.-R. 1804. “Del valore e della proporzione dei metallic monetati.” In Scrittori classici italiani di economia politica, Vol. 13. Milano: G. G. Destefanis, pp. 297366; originally published in 1764.

    • Search Google Scholar
    • Export Citation
  • Carruthers, A. G., D. J. Sellwood, and P. W. Ward. 1980. “Recent Developments in the Retail Prices Index.” The Statistician 29: 132.

    • Search Google Scholar
    • Export Citation
  • Cassel, E., and R. Mendelsohn. 1985. “On the Choice of Functional Forms for Hedonic Price Equations: Comment.” Journal of Urban Economics 18 (September): 13542.

    • Search Google Scholar
    • Export Citation
  • Castles, I. 1997. The OECD–EUROSTAT PPP Program: Review of Practice and Procedures. Paris: OECD.

  • Cavallo, A. 2016. “Are Online and Offine Prices Similar? Evidence from Large Multi-Channel Retailers.” Massachusetts Institute of Technology. Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 2–4. https://www.unece.org.

    • Search Google Scholar