Abstract

1.1 Chapter 1 provides a self-contained overview of the uses and the basic steps for compiling the consumer price index (CPI). More than just a summary of the chapters to follow, Chapter 1 guides the reader through the compilation process and highlights best practices that are explained in greater detail in subsequent chapters. The flow of the chapter follows the different steps needed to develop and maintain a CPI program that better reflects the standards and best practices set out in the Manual.

Introduction

1.1 Chapter 1 provides a self-contained overview of the uses and the basic steps for compiling the consumer price index (CPI). More than just a summary of the chapters to follow, Chapter 1 guides the reader through the compilation process and highlights best practices that are explained in greater detail in subsequent chapters. The flow of the chapter follows the different steps needed to develop and maintain a CPI program that better reflects the standards and best practices set out in the Manual.

1.2 Both compilers and data users will benefit from reading Chapter 1. This chapter provides a primer on the methods used to compile the CPI and explains why one method is preferred over another. The chapter aims to provide a clear, easily understood summary of best practices and compilation methods without being overly technical.

Developing the Consumer Price Index

1.3 CPIs measure changes over time in the general level of prices of goods and services that households acquire (use or pay for) for the purpose of consumption. In many countries, they were originally introduced to provide a measure of the changes in the living costs faced by workers, so that wage increases could be related to changing levels of prices. However, over the years, CPIs have widened their scope and now are widely used as a macroeconomic indicator of inflation, as a tool by governments and central banks for monetary policy and for monitoring price stability, and as deflators in the national accounts. With the globalization of trade and production and the liberalization of the markets, national governments, central banks, and international organizations place great importance on the quality and accuracy of national CPIs, and their international comparability.

1.4 Different conceptual frameworks can be used to address fundamental issues relating to the nature of the index. For example, different concepts are used if the CPI intends to measure the change in the cost of a fixed-weight basket of goods and services or whether the target is to measure the change in the cost of living, that is, the cost of maintaining a given standard of living, taking into account the fact that when prices change consumers change their expenditure patterns. The use and conceptual basis of the index will determine the method of construction.

1.5 The method of construction also allows (or should allow) CPIs to be adapted for a wide range of specific uses. For example, they can be adapted to calculate specific inflation rates for social groups such as pensioners or low-income households. Their product coverage can be adapted to show what the rate of inflation is in particular sectors, such as energy or food, or excluding particular products, such as alcohol and tobacco. They can shed light on the effect of tax changes or government-regulated price changes on the rate of inflation. They can be compiled on a regional basis, showing different inflation rates within different parts of a country or between urban and rural areas.

1.6 CPIs are now considered as one of the most important economic and social indicators produced by national statistical offices (NSOs)1 throughout the world. Against this background, the challenge of NSOs is fourfold: to identify user needs; to conceptualize user needs with regard to economic concepts; to translate the underlying concept into statistical measurement terms following the fundamental principles of price index measurement; and to construct the indices so defined and evaluate them against purpose.

Overview of the Consumer Price Index Uses and Needs

1.7 A CPI can be used for a variety of purposes, the most common ones being: the indexation of wages, rents, contracts, and social security payments; the deflation of household final consumption expenditure in the national accounts; and the use as a general macroeconomic indicator, especially for inflation targeting and for setting interest rates. Elements of a CPI are also often used in the calculation of purchasing power parities and extrapolating purchasing power parities between benchmark years as required in the International Comparison Program (ICP).2

1.8 Given the many uses of CPIs, it is unlikely that one index can perform equally well in all applications. Some countries therefore construct several CPI variants for specific purposes. Each index should be properly defined and named to avoid confusion, and a “headline” CPI measure should be explicitly identified. Where only one CPI is published, it is the main use that should determine its type and scope. If there are several major uses, compromises may have to be made with regard to how it is constructed. The purpose of a CPI should guide all aspects of its construction. CPI producers need to know how their index is being used to ensure that it fulfills its purpose. In this connection, user consultation is important.

1.9 This section reviews the various uses of a CPI before examining some of the issues confronted by the index compiler relating to the scope of the index and the practical measurement and compilation decisions which must be made.

The Different Uses of a Consumer Price Index

1.10 CPIs have three main uses:

  • Indexation

    A CPI may be used for wage or contract indexation of any specific group, whether of a population acquiring products, or of a subset of products themselves. In either case, it should represent the coverage of the group concerned. For instance, it can be argued that the weights of a CPI used for indexation of pensions should cover only the expenditure of the pensioner population. The product and outlet list could also be more appropriately targeted, if the data exist. This means, for example, that a CPI used for indexing pensions may use weights relating to pensioner households and may exclude products which may be thought largely irrelevant to this group of households such as educational items. Similarly, for domestic indexation, the CPI should cover only expenditure of the resident population (see Section on Geographical Coverage and Chapter 2 for more information). More generally, it must be decided whether the CPI should be, in principle, a cost of living index (COLI) or a cost of goods index—these two very different concepts are discussed in the following text.

    For certain specific types of indexation, such as for rents, users may prefer to use just the subindex for rents. In such cases, the subindex should be of a statistical quality sufficient for that purpose.

  • National accounts deflation

    This use requires consistency between the price data used for the CPI and the expenditure data used in the national accounts. Both data sets should cover the same set of goods and services and use the same concepts and same classification, in principle the Classification of Individual Consumption According to Purpose (COICOP). For example, the national accounts require the valuation of goods produced for own consumption, whereas this is sometimes excluded from the CPI either as a matter of principle or for pragmatic reasons. This applies mainly to the valuation of the services of owner-occupied housing and the consumption of own-produced food.

  • Inflation measurement

    It can be argued that central banks ideally need a timely index relating to total inflation, not just consumer inflation. But NSOs generally are unable to construct such indices, in part because of the measurement issues relating to government consumption. In the absence of such an index, most central banks rely on a CPI, using the domestic concept (as described in the next section and in Chapter 2), but measured on as wide a basis as possible, with regard to both products and geographical coverage. The same applies to the use of the CPI as a general macroeconomic indicator.

Overview of the Consumer Price Index Concepts

Type of Index Number Formula

1.11 Experts generally agree that the ideal type of index for a CPI would be a “superlative” index such as the Fisher index, which will be discussed in the following text and in Chapter 8. Superlative indices make equal use of the prices and quantities (that is, the expenditure weights) in both periods being compared (the reference period and the current period). Current period expenditure weights are usually not known, so in practice nearly all CPIs rely on weights relating to a weight reference period some time earlier. An exception to this are actual transactions that can be captured at the points of purchase through the use of scanner data (as discussed in Chapter 10).

1.12 Some countries aim to produce a COLI. But such an index is in fact a type of superlative index and suffers from the same practical defect as mentioned previously, and it is not possible to compile in real time.

1.13 Many countries state in their published metadata that they use a Laspeyres index or a “Laspeyres-type” index for their national CPI which, in practice, is not the case. It is important, nevertheless, for NSOs to state publicly what type of index is being calculated in their CPI. A true Laspeyres index uses quantity data which relate to exactly the same period as the price reference period. In practice, however, this is difficult to obtain and rarely the case. Most NSOs have a price reference period which is later than the period to which the quantity data or weights relate. Also, the weights usually will span, say, a year rather than a month (or quarter). This is because one of the main sources of weights data is a household budget survey (HBS), as discussed in Chapter 3, which, ideally, run for 12 consecutive months. The HBS generally produces usable results a year or more after the end of a survey period.

Index Formula at Lower (Elementary Aggregate) Level

1.14 The first stage in the calculation of CPIs is the calculation of elementary price indices, which are then aggregated to obtain higher-level price indices. Expenditure weights are not usually available below the elementary aggregate level. The three most widely known elementary index formulas are the Carli, the Dutot, and the Jevons. These are all based on unweighted averages of prices or price relatives, and each is associated with a number of assumptions which will have an impact on measured inflation. The Carli (a simple arithmetic average of price relatives) and Dutot (the ratio of simple arithmetic averages of prices) formulas have a number of problems associated with their use— particularly the chained Carli, which is discouraged as it is particularly associated with having a known, and potentially substantial, upward bias. The Jevons formula (the ratio of simple geometric averages or the geometric average of price relatives) is increasingly used as it avoids many of the problems associated with the arithmetic versions. It should be noted that an arithmetic average is always greater than or equal to a geometric average and that the difference will be greater, the greater the variance in the price relatives. The choice of formula becomes more important the greater the diversity of price movements which is one argument for ensuring that elementary aggregates are as homogeneous as possible. This topic is discussed in more detail in Chapter 8.

Index Formula at Higher Level

1.15 The higher-level indices are calculated simply as weighted averages of the elementary price indices. The weights typically remain fixed for a sequence of at least 12 months. Some NSOs revise the CPI weights at the beginning of each year to try to approximate as closely as possible the current consumption patterns, but many countries continue to use the same weights for several years. At a minimum, weights should be updated every five years. The use of fixed weights has a considerable practical advantage that the index can make repeated use of the same weights. This saves time and resources. Revising the weights can be both time-consuming and costly if it requires a new HBS to be carried out. However, the longer the period between weight updates, the less relevant and representative the CPI becomes. Many NSOs are moving toward annual or biannual weight updates.

1.16 In Chapters 24 of Consumer Price Index Theory, the superlative indices Walsh, Fisher, and Törnqvist show up as being “best” in all the approaches to index number theory. These three indices give very similar results so that for any practical reason it will not make any difference which one is chosen as the preferred theoretical target index because they most closely approximate a COLI. The theoretical target index is a matter of choice and affects the choice of formula used to calculate the index. In practice, an NSO may prefer to designate a basket index that uses the actual basket in the earlier of the two periods as its target index on grounds of simplicity and practicality. As noted previously, it is not possible to compile a superlative index in real time. In other words, the Laspeyres index may be the theoretical target index because NSOs produce a CPI that lies between a COLI and a cost of goods index.

Acquisition, Use, or Payment Approach

1.17 A CPI is based on the measurement of the change in prices of the goods and services included in the basket. The vast majority of goods (but not necessarily of total values) are priced in the retail outlets selling them. It should be noted that most often the prices recorded are the labeled prices, which are assumed to be the prices actually paid by consumers. It is also generally assumed that payment for the goods is made at the time of purchase—indeed the consumer would regard the two events as identical. However, payment can be in cash or on credit, including credit cards for which the due date of payment may be several weeks after the actual purchase.

1.18 The time factor is important also in other ways. A consumer may decide to buy a larger than normal quantity of a particular good if there is a special price reduction. The product may then be stored at home and “consumed” (that is, used) over a relatively long period. Cans of food, for example, offered cheaply for a limited period, may be stored at home without deterioration for months, and consumed at the usual frequency.

1.19 Another issue concerns the definition of “usage.” A bottle of milk will typically be consumed within a few days of purchase. Consumption is likely to take place in the month for which the CPI is calculated. But a semidurable good such as a shirt will be worn many times over a period of probably several years. Durable goods such as televisions and refrigerators may be used for a decade or more. The question arises as to which CPI month (or months) should the purchase be allocated.

1.20 With services, these questions can be even more complex. Take, for example, the purchase of a season ticket for a bus service. This may be a single payment for a pass which gives (“free”) unlimited bus transport for a year. It can be seen that although this example is clearly a service (the use of bus transport over a period of time), it has much in common with the purchase of a durable or semidurable good such as a television or shirt which provides a type of service over a long period. A service such as a medical operation can also be regarded as durable, since it is likely to give long-term health benefits to the patient.

1.21 CPI theory devotes much thought to these issues, which can have important implications not only for how a CPI is compiled but also for the results. Three different approaches can be identified:

  • The acquisitions approach relates to when the good or service is acquired, irrespective of when it is actually used or consumed. The time of acquisition of a good is the moment at which the legal ownership of the good passes to the consumer. This is usually the point at which the purchaser incurs a liability to pay. On the other hand, with a service there is no change in ownership; it is “acquired” at the time the producer provides it (for example, a single bus journey or airline fights). A CPI based on this approach measures the change in the cost of acquiring a product. The timing of the recorded prices should be consistent with the way in which the value would be recorded in the expenditure data used for the CPI weights.

  • The use approach relates to the period over which the product is consumed or used. A CPI based on this approach measures the change in the cost of using the product over time; in other words, the cost of the good is distributed over its useful life. Expenditures on durable goods and services are liable to fluctuate depending on the expected duration of their useful life.

  • The payments approach relates to the period of time when the actual period-to-period payments for the product are made. This can differ from the period when it is acquired and when it is used. When payments are not made in cash, there may be a long period before the purchase is paid for, whether by credit card or other methods. The time at which these debits are made is irrelevant for the recording of the price. The price to be recorded is the price payable at the time of acquisition (though sometimes the method of payment may itself affect the price).

1.22 NSOs need to have a clear policy on which of these approaches is used in their CPI. In practice, the choice between the three approaches is an issue relating to durable goods and its impact is likely to affect the weight given to owner-occupier housing costs. Each of these approaches is discussed in Chapter 10 of Consumer Price Index Theory. In countries where food expenditures and other expenditures on nondurables, semidurables, and services account for a significant share of the CPI basket and where credit financing is rarely used, the acquisition, use, and payment approaches will give very similar results and hence the CPI can satisfy many uses equally well. This is the principal reason why most countries use, either implicitly or explicitly, the acquisitions approach to define what constitutes consumption expenditure.

Geographical Coverage

1.23 There are two distinct aspects to the question of the geographical coverage of a CPI. The first relates to the country as a whole (domestic versus national coverage), the second to its regions.

The National versus Domestic Concept

1.24 A CPI can have “national” or “domestic” coverage.

  • National coverage means that the CPI should cover the expenditure made by resident households (at purchasers’ prices)3 of the country, regardless of where the expenditure takes place. The national concept is appropriate when the CPI is being used for indexation of incomes and cost of living measures. The weights for expenditure abroad can be included in the HBS, but measuring prices paid abroad poses problems. The national concept thus poses a measurement problem for collecting prices abroad.

  • Domestic coverage means that the CPI would cover all the expenditure made within the economic territory of the country, including the household final consumption expenditure made by foreign visitors. It is appropriate where the CPI is used for national inflation analysis and monetary policy. Many countries carry out surveys of the expenditures of foreign visitors, for example, via International Passenger Surveys conducted at major border crossings and airports by NSO staff. This is particularly important for those countries which have a large number of foreign tourists, or a high level of cross-border shoppers. Foreign visitors will generally have very different expenditure patterns from those of resident households (for example, they will spend more on hotels and restaurants) and to omit them could introduce serious distortions into a CPI aiming to follow the domestic concept, especially if the main purpose of the index is to measure the inflation trends in the economy.

1.25 For the national concept, internet purchases from foreign websites or websites of retailers based abroad should be included in the CPI. So should purchases made abroad more generally, including such items as fees for foreign boarding schools, even if the item, in this case education, is consumed outside the country. Where such purchases are made in the foreign currency, they should be converted to the domestic currency at the relevant exchange rate. Clearly, it would be impracticable to collect prices directly in foreign countries on a continuous basis, although surveys of prices done in order to compute purchasing power parity may provide an occasional benchmark. Where the regular collection of the relevant prices is not practical, it may be possible to obtain a reasonable proxy for price movements using published subindices of the other countries’ CPIs.

1.26 Under the domestic concept, the treatment of internet purchases requires a broader approach, especially given its growing importance. Most NSOs which have examined the issue have concluded that internet purchases from home whether from domestic or foreign websites, should be included in the CPI. Care must be taken to convert foreign purchases to the national currency as differences in exchange rates will affect the price paid (for more information on this topic, see Chapters 2 and 11). Chapter 2 provides more information on the domestic versus national concepts, while Chapter 3 provides more detail on the application of these concepts with regard to developing weights.

Regional Coverage

1.27 Concerning the regional coverage of the CPI, the general rule is that a national CPI should cover expenditures and prices throughout the country. However, comprehensive coverage is not always necessary, especially if regional CPIs are not published and the sampling program ensures that the index is representative of the whole country. In such situations, CPI compilers should collect evidence from time to time on the trends in prices in different regions over periods which cover differences in seasonal variations, to ensure that the sample remains representative. Any region which shows price trends significantly different from the others should be covered by the CPI if its inclusion is likely to have a significant effect on the national CPI and will improve representativeness. But there is little point in spending scarce resources collecting prices in sparsely populated regions if that would have little or no impact on the national CPI. When carrying out these sensitivity tests, regional weights can often be an issue. In this case, population may sometimes be used as a proxy for regional consumer expenditure. However, where regional CPIs are aggregated to compute the national CPI, weights should be based on regional expenditure rather than population data.

1.28 Another difficulty regarding regional CPIs is related to the national versus domestic concept. It can sometimes be the case that a household lives in one region but does most of its shopping in an adjacent region, particularly when a household lives close to a regional “border.” The question of whether the expenditure weights and the prices should be allocated to the region of expenditure or the region of residence is usually dictated by practical issues. As with the national concept discussed previously, if the region of expenditure is used (equivalent to expenditure abroad by a domestic resident) some method must be found for estimating the proportions of expenditure made by “visiting” consumers in the various regions so that this can be reflected in the price indices.

1.29 Finally, the question often arises as to whether a CPI can be limited to urban areas or if rural areas should also be covered, as in many countries a significant part of population resides in rural areas. Having a rural CPI is important for poverty analysis. Again, in principle, the whole territory should be covered, but clearly, the impact on the national CPI of including rural areas where relatively few monetary transactions take place will often argue against their inclusion on grounds of cost. The view taken will depend, at least in part, on the size and treatment of own-account production. If own-account production is included in the CPI, the weights should include a valuation of the physical quantities of such products, often derived from the HBS. The prices will normally be the same as those used for actual transactions for the same goods sold in the same locality.

1.30 Where the weights derived from an HBS are available for rural as well as urban households it is generally better to use the weights for urban and rural households combined, even if price collection is limited to urban areas, as this will normally improve the representativeness of the index. But, where feasible, price statisticians should undertake pilot calculations to test whether this is the case.

Reference Population for the Consumer Price Index

1.31 The group of households included in the scope of a CPI is referred to as the “reference population.” According to the 2008 SNA, households are made up of private households and institutional households. Private households are defined as groups of persons who share the same living accommodation, who pool some or all of their resources and consume select goods and services collectively. Members of the same household do not necessarily have to belong to the same family, as long as there is some sharing of resources and consumption. Institutional households consist of persons living permanently, or for a very long period of time, in institutions other than private households. These include religious institutions, hospitals, the military, prisons, or retirement homes. Persons who enter such institutions only for short periods of time should be treated as members of the individual households to which they normally belong (the 2008 SNA defines household, private, and institutional household in paragraphs 4.149–4.154). Temporary foreign workers may live together in special housing blocks, which may also be treated as institutional households in the population census.

1.32 Expenditure on accommodation and living costs, such as lodging fees and charges for meals, imposed by the institution, as well as personal expenditure by the individual on, for example, clothes and toiletries, should be included in the CPI. However, care should be taken to ensure there is no double counting where, for instance, a family rather than an individual pays the accommodation costs charged to a patient in a hospital. A data source like the HBS should be designed to pick up the amount spent on such charges just once and the standard convention is to record it against the household which incurs the costs. In the previous example, the accommodation costs should be included under the expenditure of the family, and not the individual. If individuals spend their own money on clothes and other incidental expenditure, then the HBS should record this expenditure as being incurred by the individuals. In practice, many HBSs do not cover institutional households and, where this expenditure is considered to be significant, estimates will need to be made from, for example, special surveys of people living in institutions or by reference to the expenditure patterns of similar people, say, the same sex, age, and socioeconomic group, living in noninstitutional households.

1.33 In considering the practical issues relating to the inclusion of institutional households in a CPI, two questions need to be asked. First, is the expenditure pattern of institutional residents likely to be significantly different from household residents? Second, even if the answer is yes, would their exclusion from the CPI be likely to significantly affect the national (or regional) CPI? To answer these questions, some research should be carried out on a sample basis.

1.34 Some countries exclude certain household types from the CPI, such as the very wealthy or the very poor. Such exclusions may be on theoretical grounds (for example, using the argument that the expenditure of the wealthy, who are relatively few in number, should not be allowed to affect a CPI which may be used for indexation of wages of ordinary workers) or on practical grounds (for example, using the argument that wealthy households tend to have low response rates to HBSs, and their inclusion can lower the quality of expenditure weights). Such exclusions make the CPI inconsistent with the national accounts coverage. For a CPI which is used for indexation of wages, the exclusion of pensioners and wealthy households may be justified on conceptual grounds. For example, it may be considered that such households are likely to spend their money on atypical products and including them would distort the relevant overall average. It is also argued by some that pensioner households should be excluded in principle from an index used for the escalation or indexation of state pensions because of the circularity involved (the level of state pension influences expenditure patterns which are then used in the up-rating calculation) while others would argue that it is logical that indexation should be based on an index reflecting the expenditure of pensioner households and their specific inflation experience. Note that if wealthy households are excluded, the CPI basket should not include products likely to be bought only by the excluded group, nor should outlets specializing in such products be included in the sample. Conversely, if the wealthy are included, some “luxury” products and outlets should also be included in the sample. For the analysis of national inflation, it is considered that the more comprehensive the CPI, the better.

Democratic versus Plutocratic Weights

1.35 A “democratic” CPI uses weights based on the average proportionate share that households in the whole population have spent on the item. Hence, the share of a specific item in the basket is calculated for each household. The weight for the item is the sum of the household shares divided by the number of households. Many households may purchase the item, but also, a number of households will not make such purchases. The average share is thus based on the experience of each individual household, whether they have purchased the item or not. Each household share is equally weighted in deriving the average. The alternative “plutocratic” CPI calculates the item shares as the total expenditure on the item by all households divided by the total expenditures of all items purchased by households. This latter method gives more weight to the high-spending households. It is argued that a democratic index is more suitable for showing the impact of inflation on the average household but is very rarely compiled by NSOs. There is a consensus that a plutocratic index is the appropriate index to use for national accounts deflation and for a general measure of inflation.

1.36 Many countries publish a range of CPIs relating to subsectors of the population such as all households, low-income households, or pensioner households, but a CPI based on democratic weights is very rare.

Product Exclusions

1.37 In its role as an indicator of total consumer inflation, the CPI should in principle cover all types of goods and services which are consumed in the national retail market. In practice, some types of products may be excluded for policy reasons while other exclusions are unavoidable in practice. These may include: goods sold illegally, such as narcotics; black market sales of mobile phones and other goods; gambling; and prostitution. In most of these cases, except perhaps for legal lotteries, there will be no expenditure data from the HBS and prices are difficult if not impossible to collect. Thus, in principle, estimates of weights and prices will need to be made for the purpose of producing deflators, even if the expenditure is not covered in the CPI. Solutions will need to be found to the practical measurement issues. For instance, if a CPI covers gambling, it is not the gross stakes which should be included in the weight, but the net stakes, which is broadly equivalent to the margin taken by the gambling operator. As it is not likely that the net stakes can be measured, one solution may be to distribute the weight for gambling across other subclasses in COICOP class 09.4.6 (recreational and sporting services).

1.38 The treatment of second-hand goods is often found to be problematic. As far as transactions within the household sector are concerned, sales will balance purchases, so the weights will be zero, and they may be excluded from the CPI. But in many countries, there are significant sales of imported second-hand goods from dealers and other third parties, such as cars and clothing. Where sales of imported second-hand goods sold by dealers or other third parties are significant relative to sales of new goods of the same product, such sales should be included, in both weights and prices (as described in Chapter 11).

1.39 CPI compilers sometimes face proposals from governments or pressure groups to exclude certain categories of products for nonstatistical reasons. Common examples are alcohol and tobacco in some countries where their consumption is socially discouraged or they can only be purchased illegally. While it is acceptable to produce a variant of the CPI excluding such products, the all-items CPI should include them, where practical, to ensure that the index presents a true and accurate picture of national inflation.

1.40 Chapter 2 covers in some depth the treatment of other excluded or partly excluded products, including taxes and licenses, subscriptions, insurance, gambling, financial transactions, hire purchase, and interest payments.

Imputed Transactions and Imputed Prices

1.41 A distinction can usefully be made between imputed transactions and actual transactions where a price is imputed. In the category of actual transactions, a prescribed medicine would be provided free as part of a national health service. There is a “transaction” in the sense that a product changes hands but at “zero” price so that it does not constitute a monetary transaction. The conventions for a CPI constructed for the purposes of indexation or the measurement of inflation as a macroeconomic indicator exclude from coverage this category of nonmonetary transaction, so no price should be imputed.

1.42 A CPI should measure the prices of final consumption by a household. In principle, the first category, imputed transactions where households do not incur a liability but bear the costs of acquiring the good or service in another way, should be included in a CPI used for deflation in the national accounts compilation and, in principle, can be included in a CPI compiled for other purposes. Perhaps the most important example is the consumption of own-account production products such as food and owner-occupied housing services. Here, there is no actual transaction at all, and thus no price. If the transaction is to be valued, a price has to be imputed. This would usually be done by reference to actual purchases of the same product in, say, nearby markets. But even when this is done for the purpose of national accounts compilation, it is not necessarily appropriate to include production for own consumption in a general-purpose CPI or in a CPI used for indexation. Indices used as a general measure of inflation or for indexation should be based on a narrow definition of consumption that includes only monetary expenditure. From the point of view of measuring inflation and also for the purposes of income indexation, the most common view is that it is best to omit production for own consumption on pragmatic grounds. Furthermore, goods and services purchased by households which are then used as inputs into own-account production are normally treated as if they themselves were consumption goods and services, and therefore are included in the CPI. Some countries may find it useful to produce two versions of the CPI: one including and the other excluding own-account production. This topic is further discussed in Chapter 11.

Deciding on the Index Coverage and Classification Structure

1.43 Classification is a central theme in the compilation of the CPI. Choosing a classification system is the first step in compiling the CPI because its subaggregates must be defined in such a way that the expenditure weights and prices will relate precisely to the coverage of the subaggregates. The classification is also important because it establishes the framework to define and draw the boundaries for the inclusion of the representative items in the index (and sometimes the outlets). Finally, the classification system helps in defining which level of the hierarchy will be suitable for publication.

1.44 In years past, countries used their own distinct systems for classifying the range of products covered by their CPI. Most countries have now, however, moved to the international standard classification COICOP.

1.45 COICOP was first developed for the 1968 SNA to provide the structure for classifying household final consumption expenditure. The various components of household final consumption expenditure are often used as the basis for the weights in the CPI. The 2003 International Labour Organization (ILO) Resolution on CPIs requires that national CPI classifications should be reconcilable with COICOP at least at its higher aggregation levels. Most countries have adopted COICOP in their economic statistics (for example, in the CPI, national accounts, ICP, and HBSs), with a clear advantage for integration of data sets and enhanced analytical capabilities.

Classification Systems: The General Case

1.46 In its broadest sense a classification is a procedure in which individual items are organized into categories (classes) and subcategories (subclasses) based on information on one or more characteristics inherent to the items. A classification structure will usually have these same items (or elements) arranged in a hierarchically ordered system based on category–subcategory relationships where the subcategory has the same description as the associated category in addition to one or more descriptions. For example, an apple is a subclass of fruit. So any apple is a fruit, but not every fruit is an apple. A product needs to have a more detailed description to be an “apple” and not just a “fruit.”

1.47 In principle, a classification system can be based on any attribute of the objects being classified. Normally, organizing a population of items into categories must leave no two categories with any item in common; in other words, the categories must be mutually exclusive. Also, the categories must collectively include all of the items which are in the population—the categories must be exhaustive. For example, in the case of the CPI, its classification should include the entire universe of goods and services that are covered by the index (for example, fresh food purchased in a store by a consumer is part of the CPI, while heavy machinery such as a tractor is not) and no product should be included in two different categories in the structure.

The Consumer Price Index Classification System

1.48 COICOP, as its name implies, is founded on the principle of “purpose” (see Box 1.1). It is a purpose-type classification because throughout the aggregation program the products are grouped according to the purpose (or function) they usually fulfill such as transport, nourishment, shelter, and so on. Most national CPIs aim at measuring the change of the cost of a basket of goods and services, which is consumed for the purpose of satisfying certain needs. A purpose-based classification would therefore appear to be the logical classification system for a CPI.

1.49 The official COICOP is a five-digit classification. NSOs will expand the COICOP to six and seven digits to obtain more detail for their use. At the higher level of the classification, the products are grouped according to the purpose. Households will select various goods and services in order to satisfy their consumption objectives (that is, renting an apartment for the provision of shelter or eating an apple for the purpose of nourishment). These goods and services are disaggregated further into various groups and may not be based solely on the principle of purpose but also according to product type. For example, oranges and apples are included in the “Fruit” category. The more detailed breakdown is often a product-type classification because these items share a similar production process and are certainly sold at fruit stands or the same location in the supermarket.

Classification of Individual Consumption According to Purpose (COICOP)

The COICOP classification structure has been updated and the new version, COICOP 2018, was endorsed by the United Nations Statistical Commission in March 2018. COICOP 2018 replaces COICOP 1999 as the official standard classification for the CPI.

While a number of countries have begun to implement COICOP 2018, many others will implement the updated classification in the coming years during a future CPI update. Because many countries continue to use COICOP 1999, the Manual presents some examples using COICOP 2018 and others using COICOP 1999.

Chapter 2 summarizes the key changes made to COICOP. The detailed classification structures for both COICOP 2018 and COICOP 1999 are included as appendices to the Manual.

Deriving the Weighting Pattern

1.50 A CPI measures changes in the cost of a representative basket of goods and services. This involves weighting aggregated prices for different categories of goods and services so that each takes an appropriate share to reflect the budgets of the households covered by the index. For instance, if most people spend far more on fresh vegetables than on electricity, then a price rise for fresh vegetables must have more effect on overall price rises than a similar-sized increase for electricity. Therefore, at the lowest level, each elementary aggregate should receive a weight equal to the ratio of expenditure by the covered households on items represented by that aggregate to total expenditure by covered households on all items within the scope of the CPI. Chapter 3 discusses the derivation and sources of the expenditure weights and provides detailed guidance on specific issues.

1.51 The 2003 ILO Resolution on CPIs makes the obvious but important point that the weights follow directly from the scope of the index as well as from the choice between the “acquisition,” “use,” or “payment” approach, as described in paragraph 1.22. It also states that there are two main sources of information: HBSs and national accounts, and that the weights should be reviewed and updated at least every five years. Additionally, new sources of weight information are being developed such as actual expenditures on various types of transactions based on scanner data and other electronic formats. Such sources are being evaluated and exploited for use in development of weights now and in the future.

1.52 The use of expenditure weights is consistent in concept with a CPI based on the acquisition, payment, and user cost approaches although the treatment of major durable goods and housing can present a problem, particularly the costs of owner-occupied housing. The use of weights in a CPI based on total consumption expenditure is often referred to as plutocratic weights because this concept gives more weight to the expenditure patterns of high-spending households (which will also tend to be those with higher incomes). Paragraph 1.36 provides more details on plutocratic weights.

1.53 The goods and services consumed by the households can in principle be acquired in four ways: (1) purchase in monetary transactions; (2) from own production; (3) as payment in kind; and (4) as transfers or gifts from other economic units, including social transfers in kind provided by government and nonprofit institutions serving households.

1.54 The weights are determined by the scope of the CPI and should be derived on the basis of the relevant coverage and types of consumption and with reference to SNA concepts. But it should be noted that there are several delineations of consumption used, as described in Chapter 2. The broadest possible scope for goods and services would cover all four of the previous categories. It would include all social transfers in kind in the form of education, health and housing, and other goods or services provided free of charge or at nominal prices. The total acquisition of goods and services thus described is equivalent to household actual final consumption in the SNA. For the CPI as a general measure of inflation, the more relevant would be to include only goods and services purchased in monetary transactions by the households. Only monetary transactions generate prices that can be observed for the CPI, but this then leaves outstanding the issue of owner-occupier housing services which is considered in detail in Chapter 11.

1.55 It is against this background that a CPI often follows the concept of household final consumption expenditure, as laid down in the 2008 SNA. This approach is often recommended for a CPI being used as a macroeconomic indicator, restricted to the appropriate reference population, or “Index Households,” where the CPI is being used as a compensation index. This compensation index might, for example, exclude the very rich.

1.56 Household final consumption expenditure includes nonmonetary transactions (such as for owner-occupier housing or consumption of own production of food). The concept of “household final monetary consumption expenditure,” used in the European Union Harmonised Index of Consumer Prices defines both the goods and services to be covered, and the price concept to be used, and refers only to monetary transactions. Household final monetary consumption expenditure is a useful concept but many countries prefer to also include some nonmonetary transactions (in particular owner-occupier housing) in their CPI, sometimes using imputed costs thus moving the coverage of the index closer to household final consumption expenditure. Chapter 2 provides more details on the different types of consumption.

1.57 The conceptual issues relating to the construction of weights are discussed in detail in Chapter 3.

Weighting Structure

1.58 The weighting structure should follow the aggregation structure of the CPI. For instance, if this structure is based on COICOP, then this is the structure which should be used for the weights.

1.59 Additional subdivisions can be introduced where there is further stratification of the sample to include geographical location, outlet type, or a more detailed product level classification. Thus, the weighting structure will depend on the sample design for price collection and compilation and in particular the need for more detailed weights which may be generated by additional sample stratification. In general, NSOs will collect some prices centrally and adopt up to four levels of sampling stratification for local price collection: locations; outlets within locations; items within different sections of expenditure; and varieties. The sampling of varieties is normally conducted in the field by price collectors and does not normally involve explicit weights (see Chapter 5). Stratification is frequently used to increase sampling and operational efficiency, especially where the retail market is heterogeneous (see Chapter 5). Some levels are the following:

  • Central shops weights to represent a small number of large supermarket chains or chain stores which have uniform prices across branches and prices are provided by the shops’ headquarters.

  • Stratum weights. For some types of expenditure, purchasing patterns may differ markedly by region or type of outlet, and in these cases, stratification will improve the accuracy of item indices. For example, each locally collected item in the index could be allocated to one of the different stratum types to allow the best available information about purchasing patterns to be incorporated in the index calculation. Depending on the structure of the retail market, the stratum types could be: region and outlet type; region only; outlet-type only; and no stratification. The assignment of stratum type will depend on the information available for constructing the weights for each item and the number of prices collected per item. In principle, all locally collected items might be stratified by both region and outlet type, but if the weights data are unreliable or nonexistent at this level of detail, then the item may be allocated to another stratum type. Allocation also partly depends on which outlet types are specified for the collection of prices and the number of prices collected. If the rules for the choice of outlets did not specify that both a chain and an independent outlet should be chosen for an item, there may be too few prices collected in one of these outlet types to make stratification by outlet type meaningful. In some instances, there may be no stratification because research has shown that stratification has little effect.

The weight of an elementary aggregate (that is, the stratum weight) should reflect the expenditure on the entire elementary aggregate and not the weights of the outlets and items that have been chosen to represent it. For instance, if spaghetti is chosen as the representative product under the elementary aggregate with the heading of pasta products, then the weight of this category should reflect expenditures for all pasta products and not solely the “lower” weight of spaghetti; that is, the weight of the pasta category will be represented entirely by spaghetti. Similarly, if an expenditure category is divided into two elementary aggregates according to outlet type, say, open markets and supermarkets, with corresponding market shares of food sales, 70 percent and 30 percent, respectively, then the same rule as mentioned previously should apply. For instance, suppose a single store is selected as the representative outlet for a particular food item sold in supermarkets in a country where two supermarket chains have equal sales, then the sales from the sampled store will account for the total value of the expenditure weight of 30 percent; the weight of the elementary aggregate for this food item sold in supermarkets should not be 15 percent (0.30 × 0.50), that is, a weight based only on the sales of the selected supermarket.

  • Product or item weights (in the current context the terms can be interchangeable). Some products or items may be intended only to represent themselves; others represent a subclass of expenditure within a section. For instance, within electrical appliances, an electric cooker may represent only itself and not any other kinds of electrical appliances. However, other products or items will represent price changes for a set of products or items, which are not all priced, so for these the weight reflects total expenditure on all products or items in the set. For example, a screwdriver may be one of several items representing all spending on small tools within home improvement and maintenance materials, and there are other items within the section representing all spending on paint, timber, fittings, and so on.

  • Upper-level or section weights. It is common practice to give each section an integer weight in parts per thousand or per hundred so that the sum of the section weights is 1,000 or 100. It is likely that most of these weights will be based on the HBS results. The main exceptions will be some housing sections, including (where applicable) mortgage interest payments and depreciation, where weights are estimated from other sources (see Chapter 11), and for certain other sections (for example, tobacco, confectionary, soft drinks, and alcoholic drinks) where the HBS may be thought to under-record expenditure and better data are available elsewhere. Many countries also use national accounts household final consumption expenditure estimates where available at the COICOP group or class level and the HBS expenditure distribution at the lower levels. This is also an area where scanner data can be used to more accurately reflect the expenditure distribution.

Implicit Weights Within Elementary Aggregates

1.60 An unweighted formula (for example, Jevons or Dutot—see paragraphs 1.145–1.151 on elementary price indices) is usually used when aggregating the elementary aggregate price relatives of the sample of products at the elementary aggregate level. This practice is usually justified on the grounds that the required information such as market shares is simply not available to a sufficient level of precision. However, if broad-based estimates of market shares are available, then these can be used as implicit weights for determining the sample of price observations to enhance the accuracy of the elementary price index. Some possible sources for this information are transaction shares from scanner data, trade publications, market reports, and consultation with industry experts.

1.61 The sample of price observations based on implicit weights can be updated independently and more frequently than the weights of the elementary aggregate; however, the price statistician will need to ensure that the weights are both coherent and consistent within the elementary aggregate. It is best to review them at the time of updating the basket.

Weights for Products for Which Prices Are Not Collected

1.62 As it is not feasible to collect a full set of prices from every outlet, including market stalls and street vendors, and from every provider of a service, all prices have to be sampled. This means that in practice there will be some products which consumers spend money on for which prices will not be collected (see Chapter 4). However, the expenditures for these products need to be included in the expenditure weights. There are two ways of doing this: (1) including the weight in a related elementary aggregate (this may involve the creation of a “miscellaneous” category), or (2) having the weight of the product for which no representative prices exist equal to zero, which essentially redistributes the weight to other elementary aggregates.

1.63 In general, the prices for the product for which prices are not collected will be expected to exhibit a similar movement to the other products in the elementary aggregate and the first of the previous two methods should be used. The second method may be used where the elementary aggregate is heterogeneous, or the associated price index is not considered very reliable. Because of the negligible size of the weight value involved, the consequence on the overall index will also be negligible whichever method is used.

Data Sources

1.64 Depending on the population coverage, weights for a CPI are derived either from expenditure data based on estimates drawn from a sample covered in the HBS or from national accounts estimates of household final consumption expenditure. It should be emphasized, however, that expenditure estimates in the national accounts are usually partially based on HBS information, although they may differ with regard to coverage, and that in some countries these are not available or not compiled at a detailed level. The two sources are not entirely independent. Note also that national accounts data may also be used when the HBS is conducted too infrequently to ensure the reliability of the CPI or when the expenditure weights need to be updated more often than the periodicity of the HBS. Nevertheless, an HBS will still have to be conducted eventually because it is an important source for benchmarking the components of household final consumption expenditure of the national accounts. Other sources for the weights are also available and are usually complementary to these two main sources. These include administrative data sources or retail trade statistics data. These are discussed in Chapter 3.

1.65 When various sources of information are used for generating the weights in a CPI, the compiler should take the time to check the data to ensure that the results are consistent and plausible with what is expected or investigate further if necessary.

Household Budget Surveys

1.66 When using the HBS as the basis for developing CPI weights, the sample size (number of households) should be sufficient to ensure that the expenditure data yielded produce statistically reliable and representative weights at the elementary aggregate level. In some countries, the acceptable statistical quality is based on the coefficient of variation (the ratio of the standard deviation to the mean). For those expenditure weights that are unable to meet the minimum requirement of reliability, three options should be considered:

  • If reliable expenditure data are not available for an elementary aggregate, they can be combined with another related elementary aggregate to form a new broader elementary aggregate (for example, “wheat bread” could be combined with “rye bread” to form a new category called “bread”). This approach will often lead to a more reliable elementary aggregate but may require an adjustment to the existing structure of the CPI.

  • If annual HBS data are available, then expenditures could be averaged over more than one year thus improving the statistical reliability of the data, with regard to standard errors, but to the detriment of timeliness. It should also be noted that averaging may not improve the statistical quality of the expenditure estimates if a particular category of household expenditure is rapidly growing or declining. Averaging is useful if the particular expenditure category under consideration shows a lot of variability over several HBSs but no clear trend. This is an area where statisticians will have to use their judgment. The basket reference period should not be arbitrarily chosen and periods of less than a year should be avoided because of seasonal influences on consumption patterns. Furthermore, some countries exclude, from multi-year averages, years which are exceptional, for example, as a result of particularly poor harvests leading to high prices and distorted expenditure weights.

  • Leave the CPI structure unchanged and simply accept that the weight for the particular elementary aggregate concerned is less than ideal. Whether this is an acceptable position to take will depend on the weight of the elementary aggregate and on its importance, particularly to analysts. For example, it would be a difficult position to sustain if the elementary aggregate has a large weight and is presented as a published subindex.

1.67 It should be noted that in normal circumstances weights can tolerate a certain degree of imprecision before having a significant effect on the overall CPI, particularly at the higher levels of aggregation, or main divisions of the CPI. But this is less so at lower levels. For instance, an index described as “fruits and vegetables” where the true weight for fruit is 40 percent and the weight of vegetables is 60 percent, but with a biased estimate of the expenditure weights, fruits account for 60 percent of the index and vegetables the remaining 40 percent. The biased weights affect the relative importance of both fruits and vegetables in the basket, giving too much weight to fruit and too little weight to vegetables. Consequently, the price index for “fruits and vegetables” will be also biased. To minimize the potential for such occurrences, it is recommended that the compiler always strives to get the best possible estimates for the expenditure weights.

1.68 An annual HBS is optimal for a CPI because as well as avoiding one-off setup costs, it permits the annual updating of the weights, hence reducing the substitution bias associated with out-of-date weights in a fixed-basket index like the CPI. Furthermore, it provides the opportunity for using multiyear weights to reduce the sampling error and, where considered appropriate, minimize the sampling variance associated with unusual expenditure patterns in a particular year (for instance, abnormal circumstances affecting consumer behavior such as political events, natural disasters, or oil shocks). But obtaining reliable consumption estimates is challenging and there is a persistent trade-off between data quality and survey cost.

National Accounts

1.69 The use of national accounts weights ensures consistency and comparability between the CPI and national accounts definitions and classification systems for household consumption, which is an advantage when compiling a CPI as a macroeconomic indicator and for use as a deflator.

1.70 National accounts have two inherent advantages:

  • The household final consumption expenditure aggregate of national accounts may be derived mainly from the HBS, but national accountants will often use other sources of information before finalizing their results, especially in cases where the accuracy of the HBS is in doubt such as where underreporting is present. National accounts go through an additional quality assurance process and reestimation should increase the reliability of the weights.4

  • Even if the HBS is updated infrequently, CPI weights can still be updated at regular intervals from national accounts data for higher-level aggregates at the division or group level.

1.71 However, there are two inherent disadvantages with national accounts data:

  • They are generally only available at the national level, so deconstruction of the national accounts data to provide a finer level of detail or to produce regional expenditure weights may be necessary using other available sources of information. Other data sources include HBSs, retail surveys, aggregated transaction from scanner data, and administrative data such as statistics on excise duty. National accounts data can be used to derive weights at the more aggregate level, and HBS data can then be applied to derive weights at the lower levels of the aggregation program. If the expenditure data from the HBS are not viewed as sufficiently detailed to ensure a minimum of acceptable accuracy, or if a demand exists for indices of a finer breakdown (for example, there is a need for a price index for apples but only expenditures for all fruit can be derived from the HBS), then other potential data sources can be used for disaggregating the expenditures, including surveys of retail sales from establishments, point-of-sales surveys and aggregated scanner data, surveys of production, export and import data, and administrative data. Note that some of these sources may also be used for stratifying expenditures according to sales volumes by retail outlet type and by region.

  • National accountants apply an element of discretion and judgment when making operational decisions relating to the construction of national accounts. Some of the details of these decisions are not always readily available to users. Consequently, compilers of the CPI should consult with their national accounts counterparts regularly before using their data for weights in order to ensure that they are consistent with the objectives of the CPI.

Designing the Sample

1.72 Chapter 4 gives advice both on sample selection, that is, how to construct a sample, and on estimation procedures, that is, how to estimate the CPI from the sample of prices collected. This Manual recognizes that in practice nonprobability sampling sometimes needs to be carried out. Similarly, although the 2003 ILO Resolution states that probability sampling techniques are to be preferred, it goes on to say that “where appropriate sampling frames are lacking and it is too costly to obtain them, samples of outlets and products have to be obtained from non-probability methods” and that “statisticians should use available information and apply their best judgement to ensure that representative samples are selected.”

1.73 To construct a perfectly accurate CPI, the price statistician would need to record the price of every variety of every good and service purchased by the consumer. This would mean collecting a full set of prices from every outlet, including market stalls and street vendors, and from every provider of a service, including public utilities such as water and electricity, private transport including shared minibuses and the hire of rickshaws, modern forms of communication, such as mobile telephones, and the provision of domestic service. As this is not feasible in practice, most prices have to be sampled and this involves local price collection in a selection of outlets in a sample of locations chosen to be representative of the country as a whole and at selected times on selected days.

1.74 The exceptions to selecting a sample of prices as described in the previous one are prices which can be collected from a central source, such as a public utility provider, national retail chain headquarters, or a government department. For many of these items, all prices will be taken and no sampling will be involved. For example, the service provider may give the NSO a full price list or tariff from which all the prices can easily be extracted. This may be the case where sampling would not make sense or would be unreliable because the number of prices is very small. For instance, no sampling would be involved if an electricity tariff consisted of a standard standing charge for service provision and a standard charge per kilowatt of electricity used, which was the same for all customers regardless of location and only varied with total usage (with heavy users getting a discount after a certain threshold). In this case, the tariff prices would be collected and applied to a typical cross-section of users and varying quantities of electricity. Sampling would be used to choose a cross-section of users.

Approaches to Drawing Samples

1.75 The focus of this section is on sampling procedures for local price collection in outlets, including options relating to probability and nonprobability sampling. There is a section in Chapter 5 which specifically addresses the special challenges of sampling prices in markets and prices charged by street traders. Chapter 5 also addresses the associated issue of price bargaining and discusses the issue of volatile prices.

1.76 As only a sample of prices will be recorded in the course of local price collection, there is inevitably some sampling error in measuring the CPI. The sampling procedures should aim to minimize this sampling error, maximize sampling efficiency (that is, obtain the maximum sampling precision for minimum fieldwork and processing costs) and reduce bias as much as possible. The sample design should allow publication of subindices at all levels which have been decided upon, such as regional indices or separate subindices for urban and rural areas. As well as cost, a limiting factor in sample design is the time taken in collecting prices. Practical considerations that will need to be considered include the availability of price collectors and transportation issues.

1.77 In general, NSOs adopt four levels of sampling for local price collection: locations; outlets within locations; items within different sections of the expenditure classification; and product varieties. Stratification is also frequently used to increase sampling efficiency, especially where the retail market is heterogeneous. Often a combination of probability sampling and nonprobability sampling is used.

1.78 When using probability sampling, the units in the sample are selected so that each has a known nonzero probability of selection. For instance, locations could be randomly selected from local administrative areas with probability according to total population,5 the population representing a proxy for the retail turnover in the area. Within a location, outlets could be randomly selected from a business register, with probability according to their individual turnover or sales or by floor area measured by an enumerator listing and visiting each shop. Sample selection based on probability according to size increases sampling efficiency. Also, as the aim is to have a sample which is representative of retail turnover, the prices subsequently collected on the previous basis would then not need to be rebalanced by reweighting if the assumption holds that those population and floor areas are good proxies for turnover. Alternatively, each location and outlet could be given an equal chance of selection in the sample, regardless of the total proportion of the retail market that they account for, but then reweighting would be necessary.

1.79 In practice, sample selection is never straightforward, and compromises must be made for good practical reasons even when a sampling frame exists. Administrative boundaries may not coincide well with statistical targets. For instance, choosing administrative areas using probability according to population could ignore the inconvenience of administrative boundaries that straddle the border between a commercial district and a residential area so that, contrary to the intention, the commercial district has no chance of being selected as it contains no houses. Also, a visit to the location may indicate that it is impractical for the collection of prices, for example, because of a physical barrier such as a railway or river bisecting the area and causing difficulties of access. Similarly, very rarely do NSOs have readily available sampling frames which reliably list all retail outlets, particularly recent openings, and even fewer will have lists that cover all market stalls in all types of markets, or mobile street vendors. The relative advantages and disadvantages of random and purposive sampling should be examined at each stage of the sample selection. It is recommended that the NSO should first decide on the ideal sampling solution and then modify this to take into account practical constraints.

1.80 The ultimate goal should be:

  • An overall sample which is representative of the total population of goods and services being offered for sale and purchased. The sample chosen should be representative of price levels and, most particularly, price movements. All variations of items and outlet types should be considered for each product and chosen to reflect typical consumer purchasing habits.

  • A variance or mean square error which is as low as possible. Samples should be reasonably optimized. At the very least, a basic analysis of sampling variance should be conducted, even if an overall estimate of the precision of the CPI cannot be made.

  • Optimization. The entire set of sample prices should be optimized to meet the publication needs of the CPI, taking into account user requirements, practical data collection considerations, and cost.

Collecting and Editing the Prices

1.81 Chapter 5 provides an overview of the most appropriate survey methods for the collection of prices while Chapter 10 discusses the detailed approaches that can be used for scanner data. In large part, the considerations are the same as for sample design and will depend on local circumstance. For instance, the methods should consider: the purchasing habits of consumers and the extent to which they use licensed and unlicensed markets and internet purchases; the structure of the retail market including the balance between markets, small independent shops, and large retail chains; the extent of public ownership and price control; the diversity of goods and services being sold; the pricing structures used, including tariffs; and whether bargaining is common. The availability of central records of prices charged also has an important bearing. The 2003 ILO Resolution emphasizes the importance of well-trained price collectors who adhere to the standard procedures.

1.82 There are two basic price collection methods:

  • Local price collection where prices are obtained from outlets located around the country. This will include licensed and unlicensed markets and street vendors as well as shops. Normally the price collector will need to visit the outlet to observe the prices although the prices for some items may be collected by other means, including telephone and price lists.

  • Central price collection is often used where prices can be collected by the head office without the need for fieldwork. This may also include centrally regulated or centrally fixed prices which can be obtained from the regulatory authorities, although in these cases checks will need to be made to ensure that the goods and services in question are actually available and sold at the stated price. It is not unusual to find goods subject to price control being sold at a different “unofficial” price. Central price collection can be further broken down into:

  • Prices which are combined with prices collected locally. For instance, this may occur when a supermarket chain provides centrally determined price lists or actual transaction data, from scanners, eliminating the need for the price collector to visit a shop in person.

  • Prices which are used on their own to compute centrally constructed indices. Most tariffs fall into this category.

1.83 Goods and services that normally are paid according to a tariff can pose problems when their structures are modified over time, compromising the principle of unchanged consumption. Examples include public transport fares, electricity, main water supply, physicians, hospital services, and telecommunications. For utilities, the payment may consist of a standard rate per unit of consumption sometimes in combination with a fixed charge. A solution to this problem is to define representative services or bundles of services (for example, categories of consumers and specific services consumed). For these, it is important that the prices experienced by a representative range of customers and tariffs are observed and that customer profiles are kept constant over time. More detailed advice is given in Chapter 11 on selected special cases.

1.84 The focus in the next paragraphs is on locally collected prices. It begins by reviewing the principles behind collecting prices for a CPI and then considers the practical collection issues and how these should be managed. A working assumption has been made that the index being compiled is an acquisition index (see Chapter 4). It is also assumed that prices are being collected for a monthly price index with prices therefore being collected, in general, every month. Some countries produce only a quarterly CPI, while others produce a weekly index, especially for fresh food. The concepts and procedures discussed will apply to price collection practices, whatever the frequency of index publication.

The Principles of Price Collection

1.85 Except in a small number of cases, such as the treatment of owner-occupied housing costs, a CPI is usually designed to measure the change in the actual transaction prices of goods and services bought by consumers. However, collectors cannot normally observe individual transactions as they occur, so they must usually observe the price marked on or assigned to the product and assume that these are the transaction prices. Many countries are also improving the collection of actual transaction prices through the use of electronic technologies such as scanner data and internet purchases. An exception to observation and electronic capture is bargaining, where a price might not even be displayed, and this case is discussed in detail in Chapter 5. Special procedures apply when outlets in which items are being priced close down, or the items which the price collector was pricing at a particular outlet are no longer sold by that outlet. These procedures are covered in Chapters 4 and 7.

The Principle of a Fixed Basket

1.86 Underlying all of what follows in this section is an important principle: the necessity to compare prices on a like-to-like basis from one period to the next. This has two consequences:

  • Where the price collector has the role of selecting what variety of a product to price in a particular outlet, a consideration should be whether that variety will be available to price over a reasonably long period (tight specifications are of no use if the described variety cannot be found in the outlets). This is in addition to being typical of what is sold to customers.

  • The price collector should record additional information needed to ensure the unique identification of the variety priced so that:

  • The same variety continues to be priced in the case of price collection being carried out by a different person.

  • The identification of quality change can be evaluated when the variety disappears and is replaced by a different one allowing an adjustment for quality change to be made.

Variety Specifications

1.87 There are no firm rules, especially regarding the use of looser or tighter variety specifications: each country may choose its preferred methods—and stick to them. However, there are a number of considerations in deciding on variety specifications:

  • Tight specifications leave less discretion to the price collector, so the reliability and training of collectors are factors to consider when deciding whether to use loose or tight specifications.

  • Particular care should be taken to ensure that the specification is very detailed for heterogeneous items where there is scope for significant difference between one variety and another, and for items which by nature are highly specified. Cars and hi-tech goods fall into the latter category.

  • Tight specifications also allow for the calculation of meaningful average prices:

  • Average prices are useful in identifying outliers and assessing the accuracy of the reported prices.

  • Average prices allow comparisons of price levels, including between, for example, regions or urban and rural areas.

1.88 Responsibility for specifying the items to be priced should normally rest with the head office. Specifications should be reviewed on a regular basis to determine whether they continue to be relevant. A revision of specifications could be implied by: (1) a large number of missing price quotations; (2) a large number of substitutions; or (3) a wide variation in the distribution of collected price levels.

1.89 Some countries find structured product descriptions (SPDs) from the ICP helpful for specifying items to be priced in a CPI. As well as providing a ready-made framework for detailed item specifications the use of SPDs has the additional advantage of facilitating greater integration between the two price collection exercises leading, among other things, to savings in collection costs from using the same price quotes in both the CPI and ICP.

Data Editing

1.90 Once the price information has been collected and recorded it has to be edited. Data editing is the process of ensuring correct and usable data for calculation of price indices for elementary aggregates. Data editing is sometimes referred to as input editing. There are three steps in this process:

  • The detection of possible errors and outliers

  • The verification and correction of data

  • Output review

The detection of errors in the collection and recording of price information must occur as soon as possible after the information is collected. Detection is usually achieved by examining price movements and checking those that exceed some predefined limits or appear to be unrealistic based on an analysis of all available information. Statistical outputs should sufficiently portray the reality of the economy, and with an output review, the compiler ensures the indices reflect reality.

1.91 It should be noted that while price collectors should examine every price they collect, subjecting every collected price to the same level of examination by collection supervisors and index compilers is not considered necessary and generally is not feasible. It is recommended that, to improve cost-effectiveness, some form of significance rating should be applied to determine how much time and effort should be expended on editing individual prices.

  • In general, prices from elementary aggregates with relatively small price samples should receive more attention from the index compiler. This is because the weights of the elementary aggregates are broadly equal. Each individual price movement from these elementary aggregates will have potentially a much more significant influence on the index calculations than any individual price movement from an elementary aggregate with a large number of price quotes.

  • Price samples from elementary aggregates with high expenditure weights should be examined critically as the high expenditure weight will make all price movements within the sample significant to the index calculation.

  • The highest risk is associated with elementary aggregates with relatively large weights but few price quotes and with complex index construction. This situation is associated with utilities and other services which account for relatively large expenditures and where there may be only one or a handful of suppliers and prices are based on complex tariffs. Petrol prices could be another example.

1.92 There are two main categories of checking and identifying possible data errors and outliers, as further described in Chapter 5:

  • Nonstatistical checking

  • Statistical checking

1.93 Some of the editing techniques discussed in Chapter 5 work best when applied to large quantities of data. The best results will therefore be obtained if conducted at the location where many prices are available to any analyst. This will generally be regional offices or, more probably, in the head office. However, the techniques can be adapted and still be applied to prices held by a small collection center as a way of quickly and efficiently detecting extreme prices. Abnormal prices such as sale prices, or price movements, such as sale recovery prices, may be excluded from automated procedures for the detection of outliers, in particular the setting of upper and lower bounds, but should nevertheless be checked, for instance by reference to previous price history. For seasonal items, such as fresh food and clothing, abnormal price movements will be normal. These price movements should not be excluded from the outlier detection procedures. It is important that an appropriate method is used to validate these prices. Chapter 5 provides more details on the different data validation methods.

Maintaining and Updating the Sample

1.94 One strategy to deal with the changing universe of products would be to resample, or reselect, at regular intervals the complete set of items to be priced. For example, with a monthly index, a new set of items could be selected each January. Each set of items would be priced until the following January. Each January, both sets would be priced in order to establish a link between each set of 12 monthly changes. Resampling each year would be consistent with a strategy of updating the expenditure weights each year. For example, the Harmonised Index of Consumer Prices and many national CPIs in the European Union Member States resample items annually.

1.95 Although resampling may be preferable to maintaining an unchanged sample or selection, it may not be practical for those countries that update weights infrequently. When the CPI is updated every five years, for example, systematically resampling the entire set of products each year would be difficult to manage and costly to implement. Moreover, it does not provide a complete solution to the problem of the changing universe of products, as it does not capture price changes that occur at the moment when new products or new varieties or models are first introduced. Many producers deliberately use the time when products are first marketed to make significant price changes.

1.96 A more practical way in which to keep the sample up to date is to rotate it gradually by dropping certain items and introducing new ones. Items may be dropped for two reasons:

  • The product is believed by the price collector or head office to be no longer representative. It appears to account for a steadily diminishing share of the total expenditures within the basic categories in question.

  • The product may simply disappear from the market altogether. For example, it may have become obsolete as a result of changing technology or unfashionable because of changing tastes, although it could disappear for other reasons.

1.97 At the same time, new products or new qualities of existing products appear on the market. At some point, it becomes necessary to include them in the list of items priced. This raises the general question of the treatment of quality change and the treatment of new products.

Missing Products and Adjusting for Changes in Quality

1.98 A CPI should reflect the change in the cost of buying a fixed basket of goods and services of constant quality. In practice, this represents a challenge as products can permanently disappear or be replaced with new versions of a different quality or specification, and new products can also become available.

1.99 Chapter 6 discusses the nature of “quality” and methods for adjusting prices for quality change. It provides detailed guidance both on implicit methods of quality adjustment, such as the overlap method and class mean imputation, and on explicit methods, including expert judgment, and the hedonic approach. Chapter 7 delves more deeply into the issue of item substitution, particularly on methods of incorporating new products into the index.

1.100 The 2003 ILO Resolution advises that when a product disappears “clear and precise rules should be developed for selecting the replacement product.” It lists three selection strategies: the most similar; the most popular; and the most likely to be available in the future. On quality adjustment, this resolution states that “when a quality change is detected, an adjustment must be made to the price, so that the index reflects as near as possible the pure price change.” It guards against the automatic assumption that “all price change is a reflection of the change in quality.” It does not recommend particular explicit or implicit methods of quality adjustment but does state that “the methods used should as far as possible be based on objective criteria.”

Missing Products

1.101 In order to measure price change from one period to the next, the price statistician tracks, for each elementary aggregate, the prices of a fixed sample of items. The detailed characteristics of the products, that is, the varieties of goods and services selected for pricing, are recorded to assist the price collector in fulfilling the aim of pricing exactly the same product in the same outlet in the same location so that the CPI compares “like-to-like” in subsequent periods. Also, the recording of detailed characteristics, especially price-determining features, can help when needing to make adjustments to the recorded price due to changes in specification and hence quality. In practice, the particular product being priced in a specific outlet may become unavailable—for example, the product is discontinued, may be in temporary short supply, or may be a seasonal product which disappears when out of season. These situations are discussed in Chapters 6 and 7 of this Manual, and in Chapter 9 of Consumer Price Index Theory. In all other cases, the price statisticians need to estimate the price of any missing product that they believe will return to the market within a reasonable time, or, if they believe it will not return, find a suitable replacement. If the price statisticians believe that the product will not return, the replacement should be either (1) as similar as possible to the previous one, or (2) the most popular “similar” product in the shop, or (3) the similar product that most likely will be available for future pricing. Unlike approach (1) which leaves the sample “static” with the risk that it will be increasingly out of date and difficult to collect prices for, approaches (2) and (3) have the advantage of introducing an element of sample replenishment. This is where quality adjustment becomes an issue. The price index should reflect only pure price changes—the price index should not reflect any portion of the price difference that is due to increases or decreases in quality between the missing item and its replacement. A value needs to be placed on any change in specification between the old and replacement item and a quality adjustment applied accordingly. This approach to quality adjustment applies to any replacement strategy, but is particularly relevant where sample replenishment takes place.

1.102 Three situations that regularly occur are the following:

  • Substitution procedures where an item, product, or outlet disappears, including the introduction of new items

  • The imputation of a price when a product is temporarily out of stock (excluding seasonal goods)

  • Quality adjustment where a change of product involves changes in its price-determining characteristics

Substitution Procedures

1.103 In a dynamic retailing environment, there is a continuous turnover both in outlets and in products.

Outlets

1.104 If an outlet goes out of business or refuses to participate in the price collection survey, it should be replaced with the same sort of outlet (for example, a market stall should be replaced with a market stall, or a single shop with a single shop) in the same location and conducting the same type of business (in other words selling the same types of goods). For example, if the previous shop was a butcher selling refrigerated meat, then another butcher selling refrigerated meat should replace it. If probability sampling was used to select the original outlet, the sampling frame should be revisited and a replacement outlet selected from the same stratum. Regardless of how the replacement is found, the original outlet’s sampled items should be assigned to the replacement outlet for price observation.

1.105 If an outlet changes location, a decision on whether the price collector should follow the outlet to its new location needs to consider both sampling and operational issues:

  • Sampling issues. The principle of maintaining a like-for-like comparison holds. In practice, the balance of the sample needs to be maintained to ensure that it continues to be representative. Stratification is frequently used to increase sampling efficiency and ensure that the sample is representative. This means that when an outlet changes its location, reference needs to be made to the stratification and selection procedures used in the initial sample selection. For instance, assume that shopping locations were initially selected from local administrative areas, stratified by an urban/rural split and outlet type, and then outlets randomly selected from those outlets within the chosen local administrative areas. Then the relocating outlet can be followed to its new location if it continues to fall in precisely the same stratum. However, if the outlet moves away from the original shopping location, for example, from an urban shopping district to a rural area outside the city or to another urban district within the city, or if it relocates within the same shopping location but becomes part of a multiple chain, then it has moved to another stratum and a suitable replacement for it should be found from within the original stratum, in order to maintain the sample balance.

  • Operational issues. As mentioned in Chapter 5 there may sometimes be operational reasons for departing from the sample generated by the standard selection procedures. For instance, efficient scheduling of price collection and the availability of price collectors may make following the outlet to its new location impractical, even though it has stayed in the same stratum. Similarly, a visit to the new location may indicate that it is impractical for the collection of prices, for example, because of a physical barrier such as a railway or river bisecting the area and causing difficulties of access.

Products

1.106 If a chosen product is temporarily missing and no price is recorded, a note to this effect should be made by the price collector. For a product temporarily missing, a price has to be imputed. Nonseasonal items and varieties should be replaced if missing more than a predefined period of time. For example, if it is out of stock for three consecutive months, then the collector should be instructed to choose a replacement which matches as closely as possible the product description unless it is decided to take the opportunity of a disappearing good to update the sample. Where a product is permanently unavailable for pricing, procedures need to be in place for determining a replacement and then impute a new base price if the replacement is of a different quality. Methods for imputing a missing price are discussed in the following text.

1.107 As the issues relating to temporarily and permanently missing products differ—and their treatment is different—it is important for the price collector to establish whether the unavailability of a product is likely to be temporary or permanent. A price may be considered as temporarily missing if the same product is likely to return to the market within a reasonable time period. This includes seasonal items for which special procedures apply (see Chapter 11). Permanent unavailability, on the other hand, occurs when a variety is withdrawn from the market with no prospect of returning in the same form. Products may be temporarily missing, for example, due to supply shortages caused by factors such as the seller underestimating demand, strikes by factory or transportation workers, or supply problems with imported products. In these cases, the price collector, although not able to observe a price in the current period, may have obtained information (for example, from the shopkeeper) to suggest that the same variety will become available again at some, perhaps unknown, time in the future.

1.108 The previous discussion does not cover seasonal products, that is, where a product or item may disappear, because it is a seasonal product and it may be expected to reappear when it is next in season. The case of seasonal products is covered in detail in Chapter 11. Imputation procedures are fairly similar for both temporarily missing and seasonal products.

Temporarily (Nonseasonal) Missing Products

1.109 If it is believed that a missing product will be available again in a reasonable time, then the price statistician has three options:

  • Omit the variety for which the price is missing so that a like-for-like comparison is maintained using matched pairs. The elementary index uses only those observations for which the price collector obtained prices of exactly the same variety in the current and previous periods. In this approach, the price change for the deleted product, which was recorded up to the point immediately before its disappearance will be disregarded from that point on.

This may cause problems, for example, if it unbalances the sample.

  • Carryforward the last observed price. Carrying forward the last observed price is only recommended in the case of fixed or regulated prices. Although this provides price continuity in the periods when observations cannot be made, it is likely that short-term movements in the index are biased, since the subindices in question will show no change when prices are not available. If prices in general are rising, the bias will be downward, whereas if prices are falling, the bias will be upward. Carryforward is not recommended, particularly when there is high inflation or when period-to-period movements (as opposed to annual movements) in the price index are important. The carryforward method is appropriate only if there is reason to believe that the price has not changed. Typically, it will be difficult for the price statistician to validate the belief that the price has not changed, unless the price is fixed or regulated.

  • Imputation. The best solution by far is to impute a price. Imputation makes use of the best available information to provide an unbiased estimate of price movement. The principal methods for imputing prices are shown in more detail in Chapter 6. There are essentially two choices:

  • Impute the missing price by reference to the average price change for the prices that are available in the elementary aggregate (overall mean imputation). This assumes that the price change of the missing product, if it had been available in the shop, would have been equal to the average change in prices in the elementary aggregate. This may be a reasonable assumption if the elementary aggregate is fairly homogeneous. This method of imputation is equivalent to the “omit” method (see first bullet point), no matter whether a Jevons, Carli, or Dutot method of aggregation is used at the elementary aggregate level. In a given month this approach provides similar results to the “omit” method described in the first bullet previously; however, across time, the two approaches will not produce the same results if the index is compiled using the short-term formulation. This is because the imputed prices are used to compile the index from month to month.

  • Impute the missing price by reference to the average price change for the prices of “comparable” varieties from another similar outlet (class mean imputation). This represents a more precise match between the missing product and the products supplying an imputed price. It is normally preferable to impute using the average price change in the elementary aggregate unless the imputations are unreliable because of small sample sizes.

1.110 A detailed discussion on imputation methods is presented in Chapter 6.

Permanently Missing Products

1.111 When the situation arises where a product permanently disappears or is replaced by a new version with a different specification, normally two actions are required by the price statistician:

  • Selecting a replacement product for pricing

  • Quality adjusting the price if there are differences in quality between the outgoing product and its replacement

Each is considered in turn.

The Selection of a Replacement Product for Pricing

1.112 In practically every period for which a price index is compiled, some varieties of a product become permanently missing, not just in particular outlets but also because they are no longer produced or imported. If no action is taken, the sample of prices will diminish. Permanently missing varieties are problematic not just because of the potential impact on how representative the sample is, but also because it will lead to estimation of price change with samples that do not match from period to period; that is, the composition of the matched pairs changes. In addition, the index number for the latest month will be less reliable than that for the previous month because of the smaller sample size.

1.113 The price statistician’s task is to maintain the sample size by finding replacements for the specific varieties when they are no longer available and are not expected to return within a reasonable time. One of two alternative strategies can be adopted (replace with the most similar product and sample replenishment). Under both options it is important to identify any differences in quality between the original and replacement varieties as it is crucial to ascertain whether there is a quality difference, and, if there is, to estimate its value and calculate a quality-adjusted price.

1.114 Replace with the most similar product. This reduces the role of quality adjustment, as the more similar a product is, the less is the required quality adjustment (see paragraphs 1.118–1.131) but contributes to the deterioration in the representativeness of the sample where a product starts disappearing from the shops because it is being replaced by something new and sales are declining. Finding the replacement that is most similar to the original variety requires knowledge about characteristics of the previous variety. Good practice in price collection involves maintaining up-to-date descriptions of the variety’s characteristics. The ICP developed SPDs for most item categories: these provide a framework to list the various characteristics—prioritized in order of their importance—of the varieties for each category.6 Such descriptions of characteristics can be used to match the characteristics between the old variety and various replacements so that critical characteristics are matched and less important characteristics can be noted. Critical characteristics are those that impact or contribute to determining the price, such as type of product (canned tuna fish), brand (StarKist), size (150 milliliter), and packaging (tin, in water). Less important characteristics are those that do not affect the price, such as color of the label on the package or the location in the shop where the product is displayed. The salient considerations can be incorporated into a decision-making framework for identifying a similar product, as follows:

  • There is a basic match of the main characteristics, particularly those which determine price.

  • Consumers perceive them as similar even though some characteristics may be different. This may be the variety in the shop that buyers are most likely to buy in place of the original.

  • They are used for the same purpose and in similar situations. For food this may include a consideration of whether the brand is one for everyday consumption or only for special occasions.

1.115 Under the “replace with a similar product” strategy, an updated version of a product (that is, the one that the supplier lists as the replacement), is the logical starting point for the replacement for its predecessor. In most instances this would be the one that is the most similar to the original variety, so the price statistician can compare “like-to-almost-like.” In instances where the most similar variety is also one that is likely to soon disappear, the price collector should select the variety that is most popular within the outlet for the product class. Although this strategy is less likely to yield a replacement that is sufficiently similar to permit direct price comparison, it will reduce the likelihood of the replacement disappearing in the near future and will keep the need for quality adjustment to a minimum.

1.116 Sample replenishment—replace the missing product or variety with the currently most representative one by going through a process of resampling. The extent to which a sample remains representative is highly dependent on the rules used for item replenishment when a particular variety or product disappears from the shelf of a particular outlet. Compared with the option of replacement with the most similar product, sample replenishment has the benefit of maintaining the current representativeness of the sample. If disappearing products are always replaced with similar products, the sample will gradually become less relevant to market reality. Sample replenishment also increases the chances of the replacement being available on the shop shelf for pricing. The problem of a deteriorating sample increases with the rate of turnover in varieties and products and with the rate of product development.

Adjusting the Price for Differences in Quality

1.117 When a product disappears or is replaced with a new version of a different quality or specification, then one of the following methods of introducing the price of the replacement is adopted:

  • Comparable replacement

  • Direct (explicit) quality adjustment

  • Implicit quality adjustment (imputation)

In all cases, a nominal price in the base month is needed for the new or replacement product—this nominal base price is used until the next rebasing.

1.118 The price collector should record the specifications (the price-determining characteristics) of the new variety so that the head office can determine if the replacement is of similar quality to the original variety. The price collector should determine if the outlet is likely to continue selling the replacement. If it is also expected to be discontinued in the near future, then a different variety should be selected—either another that is similar to the first, or the most popular variety within the product line. As will be seen, there are several different methods for both the explicit and the implicit types of quality adjustment, but there are some common themes in the methods of each of these two main types. Explicit methods estimate the impact on price of changes in characteristics or features of the product. Implicit methods estimate the impact on price indirectly by reference to price differences between different varieties.

Comparable Replacements

1.119 If the selected replacement product is regarded as comparable, then the observed price change is treated as a pure price change. But the price statistician should gather and examine all the available evidence, if possible taking advice from market experts where necessary, before coming to such a conclusion. Even in cases where a replacement product is believed to be of comparable quality, care should be taken, since experience suggests that most goods tend to undergo steady improvements, especially hi-tech and electrical goods.

Direct or Explicit Quality-Adjustment Methods

1.120 There are a number of direct or explicit methods for determining the price associated with a change in quality. But quality adjustment is not an exact science, and different evaluations can yield different results. The point is that even if the evaluation methods used are somewhat imprecise in the measurement of the quality difference, it is important to make an adjustment. If quality differences are not removed, the price index will reflect the quality change in addition to pure price change and is likely to have an upward bias because quality usually increases.

1.121 Chapter 6 provides a flowchart of the decisions needed for making quality adjustments. The most common ways of making an explicit quality adjustment in practice are as follows:

  • Package size adjustment. The value of a change in package size is assumed to be proportional to the relative change in the package size.

  • Expert judgment. The price collector determines the value either through direct knowledge or in consultation with personnel in the shop where the product is sold. Alternatively, NSO staff, who may be knowledgeable about the product, can estimate a value. Reliance on an individual’s knowledge concerning the products depends, however, on the individual obtaining sufficient market information and is also liable to subjectivity. The judgment needs to be properly informed.

  • Reference to the producer cost. The production cost from the producer can be used in the case of an improvement to an existing product, although a judgment then needs to be made on whether to apply an adjustment factor. For example, the adjustment might consist of the normal retail markup to reflect wholesalers’ and retailers’ costs and profits. In the context of a COLI, a downward adjustment may be appropriate to account for the fact that the new “standard” feature will not increase the utility to all purchasers, for example, some may not welcome air conditioning in a car because of the extra running costs. Compilers of the producer price index often attempt to gather information on production costs from manufacturers for quality-adjustment purposes. However, producers may be unwilling to provide information on their marginal costs for reasons of confidentiality. There are a number of potential problems with using the producer cost method. In particular, it is not necessarily the case that the cost of production, with an adjustment factor along the lines described previously, gives a good indication of the selling price.7

  • The former “option” price. In the case where an optional feature has become standard, the former price charged for the optional feature can be used as the explicit quality-adjustment value but again, consideration will need to be given to whether a scaling-down factor should be applied—in this case possibly a downward adjustment to reflect the reduced production costs of making a feature “standard” and also, in the cost of living context, that the utility gain is less than the increase in price. Some of the concerns relating to using producers’ costs (see the previous bullet point) apply, the main difficulty being that it is likely that the market valuation of the options will have changed once they become standard, indeed, it is often because of changing market circumstances that producers make former options standard.

  • Hedonic regression. Another way to obtain a value of the quality difference is to use hedonic regression to estimate the value of changes in a product’s characteristics. See Chapter 6 and also the Handbook on Hedonic Price Indexes and Quality Adjustments in Price Indexes published by the Organisation for Economic Co-operation and Development for more information.8 Hedonic methods require large databases with a wide range of product characteristics. Such databases are seldom available in NSOs (though for some products scanner data from large stores may be a viable source in some countries) and can involve substantial development and maintenance costs. In addition, hedonic models need to be reestimated periodically. Hedonic methods should be applied only where they add significantly to the statistical integrity of the index. This is most likely to be the case with hi-tech high-turnover goods.

1.122 In explicit methods, the monetary value of the quality difference is determined directly using one of the previous methods and then applied to a previous period’s observed price for the “old” item. This will yield an estimate of what the replacement item would have cost in the previous period.

1.123 Assume that the NSO was able to determine that, based on the differences in characteristics, the value of the quality difference between Brand C and Replacement 1 was $25 in period 1. The price statistician can add this amount to the price of Brand C in period 1 to obtain an adjusted price.

Implicit Quality Adjustment

1.124 If the replacement product is of a different quality or specification and no information is available to quantify the difference, then assumptions have to be made about what proportion of the price difference is accounted for by differences in quality. Implicit quality adjustment creates an imputed “quality-adjusted” price based on price changes from similar varieties of the product. The precise nature of the imputation depends on the index formula that is used. A basic assumption underlying the most commonly used implicit quality-adjustment methods is that the difference in quality between varieties simultaneously available in the market is equal to the difference in price between the varieties or models.9 Thus, when a product disappears from the shelf, an underlying assumption is made when imputing a price that a price differential continues to reflect a difference in quality. The different methods of imputation available to the index compiler are described in paragraphs 1.127–1.131.

1.125 Most countries construct some form of a fixed-base price index. If price movements are estimated using long-term price relatives from the base period, then the base price may be adjusted proportionately for the estimated quality difference. If price movements are estimated using short-term relatives from the previous period, then an imputation adjustment can be implicitly made by estimating the missing variety’s price in the current month from the average price change in its elementary aggregate. The price change of the omitted observation is equal to the change in its elementary aggregate.

Overlap Pricing

1.126 This method requires knowledge of the prices of the two varieties in the same time period—the overlap period. If the old and new varieties are available at the same time because the price collector either knows in advance that the old variety will disappear soon and selects and prices a replacement, or the outlet is able to accurately give the price of the replacement in a previous period when the old variety was priced, then the price difference between the two is taken to be the value of the quality difference. The rationale is that it cannot be due to price change because price change occurs only over time. The price index uses the old variety in the overlap period and the new variety in the next period and the price differential between the old and new varieties never affects the index. In this case, the market has determined the value of the quality adjustment.

Overall Mean Imputation

1.127 Overall mean or nonclass mean imputation (also referred to as “linking,” “splicing,” or bridge overlap method in the context of the European Union’s Harmonised Index of Consumer Prices) imputes an overlap price for the old variety in the current period by reference to the price changes between the previous and current periods of similar varieties or items. The latter are used to impute a price change for the old variety, which can then be used to obtain the imputed price. The ratio of the imputed price for the old variety and the price of the new variety in the current period is the estimated quality adjustment.

1.128 An estimate of the price of the missing variety can still be made even though the price of the replacement variety may not be known for the previous pricing period using the overall mean price change for the elementary aggregate.

1.129 The overall mean procedure assumes that the pure price change from the replaced item to the replacement item is the same as that for the composite of all other goods in the same group. It is used frequently because of its simplicity, but there can be an inherent bias built into the methodology, particularly when major model changes are occurring. The direction and extent of this bias is a matter for debate but depends on whether the actual quality-adjusted change in price is bigger or smaller than the measured price changes of the items used in the imputation. Major price changes can frequently occur at the time new varieties or models of a product are introduced. This is quite common, for example, with new vehicles, household appliances, electronic equipment, and clothing items. As the new varieties are introduced there may still be a substantial supply of the old varieties which are showing little price change or may actually be declining in price. In consequence, using the old varieties’ price changes to impute the price changes for new ones will underestimate the actual price change for the new varieties and cause a downward bias in the price index. The use of the overall mean imputation procedure, in which all observations in the elementary aggregate are used, is not recommended for such cases. It is also possible, and can be argued (but less convincingly), that the use of the overall mean leads to an upward bias because the price changes associated with models that are unchanged in quality will be further along the evolutionary cycle and therefore will be rising less rapidly. An alternative imputation procedure, called “class mean” imputation, avoids some of the problems associated with bias.

Class Mean Imputation

1.130 Class mean imputation is similar in procedure to the overall mean imputation, but uses only the price changes of “comparable” replacements to impute the overlap price, the replacements being limited to those that have exactly the same price-determining characteristics, or those items with replacements that have been declared comparable after review or have already been quality-adjusted through one of the “explicit” methods. For example, when the arrival of a new model of a particular brand of motor vehicle forces price collectors to find replacements, some of the replacements will be of comparable quality, others can be made comparable with explicit quality adjustments, but the remaining ones will need imputed prices. Class mean imputation calculates imputed price relatives using only the prices of comparable and, where appropriate, explicitly quality-adjusted varieties or models. In general, it does not use the prices for the varieties or models that were not replaced, because these are likely to be different from those of new models. The prices of old models tend to fall as they become obsolete, while the new models (represented by the replacements) tend to have a higher price before falling. This may not be relevant in developing countries where new products appear in the retail market relatively late at a “mature” price. Using class mean imputation adds complexity but reduces two types of bias referred to earlier: bias from ignoring quality change altogether and treating all price movements as price change, and bias from overadjusting for quality change by treating some pure price change as quality change.10

New Products

1.131 An entirely new product, in contrast to a new variety of an existing product, is essentially a replacement of a previously popular product and represents a good or service that:

  • Was not included and could not be included in the price index during the initial selection of the current market basket and which is now available for possible inclusion in the index

  • Cannot be easily linked to the service flow or production technology of existing goods and services; that is, it represents a distinct departure from previously available products insofar as it is a step change with regard to technology or utility to the consumer

  • Has a recognizable and generally accepted new benefit to consumers as a result of becoming available

1.132 The last two cases help to distinguish an entirely new product—referred to as a revolutionary product—from an existing product whose features and, in consequence, “quality” has changed—an evolutionary product. A revolutionary product is an entirely new good or service that is not closely tied to a previously available product. A revolutionary product tends to be a good or service that is expected to satisfy some need in a new way and is unlikely to ft neatly into an existing CPI item category. As an example, when mobile telephones were first introduced, they provided a significantly new service. While on the one hand, they provided an extension of an existing flow of service (telecommunication), on the other hand, they provided a dimension of service that was new (the opportunity to make “mobile” calls away from a fixed telephone) and a distinct product from existing landline telephone services (it was a step change in technology). It is therefore an example of a revolutionary product. More recent examples of revolutionary products are broadcasting (streaming) services for television and smart-phones, downloads of games, and electronic storage of data (the “cloud”). Examples of evolutionary products would be new models of household appliances such as refrigerators and washing machines where improvements in quality are introduced from time to time. Evolutionary products can also be newly added brands of currently available products such as a new type of canned fish or electronic appliance which differs from those currently available. For example, a current brand of canned fish may consist of certain types of fish (mackerel, salmon, or tuna) and then a new variety for one of the canned fish is introduced which is packed in water rather than oil.

1.133 The focus is on keeping the basket of goods and services that are priced up to date and relevant. It covers both truly and completely new products, that is, those which are revolutionary, as well as those which are evolutionary and also goods or services previously provided free and thus previously excluded from the CPI. It does not deal directly with substitution and quality change when a good or service unexpectedly disappears. This is the subject of Chapter 7.

Planning for the Introduction of New Products

1.134 There are three sets of circumstances in which new goods and services are included in the CPI:

  • As “replacements” for products which no longer exist. This is normally associated with evolutionary products. Producers often discontinue old versions of their products and introduce new versions that are quite similar but may be of a different quality at a different price. Note that “quality change” includes changes in technical specifications as well as more clearly visible outward changes in design. This can happen frequently and is usually unplanned for in a CPI, although not necessarily unexpected. It is often associated with forced replacements when collectors go to price a product only to find that it is no longer sold. In the CPI, collection procedures usually instruct price collectors to replace the old versions (models) with:

  • The most similar model. For example, when the old model of washing machine is discontinued, the price collector is instructed to replace it with another model which has similar (though probably not identical) specifications and to record any changes in characteristics (specifications) to aid the evaluation of potential quality differences.

  • Alternatively, replacements can be products that are currently the most popular with consumers. This represents a deliberate attempt to refresh the CPI basket when a replacement has to be made. For example, the current varieties of canned fish may include tuna. Producers may have introduced a new variety that contains tuna packed in water rather than oil, and consumers are now shifting their buying patterns to purchase more of the new variety. There is no external factor forcing the consumer to change to the new product.

1.135 In some instances, when a model ceases to be produced, the manufacturer will indicate which model is the replacement and the CPI collection procedures instruct the price collector to start collecting the price of this replacement. This also normally contributes to the replenishment of the sample, as manufacturers usually introduce more up-to-date features into new models. However, the representativeness of the new model, as measured by its popularity, will only be determined over time.

1.136 Further discussion of the methods for introducing into the CPI basket replacements for products which no longer exist and disappear from the shop shelves is presented in Chapter 7.

  • As a supplement to the sample by adding a new variety or making a targeted replacement to drop an old variety and add a new one. This represents a more proactive approach to product substitution. Again, it is normally associated with evolutionary products. The CPI collection procedures instruct price collectors to replace the old, less popular variety of canned tuna fish in oil, with the new, more frequently bought tuna packed in water even though the former remains available. This is different from the standard reactive approach of replacing the old disappearing variety with the new one because the old variety still exists and may not be discontinued although may be increasingly difficult to find (and less popular/ lower turnover shares). The new variety is supplementary to the old variety and begins to gain market share while the old variety declines in market share. This more proactive approach requires the price statistician to monitor the market for the entry of new varieties and to get a sense of their popularity with consumers, for example, by noticing the changing proportions of shelf space occupied by the different varieties or by talking to the shopkeeper. The head office can also help by gathering sales information from other sources.

  • As a planned introduction of a revolutionary product which consumers begin to buy so that the product has an increasing share of the market. The appearance of revolutionary products in the marketplace and consumer reaction to them, as measured by sales, are less predictable than for evolutionary products. Revolutionary products also tend to have different price trends from other products in the sample and can therefore exert an influence on the CPI disproportionate to actual sales. For these reasons, revolutionary products are important, represent a significant challenge, and warrant special attention, requiring the price statistician to be particularly attentive and proactive.

1.137 The previous circumstances can be managed either in a planned way, as part of a regular process of updating the CPI basket, including chain linking (covered in Chapter 9) or in an ad hoc way when the need arises, or the circumstances warrant action to be taken.

Timing of the Introduction of New Products

1.138 The timing of introduction of new products can vary by the type of product and may be dictated by the method of incorporation into the index. For revolutionary products, it can be particularly critical to the accuracy of the index as there is a greater potential for introducing bias if these products are ignored. This is less likely to be the case for evolutionary goods.

1.139 Chapter 7 presents two alternative strategies that can be adopted when choosing a forced replacement, already mentioned in the previous paragraphs—replacing with the most similar product or replacing the missing item or variety with the currently most representative one. The strategy of replacement with the most popular is more likely to maintain the relevance of the sample by going through a process of resampling and identifying an appropriate replacement product. In order to make a properly judged decision on which strategy to follow and to inform the choice of replacement product and timing of introduction, the NSO needs to be aware of current consumer market trends, including what new products are becoming popular and what supplementary products are being introduced. It also needs to monitor product turnover, which can be an indication of the rate of product development associated with evolutionary products, and can vary between different categories of products. This information can be obtained from data gathered by price collectors and their supervisors, commodity experts in the NSO (for instance, working on the CPI or the producer price index) or from trade journals, products identified through scanner data, and consumer reports.

1.140 Evolutionary products should be included in the sample as soon as it is clear that consumers are shifting to these new products from the old versions. A frequent updating of the basket reduces the need for the ad hoc introduction of evolutionary products.

1.141 Revolutionary products usually first appear in the marketplace at a high initial price to cover development costs and to exploit the novelty value to the consumer. The prices generally start to decline as they become more established and competing varieties enter the market resulting in increased supply. The timing of introduction into the CPI basket is a critical issue—if introduced too late it will not only reduce the representativity of the CPI basket but could also give too much weight to any price decline associated with obsolescence of the product it is replacing when it nears the end of its life cycle but is still in the CPI basket. For revolutionary products the timing of their introduction into the CPI is important. In practice, often they are not introduced until they can be included in a new basket at the time of a CPI revision. This can lead to out-of-date and unrepresentative baskets if the revisions are carried out infrequently or with a long time lag, for instance as a result of delays in processing HBS data. But the price statistician is also confronted with uncertainty; it is not always clear how the retail market will react in the longer term to the introduction of a revolutionary product—some will be highly successful, achieving significant sales volumes and market stability in a relatively short time, while others may achieve high sales at an early stage which are not maintained.

1.142 Methods of introduction which overcome the problem of lack of timeliness include sample supplementation, targeted replacement procedures, and reinitiating (or rotating) the sample for the elementary aggregate or COICOP class. These methods are discussed in detail in Chapter 7 and, sample reinitiation apart, are generally applicable to evolutionary products. For revolutionary products, a new elementary aggregate must often be created. Frequent updating of the CPI basket reduces the potential problems and the introduction of revolutionary new products at the time of a basket update has a number of operational advantages: namely, the old weights do not need to be rescaled if a new product class is introduced when an old basket which does not include the new product class is still being used. Because it is not known exactly which other expenditures are being reduced relatively as the new product is purchased, this rescaling of weights is somewhat arbitrary and may lead to credibility issues.

Calculating the Consumer Price Index

1.143 The calculation of CPIs usually proceeds in two stages. First, price indices are estimated for the elementary expenditure aggregates, or simply elementary aggregates. Then these elementary price indices are averaged to obtain higher-level indices using the relative expenditure values of the elementary aggregates as weights.

Elementary Price Indices

1.144 The weights used in the CPI are generally derived from the HBS at levels that are typically for an item grouping such as cheese, butter, or milk. There is no identification of the specific variety of the product and an associated weight. NSOs select a sample of individual varieties to represent each item, but there are often no weights available at the variety level. The NSOs then use some method of averaging to produce an average price or an average price change to use in deriving the item or elementary index. This level of computation is usually referred to as an elementary aggregate because it is the first level at which an index is compiled for aggregation to higher levels of the CPI.

1.145 When weights are not available, the choice of the averaging method can be very important. Chapter 6 of Consumer Price Index Theory shows that the larger the variation in the individual prices, the larger the difference among the standard averaging methods. Both arithmetic and geometric averaging can be used, but as demonstrated in Chapter 8 of this Manual and Chapter 6 of Consumer Price Index Theory, geometric average formulas are recommended.

Arithmetic Average

1.146 The two methods used historically by NSOs to calculate the elementary indices are the ratio of average prices, known as the Dutot index, or the average of price relatives, known as the Carli index. Each of these formulas can be calculated using either the long-term price relative formula (comparing current to base period prices) or the short-term price relative formula (comparing current to previous period prices). The short-term versions of these formulas calculate a long-term relative change by chaining together the short-term price relatives. Appendix 6 illustrates the mathematical formulas for the Dutot and Carli indices. Chapter 8 provides details on the use of these formulas.

1.147 As shown in Chapter 8, it should be noted that the chained Carli produces different results than those for the fixed-base Carli using the average of long-term price relatives. The chained Carli price index has a definite upward bias. This chained version of the Carli index should not be used by NSOs for calculating elementary-level indices in the CPI.

Geometric Averages

1.148 With the introduction of the CPI Manual in 2004, a major emphasis was placed on using geometric averaging when weights are not available for the individual prices in the CPI elementary indices. The geometric price index is known as the Jevons price index and is calculated either as the ratio of the geometric average prices or as the geometric average of the price relatives. The Jevons formula can be found in Appendix 6, and Chapter 8 provides more detail on using the Jevons.

1.149 The Jevons index generally provides different estimates than both the Dutot or the Carli. Whether using the long-term price relative method or the short-term relative method Jevons indices yield the same index numbers as shown in Chapter 8. The same property holds true for the long-term and short-term Dutot indices. This property does not hold true for the Carli index. The chained short-term Carli index is always equal to or greater than the long-term Carli index.

1.150 Chapter 6 strongly encourages the use of the Jevons price index for calculating elementary indices where weights are unavailable. It notes that the Dutot price index should only be used in cases where the sample of transactions is homogeneous with respect to base price levels or price trends. It strongly discourages the use of the short-term Carli price index because of its known upward bias. The short-term method for the Jevons index will easily accommodate replacement varieties or adjustments for quality changes. As mentioned earlier, the NSO will only need to collect prices for the current and previous periods to enter in the system. If the long-term method is used, quality adjustments will involve changing the base price of the transaction for the value of the quality change as shown in Chapter 8.

Choice of Higher-Level Index Number Formula

1.151 The NSO has to decide on the kind of index number to use. The extensive references dealing with index theory in the bibliography of this Manual reflect the fact that there is a very large literature on this subject. Many kinds of mathematical formulas have been proposed over the past two centuries. While there may be no single formula that would be preferred in all circumstances, most economists and compilers of CPIs seem to agree that, in principle, the index formula should belong to a small class of indices called superlative indices. A superlative index may be expected to provide an approximation to a COLI. A characteristic feature of a superlative index is that it treats the prices and quantities in both periods being compared symmetrically. Different superlative indices tend to have similar properties, yield similar results, and behave in very similar ways. Because of their properties of symmetry, a superlative index is also likely to be seen as desirable even when the CPI is not meant to be a COLI.

1.152 When a monthly or quarterly CPI is first published, however, it is invariably the case that there is not sufficient information on the quantities and expenditures in the current period to make it possible to calculate a symmetric, or superlative, index. In cases where the prices and quantities are available, care must be taken to use index methods that do not result in biased estimates (for example, chain drift). Such methods are discussed in detail in Chapters 24 of Consumer Price Index Theory. While it is necessary to resort to second-best alternatives in practice, being able to make a rational choice between the various possibilities means having a clear idea of the target index that would be preferred in principle. The target index can influence practical matters such as whether the weights used in the index should be price updated.

1.153 A comprehensive, detailed, rigorous, and up-to-date discussion of the relevant index number theory is provided in Chapters 210 of Consumer Price Index Theory. Most of the standard indices used to compile CPIs (Laspey-res, Lowe, and Young) have known biases. The next paragraphs include summaries of the different formulas that can be used. Chapter 8 provides more details on these formulas that can be used to calculate the CPI.

Price Indices Based on Baskets of Goods and Services

1.154 The purpose of an index number may be explained as comparing the values of households’ expenditures on consumer goods and services in two time periods. Knowing that expenditures have increased by 5 percent is not very informative if one does not know how much of this change is attributable to changes in the prices of the goods and services, and how much to changes in the quantities purchased. The purpose of an index number is to decompose proportionate or percentage changes in value aggregates into their overall components of price and quantity change. A CPI is intended to measure the price component of the change in households’ consumption expenditures. One way to do this is to measure the change in the value of an aggregate, holding the quantities constant.

Lowe Indices

1.155 One very wide, and popular, class of price indices is obtained by defining the index as the relative change, between the periods compared, in the total cost of purchasing a given set of quantities, generally described as a “basket.” The meaning of such an index is easy to grasp and to explain to users. This class of index is called a Lowe index, after the index number pioneer who first proposed it in 1823 (see Chapter 2 of Consumer Price Index Theory). Most NSOs make use of some kind of Lowe index in practice. The Lowe index formula can be found in Appendix 6 and is described in more detail in Chapter 8. Lowe indices are widely used for CPI purposes.

1.156 In principle, any set of quantities could serve as the basket. The basket does not have to be restricted to the quantities purchased in one or other of the two periods compared, or indeed any actual period of time. The quantities could, for example, be arithmetic or geometric averages of the quantities in the two periods. For practical reasons, the basket of quantities used for CPI purposes usually has to be based on a survey of household consumption expenditures conducted in an earlier period than either of the two periods whose prices are compared. For example, a monthly CPI may run from January 2018 onward, with January 2018 = 100, but the quantities may be derived from an annual expenditure survey made in 2015 or 2016, or even spanning both years. As it takes a long time to collect and process expenditure data, there is usually a considerable time lag before such data can be introduced into the calculation of CPIs. The basket may also refer to a year, whereas the index may be compiled monthly or quarterly.

1.157 The index can be written, and calculated, in two ways: either as the ratio of two value aggregates, or as an arithmetic weighted average of the price ratios, or price relatives, pit/pi0 (where pit refers to the price of the item in the current period and pi0 refers to the price of the item in the reference period) for the individual products using the hybrid expenditure shares Si0b as weights. The expenditures are described as hybrid because the prices and quantities belong to two different time periods, 0 and b, respectively. The hybrid weight may be obtained by updating the actual expenditure shares in period b, namely, pibqib/Σpibqib, for the price changes occurring between periods b and 0 by multiplying them by the relative prices, pi0/pib.

Laspeyres and Paasche Indices

1.158 Any set of quantities could be used in a Lowe index, but there are two special cases which figure very prominently in the literature and are of considerable importance from a theoretical point of view. When the quantities are those of the price reference period, the Laspeyres index is obtained. When quantities are those of the other period, the Paasche index is obtained. Appendix 6 provides the mathematical formulas for the Laspeyres and Paasche indices.

Young Index

1.159 Instead of holding constant the quantities of period b, an NSO may calculate a CPI as an arithmetic weighted average of the individual price relatives, holding constant the expenditure shares of period b. The resulting index is called a Young index, again after another index number pioneer. The Young index formula can be found in Appendix 6 and is described in more detail in Chapter 8.

1.160 Whether to price update or not, and the resulting choice of index, is discussed in more detail in Chapter 9 (see also Chapter 3 of Consumer Price Index Theory).

Short-Term Price Index Formulas

1.161 Many countries use a modified version of the Lowe or Young index that compiles the index based on short-term price changes rather than the long-term price changes. This modified method can be compiled in two ways. Using the first approach, index compilation involves a two-step estimation process that breaks down the price movements into short-term, period-to-period changes that are used to bring forward the index from the previous period. In the second approach, elementary-level indices are compiled based on chained short-term price changes and the calculation of upper-level indices uses the base period weights. There is no preference as to which approach is preferred. Countries can decide which approach should be applied. The use of the short-term formula makes it easier for NSOs to introduce replacement items in the sample if the ones they have been tracking are no longer available. The short-term approach also enables the NSOs to make quality adjustments as improvement (or deterioration) is made to the sampled varieties. The NSO only needs to collect the current and previous prices for the item in order to introduce it into the index. In using the long-term method, the base price will need to be adjusted for the changes in the quality of the items in the sample. Chapter 8 provides the formulas for different approaches for the modified versions of the Lowe and Young indices.

Geometric Young, Laspeyres, and Paasche Indices

1.162 In the geometric version of the Young index, a weighted geometric average is taken of the elementary aggregate price relatives using the expenditure shares of period b as weights. The geometric Laspeyres is the special case in which b = 0; that is, the expenditure shares are those of the price reference period 0. Similarly, the geometric Paasche uses the expenditure shares of period t. It should be noted that these geometric indices cannot be expressed as the ratios of value aggregates in which the quantities are fixed. They are not basket indices, and there are no counterpart Lowe indices.

1.163 The geometric Young and Laspeyres indices have the same information requirements as their arithmetic counterparts. They can be produced on a timely basis. Thus, these geometric indices must be treated as serious practical possibilities for purposes of CPI calculations. As explained in Chapter 8, the geometric indices are less likely to be subject to different types of index number biases than their arithmetic counterparts. Their main disadvantage may be that, because they are not fixed-basket indices, they are not so easy to explain or justify to users.

Symmetric Indices

1.164 The standard price index methods used in most countries today, that is, the Lowe and Young indices, date back 90 years to those proposed by W. C. Mitchell (1927) and G. H. Knibbs (1924). Index number theory has advanced substantially, particularly in the past 30 years, to provide better information on what the target index number formula should be. Various approaches have been used to evaluate index number formulas and derive those best suited for inflation measures. The research presented in Consumer Price Index Theory has resulted in improvements for fixed-basket formulas and identified target indices that are symmetric averages of standard formulas. The target indices are the Fisher, Törnqvist, and Walsh price indices, discussed in detail in Chapter 8. However, these usually cannot be produced in final form except with a lag because they require both current and past weight information. Thus, most NSOs use the fixed-basket indices where the weight data are derived from some past period. An exception can occur in countries where scanner data are available and symmetric indices can be produced in real time using the methods presented in Chapter 10.

1.165 A symmetric index is one that makes equal use of the prices and quantities in both the periods compared and treats them in a symmetric manner. There are three symmetric indices that are widely used in economic statistics; these three indices are also superlative indices, the Fisher, Walsh, and Törnqvist. The formulas for each are provided in Appendix 6, and Chapter 8 provides details on using each of these index formulas.

1.166 Different formulas are used at different stages of aggregation. At the elementary or first stage, where prices are first combined to form an index, many countries will not use weights. At the second and higher levels, weights are applied, but these weights generally relate to some period in the past that becomes less representative with the passage of time. When compared to the target indices (Fisher, Walsh, or Törnqvist), it becomes apparent that the indices produced in practice are of substantially lower quality than the target indices. This Manual discusses these issues thoroughly and provides approaches that countries can implement over time to move closer to the target measures.

Updating the Weights and Linking of Series

1.167 Chapter 4 notes that over time the CPI weights and basket become less representative of the consumer market. The weights and basket should be updated at least once every five years to maintain their relevance. Many countries strive to conduct an HBS on a five-year cycle to use for updating the CPI basket and provide detailed information on household expenditures for use in the national accounts. From the HBS consumption estimates, NSOs will identify the most important items to use in the new CPI basket as discussed in Chapter 3. Deriving the new basket involves adding items that have gained importance since the previous HBS and deleting those that are no longer important based on their shares in the HBS.

Introducing New Weights and the Consumer Price Index Basket

1.168 The new basket is then used to review the sample of items and outlets to ensure that the samples are representative of products being purchased by consumers and the places in which they are purchased. The new weights and sample are used to start a new CPI with updated weight and price reference periods. Typically, the weight reference period precedes the price reference period. The NSO must decide how the new weights and sample will be introduced in the CPI. It also must decide on the price reference period, if it will differ from the weight reference period.

1.169 If the price reference period and weight reference period are the same (Laspeyres), then the new sample is used to compile the CPI directly. If the two periods differ, most likely with the weight period preceding the price reference period, there are two main options for updating the CPI weights. One approach is to price update the weights to the price reference period (Lowe index) to keep the implied quantities fixed at the weight reference period levels. The second approach is to keep the expenditure shares fixed (Young index).

1.170 Chapter 9 examines the different approaches to price updating the weights (paragraphs 9.6–9.10) and discusses the recent research on price updating (paragraphs 9.11–9.18). A key issue in the decision is whether there has been substantial price change between the weight reference period and the price reference period when the weights are introduced into the CPI. Generally, if there is a substantial price change between the price and weight reference periods, the weights should not be price updated. As noted previously, price updating the weights assumes that the quantities have remained fixed. If prices have changed substantially, this assumption is less likely to be true. It would be more realistic to assume the expenditure shares have remained fixed, in which case the price change is offset by a compensating quantity change.

1.171 For infrequently updated CPIs, a single year is preferred as the price reference period. Where a single month (or quarter) is used, the prices of some seasonal products will be unavailable or unusually high or low and many unusual or imputed prices may have to be used for the price reference period. For countries with infrequent weight updates, it is preferable that the price reference period is a whole year in which seasonal prices would be appropriately represented. In some months there will be no sales of, for example, seasonal fruit, but an average price for the whole year would still be available for the price reference period.

1.172 The index reference period should be one year. Using a single month or quarter to serve as the index reference period (= 100), can result in distorted index changes because of the unusual or imputed prices in that period.

1.173 Chained CPIs weights are updated on an annual basis. Because annual updating allows for a relatively small lag between the weight and price reference periods, a single month is used as the price reference period. There is a continuing flow of price data that may include imputations, and a relatively small number of changes in specifications or products; the major exercise is to introduce the new weights into the flow of price data and link this to the existing chain.

1.174 For infrequently rebased CPIs, the use of a single month as the price reference period is not advised; however, it is often the case that the country’s resources only allow for a price reference period of less than a full year. NSOs should strive to maximize the number of months used as the price reference period, with the goal of using a full year.

1.175 A primary shortcoming of using a short reference period for infrequently rebased CPIs is that out-of-season items in the price reference period will have no observed or economically meaningful price. The decision as to which month to use for the price reference month should consider when seasonal items with relatively high weights are in season. If these items are not in season, an imputed price will have to be used and consideration should be given to the validity of imputation methods for out-of-season items in this context. For example, if the carryforward method (which is discouraged) is used and the reference month uses an imputed price for the out-of-season item, the index may be unduly low. As mentioned previously, the two-stage Laspeyres aggregation is preferred since it avoids the need for long-term price comparisons with this one-month price reference period.

1.176 A primary shortcoming of using a longer price reference period are the resources needed to collect a full year of prices for all items included in the CPI basket. Some countries struggle with the resources to collect prices for two baskets (old and new) simultaneously. To minimize the burden, some countries have begun looking at preliminary expenditure data on a quarterly basis during the HBS collection period and to identify any new items that may be introduced into the basket. Once new products have been identified, prices can be collected.

Linking Previous Consumer Price Index to the New Price Index Reference Period

1.177 The NSO may choose to start the new series using the new price reference period as the new index reference period. In Chapter 9 the recommendation is that when new weights are introduced there should be an overlap period for the two indices so that they can be linked together. The overlap period is used to develop adjustment factors that may be applied to the old series to bring it to the same level as the new series. The linking of the old and new index series creates a continuous time series of data, which users need.

1.178 At minimum, a single common period is required as an overlap period between the indices. While a single overlap period can be used when the CPI is updated annually, this is not the preferred method when the index is updated less frequently. In the case of infrequent updates, an annual overlap is preferred. Some NSOs update the CPI weights each year so that the time lapse between the weight reference period and the link month is short. The single period link could be used in these instances. Chapter 9 provides a detailed discussion of annual weight updates.

1.179 Most NSOs will establish a new index reference period using an annual average from a previous year. The simplest and easiest method for users is to link the series with data for the month preceding the introduction of the new series (link month). This involves re-referencing the old series at each level to the annual average index for the new price reference period. However, there will be a discontinuity between the index level for the new index and that for the re-referenced index level for the old series in the link month. This reflects the difference in price trends between the old and new series as the new weights are being introduced. There are three steps involved to link this difference:

  • Re-reference the old index series to the new index reference period

  • Compile the new index series in the link month using the new weight structure

  • Link the new series to the old series by using forward linking factors or, if using the short-term price relative method, start the new series indices at the level of the old series in the link month

1.180 If the NSO or users want to continue the old CPI series for future periods in time, they can produce a set of forward linking factors to use in future months as the new CPI is released. The forward linking factor raises the level of the new CPI series to that of the old series thus keeping the series on the old reference period.

Organization and Management

1.181 The production of the CPI is a complex operation involving extensive fieldwork by data collectors; processing, review, and editing of the collected data; compilation of indices; and their dissemination to the public. The whole process requires careful planning and management to ensure that the data products conform to good management and statistical practice. Appropriate management procedures for the CPI are described in Chapter 13.

1.182 Price collectors should be well trained to ensure that they understand the importance of selecting the right products for pricing. Inevitably, price collectors are bound to use their own discretion to a considerable extent. As already explained, one issue of crucial importance to the quality and reliability of a CPI is how to deal with the slowly evolving set of products with which a price collector is confronted. Products may disappear and have to be replaced by others, but it may also be appropriate to drop some products before they disappear altogether if they have become unrepresentative. Price collectors need to be provided with appropriate training and very clear instructions and documentation about how to proceed. Clear instructions are also needed to ensure that price collectors collect the right prices when there are sales, special offers, or other exceptional circumstances.

1.183 The price data collected must also be subjected to careful checking and editing. Many checks can be carried out automatically, using standard statistical control methods. It may also be useful to send out auditors to accompany price collectors and monitor their work. The various possible checks and controls are explained in detail in Chapter 13.

1.184 The head office staff also need to be trained on index methods and procedures for review of collected data, imputation of missing data, quality-adjustment procedures when needed, as well as index compilation and dissemination processes. Improvements in information technology should obviously be fully exploited. New and more efficient computers and database applications are continuously being developed. As resources permit, new technologies and organization improvements should be implemented.

1.185 Staff training and process reviews are an essential part of continuous quality improvement. Staff should receive regular training in their discipline and should be invited to operational reviews where all team members can raise concerns and, where appropriate, tackle specific issues through individual or group training.

Publication and Dissemination

1.186 As noted previously and in Chapter 2, the CPI is an important statistic whose movements can influence the central bank’s monetary policy, fiscal policy, and the national budget; affect stock markets; influence wage rates and social security payments; and so on. There must be public confidence in its reliability, and in the competence and integrity of those responsible for its compilation. The methods used to compile it must therefore be fully documented, transparent, and open to public scrutiny. Many countries have an official CPI advisory group consisting of both experts and users. The role of such a group is not just to advise the NSO on technical matters, but also to promote public confidence in the index.

1.187 Users of the index also attach great importance to having the index published as soon as possible after the end of each month or quarter, preferably within two or three weeks. There are also many users who do not wish the index to be revised once it has been published. Thus, there is likely to be some trade-off between the timeliness and the quality of the index.

1.188 Publication should be understood to mean the dissemination of the results in any form. Most countries do not release their CPI in print, or hard copy. NSOs now tend to release the CPI electronically and make it available through the internet on their website.

1.189 As explained in Chapter 14, good publication policy goes beyond timeliness, confidence, and transparency. The results should be made available to all users, in both the public and the private sectors, at the same time and according to a publication schedule announced in advance. There should be no discrimination among users in the timing of the release of the results. The results should not be subject to governmental scrutiny as a condition for their release and should be seen to be free from political or other pressures.

1.190 There are many decisions to be taken about the degree of detail in the published data and the different ways in which the results may be presented. Users need to be consulted about these questions. These issues are discussed in more detail in Chapter 14; however, it is recommended to provide users with detailed data that are presented in a long time series. Detailed indices should include detailed item and area indices.

Special Cases

1.191 Certain products and issues have proven to be challenging for compilers with regard to both developing weights and collecting prices. Chapter 11 focuses on selected special cases and provides detailed advice for some of the more problematic products and issues compilers face. These include the treatment of seasonal items, housing, internet purchases, second-hand goods, own-account production, tariffs, transport services, health, education, social protection, and financial services.

1.192 Wherever possible, the chapter identifies the preferred approach for the treatment of each special case; however, currently there is no preferred approach for the treatment of owner-occupied housing services. The section on housing in Chapter 11 identifies the different possible methods for the treatment of owner-occupied housing and describes the advantages and disadvantages of each method.

Errors and Biases

1.193 The CPI, like all other statistics, may be subject to general error that may occur during any stage of the estimation process but also errors that are unique to the CPI (for example, substitution bias and quality change bias). Chapter 12 describes not only the different types and sources of potential errors, but also potential biases and their sources. Finally, Chapter 12 provides insight into how to address these errors and biases.

Annex 1.1 Formula Notations

1

NSO is used to describe the compiling agency in this Manual regardless of the institutional setting in a country.

2

See Chapter 4, Consumer Price Index Theory, and Appendix 5, International Comparison Program.

3

The System of National Accounts 2008 (2008 SNA, paragraph 6.69) formally defines purchasers’ prices. According to the 2008 SNA, the purchaser’s price is the amount paid by the purchaser, excluding any value-added tax or similar tax deductible by the purchaser, in order to take delivery of a unit of a good or service at the time and place required by the purchaser. The purchaser’s price of a good includes any transport charges paid separately by the purchaser to take delivery at the required time and place.

4

This assumes the national accounts have independent expenditure side estimates, and household final consumption expenditure is not estimated as a residual.

5

A more appropriate alternative might be number of workers employed or aggregated retail turnover in the location if this is known. This may be the case where most shopping is done in town centers but with most people living in residential areas in the suburbs.

6

See the ICP 2003–2006 Handbook on the World Bank ICP website at https://www.worldbank.org/en/programs/icp/brief/handbooks-and-operational-guides.

7

In constructing a CPI, consumer valuations of products should be decisive, and the cost of production is irrelevant in this context. In “equilibrium” situations, the cost of production (using a “normal” cost of capital in the producer’s user cost formula) should give a close approximation to the selling price (and hence the consumer’s valuation of the product). But it is in disequilibrium situations, involving new products, that research and development has to be amortized using monopolistic prices where new products and obsolescence of old products is likely to occur that the assumptions of equilibrium break down.

8

Jack Triplett. “Handbook on Hedonic Indexes and Quality Adjustments in Price Indexes: Special Application to Information Technology Products.” STI Working Paper 2004/9, Statistical Analysis of Science, Technology and Industry; OECD Publications, Rue André-Pascal, 75775 Paris, Cedex 16, France. http://www.oecd.org/dataoecd/37/31/33789552.pdf.

9

For the argument to be valid it has to be assumed that the consumers are rational and well informed about available alternative choices. It also has to be assumed that the market is “in a state of equilibrium,” that is in a state that would persist indefinitely if all given conditions remained unchanged. In practice, these assumptions may be adequate or inadequate to varying degrees in different product areas. Equilibrium conditions may be disrupted by temporary imbalances between supply and demand, such as stock clearances, bad harvests, or scarce, much-demanded new models.

10

Clothing items usually require additional procedures to control substitutions depending on the season of the year (for example, autumn/winter versus spring/summer clothing).

Author: Brian Graf
  • Abe, N., and K. Shinozaki. 2018. “Compilation of Experimental Price Indexes Using Big Data and Machine Learning: A Comparative Analysis and Validity Verification.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 7–9. https://www.unece.org.

    • Search Google Scholar
    • Export Citation
  • Abraham, K. G., J. S. Greenlees, and B. R. Moulton. 1998. “Working to Improve the Consumer Price Index.” Journal of Economic Perspectives 12 (1): 2736.

    • Search Google Scholar
    • Export Citation
  • Aczél, J. 1987. A Short Course on Functional Equations. Dordrecht: Reidel Publishing Co.

  • Advisory Commission to Study the Consumer Price Index. 1995. Toward a More Accurate Measure of the Cost of Living. Interim Report to the Senate Finance Committee, September 15. Washington, DC.

    • Search Google Scholar
    • Export Citation
  • Aizcorbe, A. M., and P. C. Jackman. 1993. “The Commodity Substitution Effect in CPI Data, 1982–91.” Monthly Labor Review 116 (12): 2533.

    • Search Google Scholar
    • Export Citation
  • Aizcorbe, A., C. Corrado, and M. Doms. 2001. Constructing Price and Quantity Indexes for High Technology Goods. Industrial Output Section, Division of Research and Statistics. Washington, DC: Board of Governors of the Federal Reserve System.

    • Search Google Scholar
    • Export Citation
  • Aizcorbe, A., C. Corrado, and M. Doms. 2003. “When Do Matched-Model and Hedonic Techniques Yield Similar Price Measures?Working Paper 2003–14, Federal Reserve Bank of San Francisco, San Francisco, CA.

    • Search Google Scholar
    • Export Citation
  • Alterman, W. F., W. E. Diewert, and R. C. Feenstra. 1999. International Trade Price Indexes and Seasonal Commodities. Washington, DC: Bureau of Labor Statistics.

    • Search Google Scholar
    • Export Citation
  • Anderson, R. G., B. E. Jones, and T. Nesmith. 1997. “Building New Monetary Services Indexes: Concepts, Data and Methods.” Federal Reserve Bank of St. Louis Review 79 (1): 5383.

    • Search Google Scholar
    • Export Citation
  • Arguea, N. M., C. Haseo, and G. A. Taylor. 1994. “Estimating Consumer Preferences Using Market Data: An Application to U.S. Automobile Demand.” Journal of Applied Econometrics 9: 118.

    • Search Google Scholar
    • Export Citation
  • Armknecht, P. A. 1996. “Improving the Efficiency of the U.S. CPI.” Working Paper No. 96/103, International Monetary Fund, Washington, DC.

    • Search Google Scholar
    • Export Citation
  • Armknecht, P. A. 2016. “Fixed Basket Methods for Compiling Consumer Price Indexes.” Meeting of the Group of Experts on Consumer Price Indices, Geneva, Switzerland, May 2–4, 2016. http://www.unece.org/index.php.

    • Search Google Scholar
    • Export Citation
  • Armknecht, P. A., and D. Weyback. 1989. “Adjustments for Quality Change in the U.S. Consumer Price Index.” Journal of Official Statistics 5 (2): 10723.

    • Search Google Scholar
    • Export Citation
  • Armknecht, P. A., and F. Maitland-Smith. 1999. “Price Imputation and Other Techniques for Dealing with Missing Observations, Seasonality and Quality Change in Price Indices.” Working Paper No. 99/78, International Monetary Fund, Washington, DC. http://0-www-imf-org.library.svsu.edu/external/pubs/ft/wp/1999/wp9978.pdf

    • Search Google Scholar
    • Export Citation
  • Armknecht, P. A., and M. Silver. 2014. “Post-Laspeyres: The Case for a New Formula for Compiling Consumer Price Indexes.” The Review of Income and Wealth 60 (2): 22544.

    • Search Google Scholar
    • Export Citation
  • Armknecht, P. A., W. F. Lane, and K. J. Stewart. (1997) 2003. “New Products and the U.S. Consumer Price Index.” In Scanner Data and Price Indexes, NBER Studies in Income and Wealth, edited by R. C. Feenstra and M. D. Shapiro. Chicago, IL: University of Chicago Press, pp. 37591.

    • Search Google Scholar
    • Export Citation
  • Arrow, K. J., H. B. Chenery, B. S. Minhas, and R. M. Solow. 1961. “Capital–Labor Substitution and Economic Efficiency.” Review of Economics and Statistics 63: 22550.

    • Search Google Scholar
    • Export Citation
  • Astin, J. 1999. “The European Union Harmonized Indices of Consumer Prices (HICP).” in Proceedings of the Ottawa Group Fifth Meeting, Reykjavik, Iceland, August 25–27, edited by R. Gudnason and T. Gylfadottir. Reykjavík: Statistics Iceland; also published in Statistical Journal of the United Nations ECE16: 12335. http://www.statcan.ca/secure/english/ottawagroup/.

    • Search Google Scholar
    • Export Citation
  • Astin, J. 2001. “New Uses of PPPs within the European Union.” Paper presented at the Joint World Bank–OECD Seminar on Purchasing Power Parities: Recent Advances in Methods and Applications, Washington, DC, January 30–February 2.

    • Search Google Scholar
    • Export Citation
  • Aten, B., R. Summers, and A. Heston. 2001. “An Explanation of Stability in Country Price Structures: Implications for Spatial– Temporal Comparisons.” Paper presented at the Joint World Bank–OECD Seminar on Purchasing Power Parities: Recent Advances in Methods and Applications, Washington, DC, January 30–February 2.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 1997. “An Analytical Framework for Price Indexes in Australia.” Information Paper, Catalogue No. 6421.0. http://www.abs.gov.au.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 2003. “Australian Consumer Price Index: Concepts, Sources and Methods.” Catalogue No. 6461.0. http://www.abs.gov.au/.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 2009. “Consumer Price Index: Concepts, Sources and Methods.” Catalogue No. 6461.0, ABS, Canberra. http://www.abs.gov.au/ausstats/abs@.nsf/mf/6461.0.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 2013. “Feature Article: The Use of Transactions Data to Compile the Australian Consumer Price Index.” Catalogue No. 6401.0, ABS, Canberra. http://www.abs.gov.au/.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 2015. “Enhancing the Australian CPI: A Roadmap.” Catalogue No. 6401.0.60.001, ABS, Canberra. http://www.abs.gov.au/.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 2016a. “Making Greater Use of Transactions Data to Compile the Consumer Price Index.” Information Paper 6401.0.60.003, ABS, Canberra. http://www.abs.gov.au/.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 2016b. “Measuring Price Change of Attached Dwellings in the CPI.” Feature Article, ABS, Canberra. http://www.abs.gov.au/.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 2017a. “An Implementation Plan to Maximise the Use of Transactions Data in the CPI.” Information Paper 6401.0.60.004, ABS, Canberra. http://www.abs.gov.au/.

    • Search Google Scholar
    • Export Citation
  • Australian Bureau of Statistics. 2017b. “Introduction of the 17th Series Australian Consumer Price Index.” Information Paper 6470.0.55.001, ABS, Canberra. http://www.abs.gov.au/AUS-STATS/abs@.nsf/DetailsPage/6470.0.55.0012017.

    • Search Google Scholar
    • Export Citation
  • Bäckström, Peter, and Muhanad Sammar. 2012. “The Use of Superlative Index Links in the Swedish CPI.” Paper presented at the meeting of the Group of Experts on CPI, Geneva. https://www.unece.org/index.php.

    • Search Google Scholar
    • Export Citation
  • Bailey, M. J., R. F. Muth, and H. O. Nourse. 1963. “A Regression Method for Real Estate Price Construction.” Journal of the American Statistical Association 58: 93342.

    • Search Google Scholar
    • Export Citation
  • Baker, D. 1998. “Does the CPI Overstate Inflation? An Analysis of the Boskin Commission Report.” In Getting Prices Right, edited by D. Baker. Washington, DC: Economic Policy Institute, pp. 79155.

    • Search Google Scholar
    • Export Citation
  • Baldwin, A. 1990. “Seasonal Baskets in Consumer Price Indexes.” Journal of Official Statistics 6 (3): 25173.

  • Baldwin, A., A. Nakamura, and M. Prudhomme. 2010. “An Empirical Analysis of the Different Concepts for Owned Accommodation in the Canadian CPI: The Case of Ottawa, 1996–2005.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 10–12. https://www.unece.org.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 1980a. “Seasonal Products in Agriculture and Horticulture and Methods for Computing Price Indices.” Statistical Studies No. 24. The Hague: Netherlands Central Bureau of Statistics.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 1980b. “Seasonal Commodities and the Construction of Annual and Monthly Price Indexes.” Statistische Hefte 21: 1106.

  • Balk, B. M. 1980c. “A Method for Constructing Price Indices for Seasonal Commodities.” The Journal of the Royal Statistical Society Series A 143: 6875.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 1981. “A Simple Method for Constructing Price Indices for Seasonal Commodities.” Statistische Hefte 22: 728.

  • Balk, B. M. 1983. “Does There Exist a Relation between Inflation and Relative Price Change Variability? The Effect of the Aggregation Level.” Economic Letters 13: 173180.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 1985. “A Simple Characterization of Fisher’s Price Index.” Statistische Hefte 26: 5963.

  • Balk, B. M. 1989a. “Changing Consumer Preferences and the Cost of Living Index: Theory and Nonparametric Expressions.” Zeitschrift für Nationalökonomie 50 (2): 15769.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 1989b. “On Calculating the Precision of Consumer Price Indices.” Contributed Papers 47th Session of the ISI. Paris: International Statistical Institute.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 1990. “On Calculating Cost-of-Living Index Numbers for Arbitrary Income Levels.” Econometrica 58 (1): 7592.

  • Balk, B. M. 1994. “On the First Step in the Calculation of a Consumer Price Index.” Paper presented at First Meeting of the International Working Group on Price Indices, Ottawa, October 31–November 4. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 1995. “Axiomatic Price Index Theory: A Survey.” International Statistical Review 63: 6993.

  • Balk, B. M. 1996a. “A Comparison of Then Methods for Multilateral International Price and Volume Comparisons.” Journal of Official Statistics 12: 199222.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 1996b. “Consistency in Aggregation and Stuvel Indices.” The Review of Income and Wealth 42: 35363.

  • Balk, B. M. 1998a. Industrial Price, Quantity and Productivity Indices. Boston, MA: Kluwer Academic Publishers.

  • Balk, B. M. 1998b. “On the Use of Unit Value Indices as Consumer Price Subindices.” Paper presented at the Fourth Meeting of the International Working Group on Price Indices, Washington, DC, April 22–24. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 2000a. “Divisia Price and Quantity Indexes 75 Years After.” Draft Paper, Department of Statistical Methods, Statistics Netherlands, Voorburg.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 2000b. “On Curing the CPI’s Substitution and New Goods Bias.” Research Paper 0005, Department of Statistical Methods, Statistics Netherlands, Voorburg.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 2001. Aggregation Methods in International Comparisons: What Have We Learned? Report Series Research in Management ERS-2001–41-MKT, Erasmus Research Institute of Management. Rotterdam: Erasmus University.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 2005. “Price Indexes for Elementary Aggregates: The Sampling Approach.” Journal of Official Statistics 21: 67599.

  • Balk, B. M. 2008. Price and Quantity Index Numbers. Cambridge: University Press.

  • Balk, B. M. 2008. Price and Quantity Index Numbers: Models for Measuring Aggregate Change and Difference. New York: Cambridge University Press.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M. 2010. “Lowe and Cobb-Douglas Consumer Price Indexes and Their Substitution Bias.” Jahrbücher für Nation-alökonomie und Statistik (The Journal of Economics and Statistics) 230 (6): 72640.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M., and H. M. P. Kersten. 1986. “On the Precision of Consumer Price Indices Caused by the Sampling Variability of Budget Surveys.” Journal of Economic and Social Measurement 14: 1935.

    • Search Google Scholar
    • Export Citation
  • Balk, B. M., and W. E. Diewert. 2001. “A Characterization of the Törnqvist Price Index.” Economics Letters 73: 27981.

  • Ball, A., and D. Fenwick. 2004. “Costing Mobile Telephone Calls: The Use of Constrained User Profiles.” Paper presented at the Eighth Meeting of the International Working Group on Price Indices, Helsinki, August 23–25. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Bartik, T. J. 1988. “Measuring the Benefits of Land Improvements in Hedonic Models.” Land Economics 64 (2): 17283.

  • Bas, C. 2018. “Different Approaches and Methods for Strong Seasonal Items in the CPI.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 7–9. https://www.unece.org.

    • Search Google Scholar
    • Export Citation
  • Bascher, J., and T. Lacroix. 1999. “Dishwashers and PCs in the French CPI: Hedonic Modeling, from Design to Practice.” Paper presented at the Fifth Meeting of the International Working Group on Price Indices, Reykjavik, August 25–27. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Baxter, M. (ed.). 1998. The Retail Prices Index. Technical Manual. London: Office for National Statistics, UK.

  • Bean, L. H., and O. C. Stine. 1924. “Four Types of Index Numbers of Farm Prices.” Journal of the American Statistical Association 19: 305.

    • Search Google Scholar
    • Export Citation
  • Becker, G. S. 1965. “A Theory of the Allocation of Time.” Economic Journal 75: 493517.

  • Beck-MacNeil, R. 2018. “Pricing the Internet—The Internet Access Services Component of the Canadian CPI.” Statistics Canada. Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 7–9. https://www.unece.org.

    • Search Google Scholar
    • Export Citation
  • Beidelman, C. 1973. Valuation of Used Capital Assets. Sarasota, FL: American Accounting Association.

  • Beidelman, C. 1976. “Economic Depreciation in a Capital Goods Industry.” National Tax Journal 29: 37990.

  • Beisteiner, A. 2008. “Optimal Allocation of Prices in Practice— Using the Neumann Formula as Tool.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 8–9. https://www.unece.org.

    • Search Google Scholar
    • Export Citation
  • Bentley, A., and F. Krsinich. 2017. “Towards a Big Data CPI for New Zealand.” Paper presented at the Fifteen Meeting of the International Working Group on Price Indices, Eltville am Rhein, May 10–12. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Bernardini, A., C. De Vitiis, A. Guandalini, F. Inglese, and M. D. Terribili. 2016. “Measuring Inflation Through Different Sampling Designs Implemented on Scanner Data.” Paper presented at Meeting of the Group of Experts on Consumer Price Indices, Geneva, Switzerland, May 2–4.

    • Search Google Scholar
    • Export Citation
  • Berndt, E. R. 1991. The Practice of Econometrics: Classic and Contemporary. Reading, MA: Addison-Wesley.

  • Berndt, E. R., D. Ling, and M. K. Kyle. 2003. “The Long Shadow of Patent Expiration: Generic Entry and Rx to OTC Switches.” In Scanner Data and Price Indexes, NBER Studies in Income and Wealth, edited by M. Shapiro and R. C. Feenstra. Chicago, IL: University of Chicago Press, pp. 22973.

    • Search Google Scholar
    • Export Citation
  • Berndt, E. R., L. T. Bui, D. H. Lucking-Reiley, and G. L. Urban. 1997. “The Roles of Marketing, Product Quality and Price Competition in the Growth and Composition of the U.S. Anti-Ulcer Drug Industry.” In The Economics of New Goods, NBER Studies in Income and Wealth, edited by T. Bresnahan and R. J. Gordon. Chicago and London: University of Chicago Press, pp. 23277.

    • Search Google Scholar
    • Export Citation
  • Berndt, E. R., Z. Griliches, and N. J. Rappaport. 1995. “Econometric Estimates of Price Indexes for Personal Computers in the 1990s.” Journal of Econometrics 68: 24368.

    • Search Google Scholar
    • Export Citation
  • Berry, S., J. Levinsohn, and A. Pakes. 1995. “Automobile Prices in Market Equilibrium.” Econometrica 63 (4): 841–90; also published as NBER Working Paper No. W4264, July 1996. http://www.nber.org.

    • Search Google Scholar
    • Export Citation
  • Beuerlein, I. 2001. “The German Consumer Price Index for Telecommunication Services: A User Profile Approach for Mobile Technology and Internet Access.” Paper presented at the Sixth Meeting of the International Working Group on Price Indices, Canberra, April 2–6. http://www.ottawagroup.org/.

    • Search Google Scholar
    • Export Citation
  • Bhardwaj, H., T. Flower, P. Lee, and M. Mayhew. 2017. Research Indices Using Web Scraped Price Data. August 2017 Update, UK Office for National Statistics, London.

    • Search Google Scholar
    • Export Citation
  • Bird, D., R. Breton, C. Payne, and A. Restieaux. 2014. Initial Report on Experiences with Scanner Data in ONS. Report, Office for National Statistics, Newport, UK.

    • Search Google Scholar
    • Export Citation
  • Bode, B., and van J. Dalén. 2001. “Quality-Corrected Price Indexes of New Passenger Cars in the Netherlands, 1990–1999.” Paper presented at the Sixth Meeting of the International Working Group on Price Indices, Canberra, April 2–6. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Boettcher, I. 2015. “Automatic Data Collection on the Internet (Web Scraping).” Statistic Austria. Paper presented at the Fourteen Meeting of the International Working Group on Price Indices, Tokyo, May 20–22. http://www.ottawagroup.org.

    • Search Google Scholar
    • Export Citation
  • Böhm-Bawerk, E. V. 1891. The Positive Theory of Capital. Translated from the original German edition of 1888 by W. Smart. New York: G. E. Stechert.

    • Search Google Scholar
    • Export Citation
  • Bortkiewicz, L. V. 1923. “Zweck und Struktur einer Preisin-dexzahl.” Nordisk Statistisk Tidsskrift 2: 369408.

  • Boskin, M. J. (Chair), E. R. Dullberger, R. J. Gordon, Z. Griliches, and D. W. Jorgenson. 1996. Final Report of the Commission to Study the Consumer Price Index, U.S. Senate, Committee on Finance. Washington, DC: U.S. Government Printing Office.

    • Search Google Scholar
    • Export Citation
  • Boskin, M. J. (Chair), E. R. Dullberger, R. J. Gordon, Z. Griliches, and D. W. Jorgenson. 1998. “Consumer Prices in the Consumer Price Index and the Cost of Living.” Journal of Economic Perspectives 12 (1): 326.

    • Search Google Scholar
    • Export Citation
  • Böttcher, I., and S. Sergeev. 2014. “Austrian Scanner Data Project—Report Multipurpose Consumer Price Statistics: The Use of Scanner Data.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 26–8. https://www.unece.org.

    • Search Google Scholar
    • Export Citation
  • Bowley, A. L. 1899. “Wages, Nominal and Real.” In Dictionary of Political Economy, Volume 3, edited by R. H. I. Palgrave. London: Macmillan, pp. 64051.

    • Search Google Scholar
    • Export Citation
  • Bowley, A. L. 1901. Elements of Statistics. Westminster: Orchard House.