3.1 A consumer price index (CPI) is usually calculated as a weighted average of the relative price changes of the goods and services covered by the index. The weights attached to each good or service reflect their relative importance as measured by their shares in the total consumption of all households. The weight determines the impact that its price change will have on the overall index. The weights should be made publicly available for the information of data users, to ensure public confidence in the index and enhance transparency. The International Monetary Fund maintains a CPI database that includes detailed weight data.1
3.1 A consumer price index (CPI) is usually calculated as a weighted average of the relative price changes of the goods and services covered by the index. The weights attached to each good or service reflect their relative importance as measured by their shares in the total consumption of all households. The weight determines the impact that its price change will have on the overall index. The weights should be made publicly available for the information of data users, to ensure public confidence in the index and enhance transparency. The International Monetary Fund maintains a CPI database that includes detailed weight data.1
3.2 Because the weights assigned to the different goods and services in the basket influence changes in the CPI, the accuracy and reliability of CPI estimates depend upon the quality of the weights used. Therefore, it is necessary to have weights that reflect, as closely as possible, current consumer expenditure patterns.
3.3Chapter 3 discusses what the CPI weights should represent and how they can be derived. First, this chapter provides an overview of the conceptual basis of the weights to clarify what expenditure should be included when developing weights. Next, the main data sources for the weights are introduced and the practical steps that must be completed when deriving weights are discussed. Finally, some special cases are discussed at the end of this chapter.
Conceptual Basis of the Weights
3.4 The expenditure weights used in a CPI have to be consistent with the conceptual framework of the index as discussed in Chapter 2. Expenditure data can be obtained from different data sources with household budget surveys (HBSs) being the most common. When exploiting these data sources, the scope and the concept of the index will determine which goods and services should be covered in the CPI weights. If countries compile supplementary or alternative CPIs to meet multiple user needs, separate weighting structures would then be estimated, reflecting the different concepts or intended uses of the index. The main conceptual aspects that must be considered when deriving weights are discussed in paragraphs 3.5–3.18.
Geographical Coverage: National versus Domestic Concept
3.5 The geographical coverage of a CPI may follow either the “national” concept or the “domestic” concept. The national concept measures price changes experienced by resident households, regardless of whether expenditure is made within the country or abroad. Under the domestic concept, the scope of the CPI (with regard to both prices and weights) considers the economic territory and includes expenditure of both residents and nonresidents.
3.6 An HBS generally identifies all relevant expenditures made by resident households and may make a distinction between expenditure made within the economic territory and abroad. If the main purpose of the index is to measure price changes experienced by resident households, the weights should, in principle, include their expenditure abroad. If, however, the aim is to include the expenditure made by foreign visitors, thus reflecting all purchases of consumer goods and services made by resident or nonresident households within the country, sources other than the HBS must be used. An HBS does not cover expenditure made by nonresident households.
3.7 For practical reasons, even if the weights cover the expenditure made both at home and abroad, prices may be collected only for those goods and services acquired within the economic territory of the country. Such an approach assumes that the price changes of the goods and services acquired abroad are similar to the price changes for the same goods and services acquired at home. Alternatively, it may be possible to use CPI subindices compiled by the respective countries to measure the price changes of the goods and services acquired abroad.
3.8 In principle, the weights should represent the whole country and all regions should be covered. Separate expenditure weights can also be derived for each region, in which case the expenditures must be sufficiently representative at the regional level. This is especially important if the expenditure pattern differs between regions. For instance, it is common that the consumption habits vary between urban and rural areas. The same principle applies to harmonized CPIs, covering a group of countries, where each country can be regarded as a “region” with its own national weights.
3.9 The discussion of national and domestic concepts applies to regional subdivisions. It can sometimes be the case that a household lives in one region but does most of its purchases in an adjacent region, particularly if it lives close to a regional border. Practical issues dictate whether the expenditure weights (and the prices) should be allocated to the region of expenditure or the region of residence. In any case, treatment should be consistent across all regions to avoid missing or double counting parts of household expenditure.
3.10 The target or reference populations will be defined based on the main purpose and use of the index, as described in Chapter 2. In principle, all types of households should be covered, irrespective of their income or other socioeconomic factors. If any income groups, types of households, or geographic areas are excluded, for example, for cost or practical considerations, this should be explicitly stated in the index metadata. In some countries, the wealthiest households are excluded because their expenditure may be atypical or the HBS information may be less reliable. Other countries may exclude the expenditures of the very poor for the same reason. Another practice is to compile a CPI which excludes both extremes of the income distribution. If the primary use of the CPI is for adjusting incomes of a certain subgroup of the population for increases in the cost of living, then such subgroup may be the appropriate target population.
3.11 The weights may or may not include the expenditure made by people living in institutional households. Institutional households refer to people living permanently in an institution or who may be expected to reside in an institution for a very long time, as described in Chapter 2. Many countries exclude expenditure of such households in their CPI because of the difficulty of obtaining reliable expenditure information, or because the expenditure associated with such households is unlikely to be significant in comparison with private households. The choice to include or exclude the expenditure made by institutional households may depend on the main source for the weights. While weights primarily based on national accounts household final consumption expenditure (HFCE) data may include consumption of institutional households, the weights primarily based on an HBS would exclude consumption expenditures of institutional households. In considering the practical challenges relating to the inclusion of institutional households in a CPI, two questions need to be asked. First, is the expenditure pattern of institutional households likely to be significantly different from private households? Second, even if the answer is yes, would their exclusion from the CPI be likely to significantly affect the national CPI?
Monetary and Nonmonetary Transactions
3.12 A portion of HFCE consists of nonmonetary transactions, such as expenditure on goods and services produced for own account and remuneration in kind. Furthermore, as described in Chapter 2, the broader concept of household actual final consumption includes goods and services provided without charge or subsidized by governments and nonprofit institutions serving households. Depending on the main use of the CPI, a decision must be made whether to include nonmonetary transactions in the weights or if it is more appropriate to limit the scope of the index to monetary transactions only. Production for own consumption is treated by some countries as within the scope of the CPI, and in other countries as out of scope. In some countries, own-account production constitutes a significant portion of HFCE. It can be argued that while it is part of gross domestic product and should therefore be included to improve consistency with national accounts, and especially for producing deflators, it is not necessarily appropriate for a CPI used as a general measure of inflation or for indexation where the narrowest concept of consumption, based on monetary transactions, is used. If the CPI includes own-account production, the weights should include a valuation of the physical quantities of such products, the latter often derived from the HBS.
Expenditures That Are Out of Scope
3.13 In its role as a measure of total consumer inflation, the CPI should, in principle, cover all types of goods and services that are consumed by the reference population. Some types of products may be excluded for practical reasons. These may include products that are illegal, black market sales, gambling, or prostitution. Some products may also be excluded because of policy reasons. For instance, it can be decided to exclude certain categories of goods or services for purposes of compiling alternative aggregations or for analysis (for example, exclude tobacco from an index that is used for a specific purpose). Whenever certain in-scope goods or services have been excluded from the index, this should be clearly documented and explained to users.
3.14 Only household consumption expenditure is relevant for the construction of CPI weights. As explained in Chapter 2, expenditure on assets such as works of art, financial investment (as distinct from financial services), payments of social security contributions, fines or income taxes, interest payments, or repayments of debts are not considered to be household consumption expenditure and should be excluded from the coverage of the weights and the index.
3.15 Business-related expenditure is explicitly excluded from the scope of a CPI. Consequently, this expenditure must also be excluded from the CPI weights. Households may own unincorporated or informal enterprises, whereby expenditure for some products is used partly for business purposes and partly for final consumption. In principle, only the portion used for final consumption should be included in the CPI weights.
Democratic and Plutocratic Weights
3.16 The use of aggregated expenditure to derive CPI weights reflects the principle that each household contributes to the weights with an amount proportional to its expenditure. This is referred to as plutocratic weighting and means that the expenditure patterns of high-spending households have more influence on the index. The use of plutocratic weights is generally considered more appropriate particularly for CPIs which have been constructed to be a general measure of inflation, for national accounts deflation, and as the basis for monetary policymaking. In principle, it is also possible to derive democratic weights, where each household is given equal weight. While democratic weights may be considered appropriate for an index used to reflect the consumer inflation experience of the “typical” or “average” household, they are rarely used in practice. If all households have similar expenditure patterns, the democratic and plutocratic approaches lead to similar results. However, most often the expenditure pattern depends on the total level of expenditure of a household. There can be significant differences between both approaches, especially if the distribution of household expenditure is unequal. Table 3.1 presents an example. The lower-spending household (household 1) spends in relative terms more on food than the higher-spending household (household 2). Consequently, the weight for food is higher in the democratic approach than in the plutocratic approach.
3.17 In Table 3.1, plutocratic weights are calculated first by summing the expenditure made by each household in each group. Expenditure on food is summed (10 + 20) to derive the total expenditure on food (30). The same is done for expenditure on other goods and services (20 + 80) to derive total expenditure on other goods and services (100). The total expenditure for each group is then summed to derive total expenditure on all items (30 + 100 = 130). Plutocratic weights reflect the relative importance of each group (30/130 = 23 percent and 100/130 = 77 percent). For the democratic weights, an average of the individual shares for each household is calculated and used for the weight. The democratic weight for food equals the average of the shares for each household ([33 + 20]/2 = 27 percent). The same calculation is made for other goods and services ([67 + 80]/2 = 73 percent).
3.18 For analytical purposes, additional weighting structures may be derived to measure the inflation experience of different subgroups of households. Such an analysis is typically conducted by classifying households according to a sociodemographic variable such as income, age, or educational level. Depending on the objective of the analysis, either democratic or plutocratic weights could be derived for the different household groups.
The Weighting Structure of the Consumer Price Index
3.19 The calculation of a CPI usually proceeds in two stages. In the first stage, elementary indices are estimated for each of the elementary aggregates. In the second stage, a weighted average is taken of these elementary indices using the expenditure shares of the elementary aggregates as weights. Elementary aggregates are usually the smallest groups of goods and services for which expenditure data are available. They may cover the whole country or separate regions within the country. Likewise, elementary aggregates may be developed for different types of outlets. The nature of the elementary aggregates depends on data needs and the availability of expenditure data. Elementary aggregates may therefore be defined differently in different countries. Other types of weights (for example, nonexpenditure weights representing market shares), if available, may be used within the elementary aggregate.
3.20 The weights are usually classified according to the Classification of Individual Consumption According to Purpose (COICOP), also used in the national accounts and the HBS. Some countries develop a more detailed product classification that refines the products defined at the subclass level. These product categories are not part of COICOP, but in many cases, more detailed breakdowns of COICOP subclasses are needed for CPI purposes to better reflect country-specific circumstances and needs. For example, a country may develop a more detailed weighting structure for “Rice” to include specific types of rice (for example, basmati rice or long-grain rice).
3.21 The weights for the COICOP groups, classes, and subclasses are their shares in the total consumption expenditures of the reference population. The data sources used to derive these shares are discussed in the following text. In addition, the weight for a subclass can be further stratified by region, by outlet or outlet type, or by a combination of both region and outlet. The elementary aggregate weights are the stratum weights according to expenditure class or subclass, region, and type of outlet. If no breakdown by region or outlet is used, the subclass becomes the elementary aggregate.
3.22 The weight of an elementary aggregate should reflect the expenditure on the entire elementary aggregate and not the weights of the outlets and varieties that have been chosen to represent it. For instance, the weight for the subclass “Rice” should be based on the total expenditure made on rice, although the rice varieties selected for regular price collection only represent a fraction of this expenditure. Likewise, if an expenditure category is divided into two elementary aggregates according to outlet type (for example, open markets and supermarkets, with corresponding market shares of food sales, 60 percent and 40 percent, respectively), these proportions would be used to estimate the stratum weights, whatever the importance of the specific outlets eventually sampled.
3.23 The methods used to calculate the elementary price indices from the individual price observations collected within each elementary aggregate are explained in Chapter 8. Working upward from the elementary price indices, all indices above the elementary aggregate level are described as higher-level indices that can be calculated from the elementary price indices using the weights of the elementary expenditure aggregates. The aggregation structure is consistent, so that the weight at each level above the elementary aggregate is always equal to the sum of its components. The price index at each higher level of aggregation can be calculated using the weights and price indices for its components, that is, the lower-level or elementary price indices.
3.24 Weights stratified by region may be used in the CPI, depending on the size and structure of the country, data availability, resources, and the purpose of the index. Introducing regional weights creates more homogeneous entities which are likely to experience similar price movements and have similar consumption patterns. It may be necessary to distinguish different regions because CPIs for individual provinces or states may be required for administrative or political purposes. In addition, in federal countries, indirect taxes and hence price developments may differ between the provinces or states.
3.25 In some countries, there may be large differences between urban and rural areas with regard to consumption patterns and price developments. A common practice is to introduce a stratification level that distinguishes between urban and rural areas. This approach assumes that data sources are available to derive expenditure weights separately for urban and rural areas. Moreover, prices must be collected in both rural and urban areas to compile the respective stratum indices.
3.26 If the weights derived from the HBS are available for rural and urban households and if price collection is limited to urban areas, one approach is to combine the weights for urban and rural households. This approach enhances the representativity of the index because the weights represent all households, both urban and rural. In this case, the prices faced by rural households will implicitly be imputed by those collected in urban areas. This can be an acceptable assumption if most of the monetary transactions made by households living in the rural areas take place in urban areas and if the relative changes in prices in rural and urban areas follow the same general trend. Alternatively, if prices are only collected in urban areas, it could also be decided to restrict expenditure weights to urban households and compile an accurate urban CPI.
3.27 Within a given COICOP subclass, the regional weight represents the consumption expenditure in the region in proportion to the expenditure in the whole country for that subclass. For example, if 60 percent of the total expenditure on fresh fruits occurs in the North region and 40 percent in the South region, then these proportions can be used to derive the regional elementary aggregate weights. If at the country level, the expenditure share for fresh fruits is 5 percent, this share can then be split between the regions so that 5 percent × 60 percent = 3 percent of the total national expenditure relates to fresh fruits in the North and 5 percent × 40 percent = 2 percent to fresh fruits in the South.
3.28 Regional weights may typically be obtained from the HBS if the sample design of the HBS is representative at the regional level and supports the development of reliable regional weights. When reliable estimates are not available from an HBS, population statistics are sometimes used to split household expenditure across regions; however, this approach is not preferred as it assumes that expenditures per capita or per household are the same in all regions. For instance, there are usually large differences between urban and rural populations in the level and pattern of items consumed. Finally, national accounts data, if compiled by the expenditure approach and available at the regional level, can also be used to estimate regional weights.
3.29 In practice, there are different strategies to derive regional weights depending on the availability and the quality of data sources. To develop a national weighting structure based on regional weights, first estimate household expenditure by region. Each of the regional expenditure values can then be summed up to obtain the expenditures at the national level. Alternatively, expenditures are first estimated at the country level before being distributed across different regions.
3.30 Consistency should be ensured between the product and the regional dimensions. In the example in Table 3.2, expenditure is estimated for three products in two regions (Table 3.2A). The expenditure share of each stratum in the national index corresponds to the stratum expenditure divided by the national total (Table 3.2B). According to this example, region 1 has a total weight of 38.8 percent. At the same time, product 1 has a weight of 24.5 percent at the national level.
Table 3.2C Expenditure by Region (Regional Product Expenditures Not Available)
Table 3.2D Product Expenditures for the Country Disaggregated by Region
1,200 × 1,900/4,900 = 465
1,200 × 3,000/4,900 = 735
1,500 × 1,900/4,900 = 582
1,500 × 3,000/4,900 = 918
2,200 × 1,900/4,900 = 853
2,200 × 3,000/4,900 = 1,347
Table 3.2E Regional Product Shares (estimated)
Region 1 (percent)
Region 2 (percent)
465/4,900 = 9.5
N.A. = Not availableAll totals are shown in bold.
3.31 If detailed product expenditures are only available at the national level, the regional product expenditures must be estimated. This can be done by using shares derived from the total expenditures that are made within a region. Table 3.2C assumes that product expenditures at the country level are available, and only total expenditure by region is known. In Table 3.2D the product expenditures for the country are then disaggregated by region, using a weight of 1,900/4,900 = 38.8 percent for region 1 and a weight of 3,000/4,900 = 61.2 percent for region 2. The regional product shares in Table 3.2E obtained with this method differ slightly from those in Table 3.2B. Ideally, the sample size by region will support expenditure estimates by product within each region, as shown in Tables 3.2A and 3.2B.
3.32 Prices are collected from a variety of outlets and outlet types. In addition to the regional dimension, the subclass could also be stratified by outlet or by outlet type. This can be especially useful if price levels and price changes differ significantly across outlets. Information about the sale or market share of the outlets may be used to form elementary aggregate weights specific to a given outlet type. In some countries, the HBS directly collects expenditure data by type of outlet which can then be used to estimate these stratum weights. The use of HBS data ensures consistency between the product weights and outlet-type weights. Other potential sources to derive outlet or outlet-type weights include point-of-purchase surveys, scanner data, business registers, or retail trade statistics. It may only be possible to develop outlet or outlet-type weights for a broader category of products so that the same proportions have to be used to disaggregate the expenditures on the different items within product groups. One should bear in mind that these additional data sources may not be completely aligned with the scope and the coverage of the CPI and may have their own limitations. It may, therefore, be necessary to make further adjustments to the data extracted from such sources.
3.33 In the example in Table 3.3, outlet types 1 and 2 have market shares of 60 percent and 40 percent, respectively, and this same breakdown holds for both regions and for all three products. If a division is made according to the type of outlet and region, then each item within a given region comprises two elementary aggregates: one for outlet type 1 and one for outlet type 2. In this example, the weight for the elementary aggregate for product 1 in region 1 sold in outlet type 1 would then be 8.2 percent × 60 percent = 4.9 percent. This same calculation would be used to develop weights for outlet type 2.
Deriving Expenditure Weights by Region and by Outlet Type
Region 1 (percent)
Region 2 (percent)
Outlet Type 1
Outlet Type 2
Outlet Type 1
Outlet Type 2
3.34 The HBS serves as the primary data source for CPI weights for most countries. These are household-based surveys that collect data on households’ expenditure and consumption of goods and services. Alternatively, national accounts data can serve as a data source. While national accounts data rely to some extent on expenditure data from the HBS, other data sources can be used to supplement the HBS data. Apart from the HBS, many other data sources can and should be used to improve the accuracy of the CPI weights. Such complementary data sources can be used in a number of ways, including (1) validating and correcting HBS estimates; (2) disaggregating the higher-level expenditures to develop a more detailed weighting structure; and (3) estimating expenditures for certain product categories.
Household Budget Surveys
3.35 In most countries, the HBS serves as the primary data source for deriving expenditure shares for the goods and services covered by the CPI. As the HBS may have been designed to serve more than one purpose, the survey design should meet the requirements for the CPI. Ensuring that CPI specific needs are met requires close coordination between the CPI and HBS staff. The classifications used for HBS and CPI should be consistent and preferably based on COICOP.
3.36 Data for a normal year should be used as the basis for developing CPI weights. Given that many HBSs are conducted infrequently, this can be challenging to manage in practice. If possible, an average of multiple years could be used, but this can only be done if the HBS is conducted continuously. In the absence of a continuous HBS, alternative data sources would be needed to develop weights. This issue of relying on a normal year to develop weights supports the need for more frequent weight updates.
3.37 The main requirements for the CPI include ensuring that the survey broadly represents the CPI reference population, and that all types of final consumption expenditure made by households are covered. The HBS sample size (number of households) should ensure that the expenditure data yielded are suitable to develop statistically reliable weights for the CPI at the elementary aggregate level. The HBS item list should be designed so that the information obtained maps directly into the CPI classification system. Ideally, the survey should also collect the data needed for deriving net weights for second-hand goods (see paragraphs 3.89–3.92). The interview and recording periods should be appropriately distributed over time to ensure that annual estimates can be obtained taking into account possible seasonal patterns.
3.38 The expenditure values used to derive weights should be consistent with the conceptual approach (acquisition, use, or payment) adopted for the CPI (for additional information on this topic, see Chapters 2 and 11). For some services, the moment when the service is consumed and acquired can differ from the moment when the service is paid for. The approach used in the HBS for determining the timing of consumption should be the same as the one used in the CPI. For most goods, the moments of acquisition, payment, and consumption coincide. Sometimes, the HBS focuses on the consumption of food products, which includes food purchased by the household, but also own production of food or food received for free. For durable goods, there is no difference between the acquisition approach and the payment approach if the purchase has not been financed with a loan. Once acquired, the use of such goods can span over several years. The HBS may collect different types of information regarding durable goods.
3.39 Theoretically, the HBS provides information on the regional breakdown of expenditure used to develop reliable regional weights. For this purpose, the regional dimension must be appropriately incorporated in the sampling design of the HBS. Depending on its design, the HBS may also provide information on the types of outlets and the brands purchased. In particular, it can be possible to identify internet purchases. Disaggregation of expenditure by type of outlet can be useful not only to construct elementary aggregates at a more detailed level but also to improve the sample design for outlets and items for price collection.
3.40 Like the HBS, national food surveys are special surveys with the primary emphasis on collecting information on family expenditure for food products. These surveys provide a very detailed breakdown of food expenditure that can be used to derive the weights for detailed elementary aggregates.
3.41 The detailed expenditure obtained from the HBS can be subject to measurement errors. There is often under or overreporting on luxury goods and services, as well as on certain types of products such as alcoholic beverages and tobacco. Moreover, the respondent household may not correctly remember the details of all the expenditure that took place during a given recall period. Therefore, HBS estimates should be reviewed and evaluated for completeness of coverage by comparing with secondary source data.
3.42 Given that the HBS in many countries is based on probability sampling methods, statistical quality indicators (for example, standard error and coefficient of variation) for the weights can be calculated. The analysis of these indicators can provide the index compiler with information on the structure of the elementary aggregates. If an analysis of the HBS data shows, for example, that the expenditure data for oranges are of poor statistical quality, then alternative data sources could be used to improve the reliability of the weights or a more broadly defined elementary aggregate such as citrus fruit could be considered instead. In a similar way, suppose that the HBS provides an unreliable expenditure estimate for an item at a detailed local level. It may then be preferable to derive an estimate based on the expenditure share of that item within a broader geographical area. To find the best compromise between lowering the variance of the item expenditure estimate and biasing it toward the spending pattern of the broader geographical area, a composite share can be computed that averages the initial direct share with the share obtained for the broader geographical area.
3.43 The frequency of updating the CPI weights depends on how often an HBS is conducted. For the purposes of the CPI, it is desirable for the HBS to be conducted at least every five years. This will allow countries to revise and update their expenditure weights every five years (or more frequently). Some countries conduct continuous HBSs with gradually rotating samples. However, a program of annual surveys with samples large enough to provide the type of estimates required for CPI weights can be very costly. For this reason, some countries conduct large-scale surveys at five-year intervals, perhaps supplemented with a smaller annual sample. Other countries distribute a large sample over several years. The average of the results over several successive years of smaller-scale surveys may provide a set of satisfactory annual estimates. The weights derived as the average of two or three years will also smooth any erratic consumer behavior over a shorter period, for example, because of events such as droughts or foods, civil strife, oil price shocks, or exceptionally mild or cold winters.
National Accounts Data
3.44 National accounts can be an alternative source for deriving CPI expenditure weights, if reliable estimates for HFCE are available. The practical advantage is that HFCE is updated every year, whereas an HBS may only be conducted on a less frequent basis. However, national accounts may only be available at the national level and the use of other available data sources would be needed to develop more detailed or regional expenditure weights.
3.45 The index compiler must understand the differences in scope and definition of consumption before using national accounts data for CPI weights. If the CPI is restricted to monetary transactions, then only a subset of HFCE must be used, excluding its nonmonetary components. Additionally, the CPI may adopt a different treatment for expenditure on types of goods and services, such as owner-occupied housing, or even a different scope, especially concerning institutional households.
3.46 National accounts data may be used to improve HBS weights for products that are underreported in the HBS. Note that national accounts’ estimates for HFCE are usually based on data from the HBS as well as a wide range of other sources such as domestic production, retail sales, tax information, and import and export data.
3.47 In practice, weights for the main product groups can be obtained from the national accounts down to a certain level of disaggregation. Each of these weights can then be disaggregated by applying the detailed HBS expenditure groups to the national accounts consumption groups. The combination of national accounts and HBS data ensures consistency between the CPI and the national accounts data on HFCE at the level of the main consumption groups. The use of national accounts data also facilitates more frequent weight updates. For instance, CPI weights can be updated at regular intervals using national accounts data for the higher-level aggregates. The updated expenditure is then distributed using the shares obtained from the HBS or other sources that may only be updated less frequently.
3.48 CPI compilers should consult with their national accounts counterparts regularly before using national accounts data for weights to ensure that they are consistent with the objectives and uses of the CPI. In some cases, national accountants need to apply an element of discretion and judgment when making operational decisions related to the construction of some national accounts aggregates. Moreover, the preliminary national accounts estimates are in general revised several times before the final estimates are available. The most recent available data may not be sufficiently stable for CPI purposes. There is a trade-off between timeliness and quality.
Other Data Sources
Administrative Data Sources
3.49 For some product categories, one option is to estimate expenditures from administrative data sources instead of relying on survey data. For instance, expenditure on medical products can often be obtained from relevant bodies in charge of the national health systems. In some countries, tax data for certain products such as alcohol or tobacco lead to more accurate estimates of expenditure than HBS data which suffers from underreporting. It can be also difficult for households to properly report explicit charges paid for financial services in an HBS. Moreover, the derivation of weights for insurance services requires special consideration (see paragraphs 3.87 and 3.88). Therefore, it may be preferable to use regulatory data sources to derive expenditure estimates for financial services and insurance. Administrative data sources are not always perfectly comparable with CPI coverage and may suffer from their own errors. For instance, tax revenue on tobacco also covers sales made to nonresident households whereas the CPI may only be limited to resident households. Most of the administrative data sources are also used in the national accounts which, as noted previously, can serve as a data source for weights.
Retail Trade Statistics
3.50 Statistics on retail sales by region and type of outlet may be available for broad groups of products. One disadvantage of using these data is that some of the sales may be to groups outside the reference population, including corporations or the government. The corresponding purchases do not form part of household final consumption. Some sales may also be to nonresidents, who may be part of the reference population. Furthermore, for regional sales data, it needs to be kept in mind that sales may include purchases made by households living in other regions.
3.51 Population censuses provide data on the geographical distribution of the population and households, as well as on the regional differences in household size and composition. Combined with estimates of regional levels of household expenditure, these data can be used to estimate regional expenditure weights, especially when such estimates are not available from an HBS with a satisfactory degree of precision. In the absence of any expenditure statistics, population statistics might be used as the basis for regional weights. However, such an approach should be avoided because it assumes that expenditure per capita or per household is the same in all regions and ignores the fact that there are usually large differences between the urban and rural populations in the level and patterns of consumption.
3.52 Scanner data can also be used to derive and update weights in a more frequent and timely manner. These data are based on electronic data records that are stored in the databases of sellers. Such scanner data sets include the quantities sold and the corresponding value aggregates. The limitations of this information should, however, be borne in mind. The first one is that scanner data cannot be connected to a specific type of household, whereas the data from the HBS can. Moreover, scanner data may only have limited outlet coverage. Finally, scanner data may not be fully consistent with the scope of the CPI, as no distinction can be made between sales to businesses, government, or households.
3.53 Scanner data can also be a good source for deriving detailed weighting structures especially at the lower levels of the index hierarchy. For instance, a detailed product and outlet stratification can be introduced by disaggregating the expenditure for a broader product category that was obtained from the primary data source. The use of scanner data to construct weights is further discussed in Chapter 10.
Market Intelligence and Trade Associations
3.54 If the product categories are sufficiently important, additional data sources might be consulted. A survey may be conducted with a small selection of outlets to obtain general information on the breakdown of sales for a specific product category. Existing market information can also be an option. Associations of importers or distributors, other industry groups, or marketing agencies and boards are likely to have some general information on the breakdown of sales for specific products.
Tourism Expenditure Surveys
3.55 If the CPI follows the domestic concept, the expenditure of nonresident households within the national territory must be included in the weights. In countries where tourism is important, tourism expenditure surveys can be conducted to estimate nonresident household expenditures that can be added to the expenditure made in the country by resident households obtained through the HBS. Nonresident visitors will generally have very different expenditure patterns from those of residents (for example, they will spend more on hotels and restaurants).
3.56 Point-of-purchase surveys provide data that can be used to estimate weights for different types of outlets and can be used to develop a sample frame of outlets where households make purchases. For products purchased, households are asked about the amounts spent in each outlet where purchases have been made. Given that household surveys are costly and that there is overlap between the HBS and point-of-purchase surveys, it is possible to combine the two into an integrated survey that collects expenditure and outlet data at detailed levels.
3.57 A simpler version of this survey may be conducted to obtain weights for groups of products by outlet type. As an alternative, in the absence of this type of survey, national retail sales statistics by outlet type from a survey of outlets could be used to estimate a breakdown of sales by outlet type.
Deriving the Weights in Practice
3.58 Once the reference population and the coverage of goods and services have been decided, the weights need to be derived. The weights are calculated as the proportions of the total consumption expenditure of all goods and services included in the index basket for the reference population during the reference period. The reliability of the CPI weights will obviously depend, to a large extent, on the reliability of the household expenditure data. In practice, the derivation of weights involves a series of steps.
Arrange the Data According to the Classification and Coverage of the CPI
3.59 The detailed expenditure items identified in the HBS, or other data sources, must be mapped to the CPI expenditure classes. If HBS classes do not match CPI expenditure classes, the HBS results must be transformed to match the CPI classes. This can be done by aggregating or disaggregating the relevant HBS headings over the relevant CPI expenditure subclasses. Such transformation is achieved much more easily and more reliably if the coding list for expenditure items in the HBS is coordinated with the corresponding list of items used for collecting price observations for the CPI. As noted previously, both the HBS and CPI should use the same classification system (ideally COICOP).
3.60 The HBS expenditure data may include payments that are outside the scope of the CPI. For example, payments of income taxes or social security contributions, life insurance premiums, remittances, gifts and other transfers, investments, savings, and debt repayments should not be considered because they are not consumption expenditure. These should be excluded from the total used to calculate the expenditure shares that serve as the basis to estimate the CPI weights.
Correcting for Over- and Underreporting by Combining Different Data Sources
3.61 The results from the HBS need to be carefully examined and adjusted to take account of under or over-reporting of consumption expenditure on different types of products. This is a relevant problem affecting HBS data. Evidence suggests that the understatement of expenditures can be significant for certain goods and services in the HBS if their consumption is socially discouraged, such as tobacco and alcoholic beverages. Expenditures on durable goods also tend to be underreported. Some products, such as vehicles or other major durable goods, are purchased infrequently. When purchased, the amount spent on such products can be considerable. As the HBS is a sample survey, estimates are subject to sampling errors, which may be relatively large for such infrequent expenditures. For major durable goods, HBS estimates should be compared with other sources such as import statistics or administrative data.
3.62 Other expenditures are not reported because the purchases were small, and easily forgotten by the respondent. Therefore, to the extent possible, results from the HBS should be compared and combined with other data sources when constructing CPI weights, especially when the HBS sample is small.
3.63 The usual strategy to correct for over or underreporting is to use supplementary information from other relevant data sources such as tax data, administrative data, other independent surveys, or HFCE from the national accounts, and to apply correction factors. For instance, assume that tax revenue data indicate that annual sales for cigarettes may be twice as high as the total annual expenditure estimated from the HBS. The CPI weight is thus obtained by increasing the HBS expenditure on cigarettes by two to better reflect the sales data.
3.64 In countries with reliable and detailed national accounts data, the commodity-flow method can be used to adjust data considered less reliable from the HBS. The use of the commodity-flow method within the Supply and Use Table framework, as described in the System of National Accounts, enables data drawn from different primary sources to be reconciled and balanced against each other. The commodity-flow method may be used to improve estimates of HFCE derived from the HBS by adjusting them to account for additional information provided by statistics on sales, production, and imports and exports of consumer goods and services.
3.65 The Supply and Use Tables form an integrated framework where supplies of different kinds of goods and services originating from domestic industries and imports are allocated between various intermediate or final uses, including HFCE. The product balance for any product recognizes that the sum of output at basic prices plus imports plus trade and transport margins plus taxes on products less subsidies on products is equal to the sum of intermediate consumption, final consumption, and capital formation, all expressed at purchasers’ prices, plus exports (System of National Accounts 2008 [2008 SNA], paragraph 14.5). The HFCE estimates obtained can be compared with the corresponding estimates from the HBS to provide conversion factors to adjust HBS expenditure data for under or overreporting.
3.66 There are some practical limitations to applying the commodity-flow method. Often the balance can only be established for a category of products broader than the product categories used in the CPI classification. Moreover, data are usually compiled at the national level and no detailed regional breakdown is available. Finally, the HBS may not be the only inaccurate data source for a particular product and the estimates of other components of the product balance can also be unreliable.
Adjusting Household Budget Surveys
3.67 Even if expenditure data obtained from the HBS results are considered accurate, adjustments might still be needed to account for any significant changes in expenditure patterns between the period when the survey was conducted and the period when the new weights are introduced. Adjustments will typically be made for products belonging to fast-evolving markets and which are significantly losing or gaining importance during this period. It is possible that expenditure on some products may not be available from the HBS because the products appeared on the market after the survey had been completed. Additional data sources must then be accessed to estimate expenditure for new products. Expenditure should also be reviewed if there are known changes that have occurred following administrative decisions, such as changes in taxation, that entered into force only after the HBS was conducted.
Treatment of Expenditure for Unimportant or Difficult-to-Measure Products
3.68 The HBS, which in most cases is the main source for deriving the detailed weights, usually includes observations on a much larger variety of goods and services than it is practical to collect prices for in the CPI. Some products may have a weight which for all practical purposes is negligible. The prices of products with very small weights may not be worth collecting if their contribution to the CPI is very small. In practice, a cutoff threshold can be defined to select the products to be included in the CPI basket (see Chapter 4 for a description of different sampling techniques).
3.69 Among the consumption expenditure, there are also likely to be a few products for which the prices, or price changes, cannot be directly or satisfactorily measured, such as gambling. It may be difficult if not impossible to compile a reliable price index for such products. A decision must then be made on how to treat the expenditure allocated to such difficult-to-measure products.
3.70 Even if the product weight is small or if there are measurement problems, the product is still included in the scope of the CPI. For those products with relatively low weights, it is not efficient to collect prices; however, the CPI should cover these expenditures. Some price change should be explicitly or implicitly assumed or imputed. In practice, there are different options:
The weight of the product is combined with one or several related products. For instance, if the weight for “cheese” is small, it can be combined with the weight for “milk.” This means that the elementary aggregate is now weighted using expenditures on milk and cheese although only prices for milk will be collected. This option assumes that the price index for cheese would change in the same manner as the observed price index for milk. In practice, the weights for the products could be kept separated but price changes must be explicitly imputed. To continue using the previous example, the price index attached to the weight for cheese would simply be identical to the price index for milk. This is the preferred approach.
An alternative approach is to set the weight for which no representative prices exist equal to zero. This option removes the item from the scope of the CPI and is equivalent to the assumption that the price of the excluded product would have moved in the same way as the all-items CPI for all the products actually included in the index. This approach is, in general, not recommended.
3.71Table 3.4 shows examples of different options for the treatment of unimportant expenditures. In the example, the expenditure for product 3 is not relevant. One option is to add this expenditure to item 1. Alternatively, it may be added to products 1 and 2 proportionally to the weights of these two products. Finally, it can be removed, which implies that the total expenditure for the three products is now reduced. In general, the last option should be avoided, and expenditures should be allocated to products with similar price behavior. Because of the negligible size of the weight value involved, the consequence on the overall index will in general be negligible whichever method is used.
Treatment of Products for Which No Prices Are Collected
Allocate to Item 1
Allocate to Items 1 and 2
Product 3 (minor item)
Weight Reference Period
3.72 The weight reference period refers to the time period to which the expenditure data used to estimate weights were collected. For a fixed-basket CPI which keeps weights constant over several periods, the weight reference period is typically a 12-month period, such as a calendar year. A month or quarter is too short to be used as a weight reference period, since any month or quarter is likely to be affected by accidental or seasonal influences. This is especially important in countries where the composition and size of expenditure can fluctuate significantly throughout the year. In some countries, data for a single year may not be adequate because of unusual economic conditions or due to insufficient sample size. In these cases, an average of more than one year of expenditure data may be used to calculate the weights.
3.73 As the CPI is sensitive to the selection of the weight reference period, it might be preferable to choose a “normal” consumption period as the basis for weights and to avoid periods in which there are special factors of a temporary nature at work. To achieve this, it may be necessary to adjust some of the values to normalize them and to overcome any irregularities in the data. One option might be to smooth particularly erratic observations, for example, by taking an average of HBS data over more than one year. All available information concerning the nature of consumption in a weight reference period should be taken into consideration.
3.74 During periods of high inflation, multiple year weights may be calculated by averaging value shares rather than averaging actual value levels. Averaging value levels will give too much weight to the data for the most recent year. Another option is to update the values for each year to a common period and then to compute a simple arithmetic average of adjusted yearly data.
3.75 As the weight reference period usually precedes the price reference period, the expenditure weights may be price updated to take account of the relative price changes from the weight reference period to the price reference period. The technique of price-updating expenditure data or CPI weights from a past period does not make weights reflect current expenditure patterns. New CPI weights can only be derived if new expenditure data are available. Price updating of weights is discussed in more detail in Chapter 9.
Frequency of Weight Updates
3.76 The expenditure weights should be updated at regular intervals, as often as possible, but at least every five years. The decision of when to update the weights often depends on the availability of appropriate data sources. Rather than waiting for the weights to become outdated before deciding to update them, the recommended approach is to plan for regular scheduled updates subject to the available data sources. In general, there is a lag between the weight reference period and the moment that the weights are introduced in the CPI. The overall principle is to minimize the implementation lag so that the weights used in the CPI are as up to date as possible. New weights should be introduced in a timely manner, as soon as the required source data are available. For instance, expenditure weights referring to year y – 2 could be introduced with the January index of year y.
3.77 If weights are kept fixed over longer time periods, the index will be unresponsive to substitution effects or changes in consumer preferences. In the short term, consumers may change consumption patterns in response to shifts in relative prices, mostly between products included in the same class or subclass. In the medium or long term, consumption patterns are also influenced by factors other than price changes. Most importantly, under rising incomes, changes in the level and distribution of household income will cause a shift in demand for goods and services toward goods and services with higher income elasticities. Demographic factors such as aging of the population and technological changes are examples of other factors that affect spending behavior in the longer term. Furthermore, new products will be introduced, and existing ones may be modified or become obsolete. As a result of both relative price changes and long-term effects, the weights may become out of date and less representative of current consumption patterns. As shown in Chapter 2 of the publication Consumer Price Index Theory, the bias in a fixed-basket index is likely to increase with the age of the weights. At some point, it therefore becomes desirable to use the weights of a more recent period to ensure that the index is weighting appropriately the price changes currently faced by consumers.
3.78 If data sources such as the HBS are available only at irregular or infrequent intervals, the frequency of weight revision may necessarily be linked to the availability of results from the HBS. When the weights are to be fixed for several years, the objective should be to adopt weights that are not likely to change much in the future, rather than precisely reflect the activity of a particular period that may be abnormal in some way. Even if weights are updated only every five years, it is desirable to review the weights in between to ensure that they remain sufficiently reliable and representative. The review, which may be limited to weights at the elementary index level and their major components, should examine whether there are indications that important changes may have taken place in the consumption pattern since the weight reference period.
3.79 To reduce upper-level substitution bias, it would be preferable to update the weights more often such as every three years, every two years, or even every year; however, this depends on the availability of data to develop new weights. Any bias which may follow from using a Lowe index (see Chapters 1 and 8 for information on price index formulas) with a fixed basket of goods and services will not have time to accumulate to a significant magnitude. Also, by updating the weights more frequently, there is the opportunity to introduce newly significant goods and services in a timely manner. At the same moment, the sample of outlets and varieties to be priced can be updated simultaneously, although the sample should be maintained in between two weight updates. Countries that are experiencing significant economic changes, and thus more rapid changes in the consumption pattern, should consider updating their weights more frequently.
3.80 The benefits of updating weights more often should be assessed and compared to the additional costs of such an exercise. If feasible, empirical studies can be conducted on historical data to assess the impact of updating weights more frequently, of choosing different weight reference periods, and of minimizing the time lag between the weight reference period and the price reference period. The resulting indices can then be compared to specific target indices.
3.81 An intermediate solution is to update weights only at the higher level, for example using national accounts data, which can then be disaggregated down to the lower levels using shares from the HBS, which has not been updated. Partial weight updates are further discussed and illustrated in Chapter 9.
3.82 Frequent updating of weights and chaining can lead to chain drift. For a Laspeyres or a Lowe index, the drift can be upward if there are systematic fluctuations in consumption and prices. Suppose that the prices for petroleum products went down between the old and the new weight reference period, causing an increase in the quantities consumed and expenditure for these products. If petroleum prices subsequently rise after the new weights are introduced, the aggregate CPI will rise more rapidly than before since these products now have more importance. At the level of broad product categories, expenditure data analyzed over longer periods often follow trends. Chaining can have a downward impact if, for instance, there is a gradual shift from one product to another product for which prices are rising slower. A direct CPI compiled with unchanged weights is often found to be higher than the chained index compiled, for instance, with annually updated weights. Chain drift is more problematic with high-frequency updating of weights and chaining at the level of an individual variety. Multilateral methods that are discussed in Chapter 10 (and Chapter 7 of Consumer Price Index Theory) are one solution to this problem.
3.83 The selection of the level in the index hierarchy at which the structure and weights are fixed for a period is particularly important. The main advantage of setting the level relatively high is that the actual samples of products and their prices below this level can be adjusted and updated as needed (see Chapter 7). New products can be introduced into the samples, and the weights at the lower level updated using more recent information. There is thus a greater opportunity to keep the index representative through an ongoing review of the sample of representative products. If the level is set relatively low in the index structure, there is less freedom to maintain the representativeness of the index on an ongoing basis, and there will be a greater dependence on the periodic index review and reweighting process. In such circumstances, the arguments for frequent reweighting become stronger.
3.84 Whenever the weighting pattern has been updated, the new index using updated weights should be calculated for an overlapping period with the old one so that the two can be linked. These techniques are discussed in Chapter 9. During the year that follows the weight update, the year over year rates of the higher-level aggregates compiled from the linked series not only reflect changes in prices, but are also impacted by the use of different weights and a different item structure.
Items Requiring Special Treatment
3.85 In practice, there are two types of approaches for the treatment of seasonal products:
A fixed-weight approach, which assigns the same weight for the seasonal product in all months, using an imputed price in the out-of-season months. Seasonal products are treated in the same way as other consumption products.
A seasonal-weight approach, in which a zero weight is attached to a product which is out of season, and a positive weight is used for the in-season periods when the product is available for pricing. The in-season weights are kept fixed as much as possible and only vary to the extent necessary to reflect changes in the composition of the basket. Moreover, the principle of a fixed basket (that is, fixed weights) should be maintained at least at some level of aggregation. One disadvantage of such an approach is that the monthly changes in the index may be more difficult to interpret as they may reflect not only price changes but also quantity changes.
The treatment of seasonal products is further discussed in Chapter 11. In addition, this topic is examined in Chapter 9 of the publication Consumer Price Index Theory.
3.86 As internet purchases become increasingly important, these expenditures should be included in the CPI weights according to the conceptual approach that has been adopted. Chapter 11 discusses how internet purchases should be treated either under the national or the domestic concept. Ideally, data on the share of internet purchases for all relevant product categories could be available, so that detailed strata can be defined for this outlet type. The HBS should, in principle, cover internet purchases made by resident households. The survey should be designed so that it is possible to separate internet purchases from other purchases. The estimation of expenditure on internet purchases is of interest to both the HBS and the CPI programs and therefore it would be beneficial to cooperate on this topic. Similarly, the treatment of internet purchases in the national accounts should be carefully examined if this is the primary data source for CPI weights.
3.87 Alternative data sources for measuring e-commerce transactions include bank account data and credit card data. However, excluding business expenditure from these data sets could be problematic and identifying specific products and outlets may not be straightforward. Finally, reports prepared by external organizations that monitor and describe e-commerce markets can also help to estimate weights for internet purchases. The scope, coverage, and data collection methods underlying the statistics presented in such reports must be well understood when combining such figures with those obtained from other sources.
3.88 As explained in Chapters 2 and 11, the weights for nonlife insurance could be based on either (1) the gross premiums paid, consisting of the payment for the insurance, or (2) on the implicit service charges payable to the insurance enterprise for arranging the insurance. The implicit service charges for administering the insurance and providing the insurance services are estimated by the gross premiums plus the income from investment of the insurance reserves less the amounts payable to policyholders in settlement of claims. This definition can potentially lead to negative weights for instance if there are irregular and unexpected large fluctuations in claims due to natural disasters or large-scale accidents. To avoid negative weights, an average service charge covering several years could be considered.
3.89 If the weights are based on the gross premiums, expenditure weights should normally exclude goods and services provided for or reimbursed by the insurance company on the basis of claims. If the weights are based on the implicit service charge, expenditure weights include goods and services that households buy and that are reimbursed by insurance companies, and also goods and services that are paid for and provided by insurance companies on the basis of claims. In general, it seems preferable to base the weights for nonlife insurance on the service charges.
3.90 The prices of used, or second-hand, goods purchased by households are included in the CPI in the same way as the prices of new goods (Chapter 11 discusses this topic in more detail). However, households also sell used goods, such as cars. If the price of a second-hand good rises, a purchasing household is worse off, but a selling household is better off. From a weighting perspective, sales constitute negative expenditures, which implies that price changes for used goods sold by households implicitly carry a negative weight in the CPI. In effect, purchases and sales of secondhand goods between households, whether directly or indirectly through a dealer, cancel out (except for the dealers’ margins, as explained in Chapter 2) and carry no weight in the CPI. However, households also buy from, and sell to, other sectors. For the reference population, namely the entire set of households covered by the CPI, the weight to be attached to a particular kind of second-hand good is given by the households’ total expenditure on it less the value of the households’ revenue from sales to/from outside the household sector, including the rest of the world. There is no reason why these should cancel out on aggregate. For example, many of the second-hand cars purchased by households may be imported from abroad. The difference between total expenditures and total sales is usually described as households’ net expenditure. This is the weight to be attached to the second-hand good in question. Table 3.5 illustrates how to develop weights for used goods. In this example, the net weight would be obtained as (100 + 300) – (100 + 200) = 400 – 300 = 100.
3.91 Second-hand markets may exist for a whole range of durable and semidurable goods. Except in the case of used cars, it is often very difficult to estimate the net expenditure because most HBSs do not collect the data that would allow for a comparison between expenditure and revenue from sales of individual kinds of second-hand goods. Usually, only the total amount received from the sale of secondhand goods is collected. However, this information gives an idea of the volume and significance of these transactions in the economy.
3.92 In countries where the volume of second-hand purchases is small, second-hand goods (except used cars) may be ignored when calculating the weights of the index. In countries where second-hand purchases are important, and their prices are believed to change at different rates from those of new goods, separate weights are needed. The information may be obtained, at least for some major durables, from the HBS, if the survey asks about expenditure on second-hand and new goods. Because the amounts spent on purchasing second-hand cars are usually large, they should be included in the CPI basket if the data are available.
3.93 Even if countries include expenditure on secondhand goods in the estimation of CPI weights, second-hand goods may not be covered in the price collection. By excluding second-hand goods from the price collection, it is implicitly assumed that the prices of new and second-hand goods move in the same way. If they are included, the price-determining characteristics of the second-hand good must be kept constant over time so the same good is priced each month. If the goods are different, appropriate quality adjustments must be made.
3.94 If the CPI includes owner-occupied housing under the rental equivalence approach, weights represent the rents that owners would pay for the dwelling that they occupy. The owners could be directly asked what rent they would pay for their dwelling; however, such answers are often found to be unreliable. The preferred option is to rely on observed rent data to impute a rental value for the stock of owner-occupied dwellings. The assumption is that the rent of an owner-occupied dwelling is expected to be equal to the observed rent of a dwelling with similar characteristics. Such an approach is also consistent with the 2008 SNA (paragraph 6.117) which recommends valuing the output of the housing services produced by owner-occupiers at the estimated rental that a tenant would pay for the same accommodation.
3.95 If the CPI includes owner-occupied housing under the acquisitions approach, the weights are based on the (net) purchases of dwellings plus major repairs. The purchase of a dwelling includes both a structure and a land and location component. Only the value of the structure element should be included in the weights of a CPI that aims at measuring consumption expenditure, whereas the value of the land should in principle be excluded. Moreover, only dwellings acquired for the purpose of own occupation should be covered. In addition, an owner-occupied housing index under the acquisitions approach would include transaction costs and other costs related to the use of the dwelling, in which case weights must also be estimated for these components. Possible data sources for deriving owner-occupied housing weights are statistics on building activity, business statistics in construction, or administrative records of property transactions. Chapter 11 describes in more detail the treatment of owner-occupied housing in the CPI.
Inclusion of Newly Significant Products
3.96 As discussed in Chapter 7, newly significant products that appear over time should be identified. If a new product becomes important with regard to household expenditure, it should be included in the index structure. One option is to wait and introduce the new product at the time of a complete weight update. Alternatively, the current weighting structure could be adjusted, using the techniques described in Chapter 7, to introduce the new products in the CPI in a timely manner. In any case, expenditure weights must be estimated for the new products. Such data may not always be readily available from the HBS. If that is the case, the newly identified products should be incorporated in the HBS as soon as possible. In the meantime, alternative data sources may be used to estimate the weights of the new products so that their inclusion in the CPI is not delayed.
CPI weights represent the expenditure shares for specific goods and services that define the basket.
CPI weights should be consistent with the decisions made regarding the geographic coverage, the household reference population, the type of consumption expenditure included in the CPI, and the product coverage.
The objective is to estimate expenditure consistent with the scope of the CPI at a sufficiently detailed product level, possibly stratified by region or by outlet type, so that reliable weights can be attached to the different elementary aggregates that make up the index structure.
The primary data source for deriving expenditure weights are typically HBSs, but expenditure data may also be directly obtained from the national accounts.
When using the HBS as the primary source for deriving the weights, the data have to be reviewed and corrected for possible under or overreporting by comparing them with other data sources in order to obtain a representative weighting structure.
Secondary data sources can be useful to estimate expenditure for certain product categories, to further disaggregate the expenditure at the lower levels, and to correct for over or underreporting.
The weights should be updated as often as possible, but at least once every five years, to ensure their relevance and the representativity of the index.
Countries which are experiencing significant economic changes, and thus more rapid changes in consumption patterns, should update their weights even more frequently.
The lowest level weights are likely to become out of date more quickly than upper-level weights. Therefore, these lower-level weights, for at least some categories, need to be reviewed and updated more frequently than upper-level weights to reflect changes in consumption patterns. The best use of the available statistical information (for example, market research or small annual surveys) should be made for this purpose.
A regular schedule of updating the weights (at least every five years or as often as possible) should be followed and made available to users.
Abe, N., andK.Shinozaki. 2018. “Compilation of Experimental Price Indexes Using Big Data and Machine Learning: A Comparative Analysis and Validity Verification.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 7–9. https://www.unece.org.
Abe, N., andK.Shinozaki. 2018. “Compilation of Experimental Price Indexes Using Big Data and Machine Learning: A Comparative Analysis and Validity Verification.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 7–9. https://www.unece.org.)| false
Aizcorbe, A.,C.Corrado, andM.Doms. 2001. Constructing Price and Quantity Indexes for High Technology Goods. Industrial Output Section, Division of Research and Statistics. Washington, DC: Board of Governors of the Federal Reserve System.
Aizcorbe, A.,C.Corrado, andM.Doms. 2001. Constructing Price and Quantity Indexes for High Technology Goods. Industrial Output Section, Division of Research and Statistics. Washington, DC: Board of Governors of the Federal Reserve System.)| false
Aizcorbe, A.,C.Corrado, andM.Doms. 2003. “When Do Matched-Model and Hedonic Techniques Yield Similar Price Measures?” Working Paper 2003–14, Federal Reserve Bank of San Francisco, San Francisco, CA.
Aizcorbe, A.,C.Corrado, andM.Doms. 2003. “When Do Matched-Model and Hedonic Techniques Yield Similar Price Measures?” Working Paper 2003–14, Federal Reserve Bank of San Francisco, San Francisco, CA.)| false
Armknecht, P. A.2016. “Fixed Basket Methods for Compiling Consumer Price Indexes.” Meeting of the Group of Experts on Consumer Price Indices, Geneva, Switzerland, May 2–4, 2016. http://www.unece.org/index.php.
Armknecht, P. A.2016. “Fixed Basket Methods for Compiling Consumer Price Indexes.” Meeting of the Group of Experts on Consumer Price Indices, Geneva, Switzerland, May 2–4, 2016. http://www.unece.org/index.php.)| false
Armknecht, P. A., andF.Maitland-Smith. 1999. “Price Imputation and Other Techniques for Dealing with Missing Observations, Seasonality and Quality Change in Price Indices.” Working Paper No. 99/78, International Monetary Fund, Washington, DC. http://0-www-imf-org.library.svsu.edu/external/pubs/ft/wp/1999/wp9978.pdf)| false
Armknecht, P. A.,W. F.Lane, andK. J.Stewart. (1997) 2003. “New Products and the U.S. Consumer Price Index.” In Scanner Data and Price Indexes, NBER Studies in Income and Wealth, edited by R. C.Feenstra andM. D.Shapiro. Chicago, IL: University of Chicago Press, pp. 375–91.
Armknecht, P. A.,W. F.Lane, andK. J.Stewart. (1997) 2003. “New Products and the U.S. Consumer Price Index.” In Scanner Data and Price Indexes, NBER Studies in Income and Wealth, edited by R. C.Feenstra andM. D.Shapiro. Chicago, IL: University of Chicago Press, pp. 375–91.)| false
Astin, J. 1999. “The European Union Harmonized Indices of Consumer Prices (HICP).” in Proceedings of the Ottawa Group Fifth Meeting, Reykjavik, Iceland, August 25–27, edited by R.Gudnason andT.Gylfadottir. Reykjavík: Statistics Iceland; also published in Statistical Journal of the United Nations ECE16: 123–35. http://www.statcan.ca/secure/english/ottawagroup/.
Astin, J. 1999. “The European Union Harmonized Indices of Consumer Prices (HICP).” in Proceedings of the Ottawa Group Fifth Meeting, Reykjavik, Iceland, August 25–27, edited by R.Gudnason andT.Gylfadottir. Reykjavík: Statistics Iceland; also published in Statistical Journal of the United Nations ECE16: 123–35. http://www.statcan.ca/secure/english/ottawagroup/.)| false
Astin, J. 2001. “New Uses of PPPs within the European Union.” Paper presented at the Joint World Bank–OECD Seminar on Purchasing Power Parities: Recent Advances in Methods and Applications, Washington, DC, January 30–February 2.
Astin, J. 2001. “New Uses of PPPs within the European Union.” Paper presented at the Joint World Bank–OECD Seminar on Purchasing Power Parities: Recent Advances in Methods and Applications, Washington, DC, January 30–February 2.)| false
Aten, B.,R.Summers, andA.Heston. 2001. “An Explanation of Stability in Country Price Structures: Implications for Spatial– Temporal Comparisons.” Paper presented at the Joint World Bank–OECD Seminar on Purchasing Power Parities: Recent Advances in Methods and Applications, Washington, DC, January 30–February 2.
Aten, B.,R.Summers, andA.Heston. 2001. “An Explanation of Stability in Country Price Structures: Implications for Spatial– Temporal Comparisons.” Paper presented at the Joint World Bank–OECD Seminar on Purchasing Power Parities: Recent Advances in Methods and Applications, Washington, DC, January 30–February 2.)| false
Australian Bureau of Statistics. 2013. “Feature Article: The Use of Transactions Data to Compile the Australian Consumer Price Index.” Catalogue No. 6401.0, ABS, Canberra. http://www.abs.gov.au/.
Australian Bureau of Statistics. 2013. “Feature Article: The Use of Transactions Data to Compile the Australian Consumer Price Index.” Catalogue No. 6401.0, ABS, Canberra. http://www.abs.gov.au/.)| false
Australian Bureau of Statistics. 2017b. “Introduction of the 17th Series Australian Consumer Price Index.” Information Paper 6470.0.55.001, ABS, Canberra. http://www.abs.gov.au/AUS-STATS/abs@.nsf/DetailsPage/6470.0.55.0012017.)| false
Bäckström, Peter, andMuhanadSammar. 2012. “The Use of Superlative Index Links in the Swedish CPI.” Paper presented at the meeting of the Group of Experts on CPI, Geneva. https://www.unece.org/index.php.
Bäckström, Peter, andMuhanadSammar. 2012. “The Use of Superlative Index Links in the Swedish CPI.” Paper presented at the meeting of the Group of Experts on CPI, Geneva. https://www.unece.org/index.php.)| false
Baker, D. 1998. “Does the CPI Overstate Inflation? An Analysis of the Boskin Commission Report.” In Getting Prices Right, edited by D.Baker. Washington, DC: Economic Policy Institute, pp. 79–155.
Baker, D. 1998. “Does the CPI Overstate Inflation? An Analysis of the Boskin Commission Report.” In Getting Prices Right, edited by D.Baker. Washington, DC: Economic Policy Institute, pp. 79–155.)| false
Baldwin, A.,A.Nakamura, andM.Prudhomme. 2010. “An Empirical Analysis of the Different Concepts for Owned Accommodation in the Canadian CPI: The Case of Ottawa, 1996–2005.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 10–12. https://www.unece.org.
Baldwin, A.,A.Nakamura, andM.Prudhomme. 2010. “An Empirical Analysis of the Different Concepts for Owned Accommodation in the Canadian CPI: The Case of Ottawa, 1996–2005.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 10–12. https://www.unece.org.)| false
Balk, B. M.1994. “On the First Step in the Calculation of a Consumer Price Index.” Paper presented at First Meeting of the International Working Group on Price Indices, Ottawa, October 31–November 4. http://www.ottawagroup.org.
Balk, B. M.1994. “On the First Step in the Calculation of a Consumer Price Index.” Paper presented at First Meeting of the International Working Group on Price Indices, Ottawa, October 31–November 4. http://www.ottawagroup.org.)| false
Balk, B. M.1998b. “On the Use of Unit Value Indices as Consumer Price Subindices.” Paper presented at the Fourth Meeting of the International Working Group on Price Indices, Washington, DC, April 22–24. http://www.ottawagroup.org.
Balk, B. M.1998b. “On the Use of Unit Value Indices as Consumer Price Subindices.” Paper presented at the Fourth Meeting of the International Working Group on Price Indices, Washington, DC, April 22–24. http://www.ottawagroup.org.)| false
Balk, B. M.2001. Aggregation Methods in International Comparisons: What Have We Learned?Report Series Research in Management ERS-2001–41-MKT, Erasmus Research Institute of Management. Rotterdam: Erasmus University.
Balk, B. M.2001. Aggregation Methods in International Comparisons: What Have We Learned? Report Series Research in Management ERS-2001–41-MKT, Erasmus Research Institute of Management. Rotterdam: Erasmus University.)| false
Balk, B. M.2010. “Lowe and Cobb-Douglas Consumer Price Indexes and Their Substitution Bias.” Jahrbücher für Nation-alökonomie und Statistik (The Journal of Economics and Statistics) 230 (6): 726–40.
Balk, B. M.2010. “Lowe and Cobb-Douglas Consumer Price Indexes and Their Substitution Bias.” Jahrbücher für Nation-alökonomie und Statistik (The Journal of Economics and Statistics) 230 (6): 726–40.)| false
Ball, A., andD.Fenwick. 2004. “Costing Mobile Telephone Calls: The Use of Constrained User Profiles.” Paper presented at the Eighth Meeting of the International Working Group on Price Indices, Helsinki, August 23–25. http://www.ottawagroup.org.
Ball, A., andD.Fenwick. 2004. “Costing Mobile Telephone Calls: The Use of Constrained User Profiles.” Paper presented at the Eighth Meeting of the International Working Group on Price Indices, Helsinki, August 23–25. http://www.ottawagroup.org.)| false
Bas, C. 2018. “Different Approaches and Methods for Strong Seasonal Items in the CPI.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 7–9. https://www.unece.org.
Bas, C. 2018. “Different Approaches and Methods for Strong Seasonal Items in the CPI.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 7–9. https://www.unece.org.)| false
Bascher, J., andT.Lacroix. 1999. “Dishwashers and PCs in the French CPI: Hedonic Modeling, from Design to Practice.” Paper presented at the Fifth Meeting of the International Working Group on Price Indices, Reykjavik, August 25–27. http://www.ottawagroup.org.
Bascher, J., andT.Lacroix. 1999. “Dishwashers and PCs in the French CPI: Hedonic Modeling, from Design to Practice.” Paper presented at the Fifth Meeting of the International Working Group on Price Indices, Reykjavik, August 25–27. http://www.ottawagroup.org.)| false
Beck-MacNeil, R. 2018. “Pricing the Internet—The Internet Access Services Component of the Canadian CPI.” Statistics Canada. Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 7–9. https://www.unece.org.
Beck-MacNeil, R. 2018. “Pricing the Internet—The Internet Access Services Component of the Canadian CPI.” Statistics Canada. Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 7–9. https://www.unece.org.)| false
Beisteiner, A. 2008. “Optimal Allocation of Prices in Practice— Using the Neumann Formula as Tool.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 8–9. https://www.unece.org.
Beisteiner, A. 2008. “Optimal Allocation of Prices in Practice— Using the Neumann Formula as Tool.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 8–9. https://www.unece.org.)| false
Bentley, A., andF.Krsinich. 2017. “Towards a Big Data CPI for New Zealand.” Paper presented at the Fifteen Meeting of the International Working Group on Price Indices, Eltville am Rhein, May 10–12. http://www.ottawagroup.org.
Bentley, A., andF.Krsinich. 2017. “Towards a Big Data CPI for New Zealand.” Paper presented at the Fifteen Meeting of the International Working Group on Price Indices, Eltville am Rhein, May 10–12. http://www.ottawagroup.org.)| false
Bernardini, A.,C.De Vitiis,A.Guandalini,F.Inglese, andM. D.Terribili. 2016. “Measuring Inflation Through Different Sampling Designs Implemented on Scanner Data.” Paper presented at Meeting of the Group of Experts on Consumer Price Indices, Geneva, Switzerland, May 2–4.
Bernardini, A.,C.De Vitiis,A.Guandalini,F.Inglese, andM. D.Terribili. 2016. “Measuring Inflation Through Different Sampling Designs Implemented on Scanner Data.” Paper presented at Meeting of the Group of Experts on Consumer Price Indices, Geneva, Switzerland, May 2–4.)| false
Berndt, E. R.,D.Ling, andM. K.Kyle. 2003. “The Long Shadow of Patent Expiration: Generic Entry and Rx to OTC Switches.” In Scanner Data and Price Indexes, NBER Studies in Income and Wealth, edited by M.Shapiro andR. C.Feenstra. Chicago, IL: University of Chicago Press, pp. 229–73.
Berndt, E. R.,D.Ling, andM. K.Kyle. 2003. “The Long Shadow of Patent Expiration: Generic Entry and Rx to OTC Switches.” In Scanner Data and Price Indexes, NBER Studies in Income and Wealth, edited by M.Shapiro andR. C.Feenstra. Chicago, IL: University of Chicago Press, pp. 229–73.)| false
Berndt, E. R.,L. T.Bui,D. H.Lucking-Reiley, andG. L.Urban. 1997. “The Roles of Marketing, Product Quality and Price Competition in the Growth and Composition of the U.S. Anti-Ulcer Drug Industry.” In The Economics of New Goods, NBER Studies in Income and Wealth, edited by T.Bresnahan andR. J.Gordon. Chicago and London: University of Chicago Press, pp. 232–77.
Berndt, E. R.,L. T.Bui,D. H.Lucking-Reiley, andG. L.Urban. 1997. “The Roles of Marketing, Product Quality and Price Competition in the Growth and Composition of the U.S. Anti-Ulcer Drug Industry.” In The Economics of New Goods, NBER Studies in Income and Wealth, edited by T.Bresnahan andR. J.Gordon. Chicago and London: University of Chicago Press, pp. 232–77.)| false
Beuerlein, I. 2001. “The German Consumer Price Index for Telecommunication Services: A User Profile Approach for Mobile Technology and Internet Access.” Paper presented at the Sixth Meeting of the International Working Group on Price Indices, Canberra, April 2–6. http://www.ottawagroup.org/.
Beuerlein, I. 2001. “The German Consumer Price Index for Telecommunication Services: A User Profile Approach for Mobile Technology and Internet Access.” Paper presented at the Sixth Meeting of the International Working Group on Price Indices, Canberra, April 2–6. http://www.ottawagroup.org/.)| false
Bode, B., andvan J.Dalén. 2001. “Quality-Corrected Price Indexes of New Passenger Cars in the Netherlands, 1990–1999.” Paper presented at the Sixth Meeting of the International Working Group on Price Indices, Canberra, April 2–6. http://www.ottawagroup.org.
Bode, B., andvan J.Dalén. 2001. “Quality-Corrected Price Indexes of New Passenger Cars in the Netherlands, 1990–1999.” Paper presented at the Sixth Meeting of the International Working Group on Price Indices, Canberra, April 2–6. http://www.ottawagroup.org.)| false
Boettcher, I. 2015. “Automatic Data Collection on the Internet (Web Scraping).” Statistic Austria. Paper presented at the Fourteen Meeting of the International Working Group on Price Indices, Tokyo, May 20–22. http://www.ottawagroup.org.
Boettcher, I. 2015. “Automatic Data Collection on the Internet (Web Scraping).” Statistic Austria. Paper presented at the Fourteen Meeting of the International Working Group on Price Indices, Tokyo, May 20–22. http://www.ottawagroup.org.)| false
Boskin, M. J. (Chair),E. R.Dullberger,R. J.Gordon,Z.Griliches, andD. W.Jorgenson. 1996. Final Report of the Commission to Study the Consumer Price Index, U.S. Senate, Committee on Finance. Washington, DC: U.S. Government Printing Office.
Boskin, M. J. (Chair),E. R.Dullberger,R. J.Gordon,Z.Griliches, andD. W.Jorgenson. 1996. Final Report of the Commission to Study the Consumer Price Index, U.S. Senate, Committee on Finance. Washington, DC: U.S. Government Printing Office.)| false
Boskin, M. J. (Chair),E. R.Dullberger,R. J.Gordon,Z.Griliches, andD. W.Jorgenson. 1998. “Consumer Prices in the Consumer Price Index and the Cost of Living.” Journal of Economic Perspectives12 (1): 3–26.
Boskin, M. J. (Chair),E. R.Dullberger,R. J.Gordon,Z.Griliches, andD. W.Jorgenson. 1998. “Consumer Prices in the Consumer Price Index and the Cost of Living.” Journal of Economic Perspectives12 (1): 3–26.)| false
Böttcher, I., andS.Sergeev. 2014. “Austrian Scanner Data Project—Report Multipurpose Consumer Price Statistics: The Use of Scanner Data.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 26–8. https://www.unece.org.
Böttcher, I., andS.Sergeev. 2014. “Austrian Scanner Data Project—Report Multipurpose Consumer Price Statistics: The Use of Scanner Data.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 26–8. https://www.unece.org.)| false
Bradley, R.,B.Cook,S. E.Leaver, andB. R.Moulton. 1997. “An Overview of Research on Potential Uses of Scanner Data in the U.S. CPI.” Paper presented at the Third Meeting of the International Working Group on Price Indices, Voorburg, April 16–18. http://www.ottawagroup.org.
Bradley, R.,B.Cook,S. E.Leaver, andB. R.Moulton. 1997. “An Overview of Research on Potential Uses of Scanner Data in the U.S. CPI.” Paper presented at the Third Meeting of the International Working Group on Price Indices, Voorburg, April 16–18. http://www.ottawagroup.org.)| false
Brand, M.,C.Vermeulen,D.Fischbach, andY.Carpy. 2017. “Swiss Residential Property Price Index: The Use of Geolocal-ised Information for Quality Adjustment in Location.” Paper presented at the Fifteenth Meeting of the International Working Group on Price Indices, Eltville, May 10–12. http://www.ottawagroup.org.
Brand, M.,C.Vermeulen,D.Fischbach, andY.Carpy. 2017. “Swiss Residential Property Price Index: The Use of Geolocal-ised Information for Quality Adjustment in Location.” Paper presented at the Fifteenth Meeting of the International Working Group on Price Indices, Eltville, May 10–12. http://www.ottawagroup.org.)| false
Bresnahan, T. F.1997. “Comment.” In The Economics of New Goods, NBER Studies in Income and Wealth, edited by T. F.Bresnahan andR. J.Gordon. Chicago, IL: University of Chicago Press, pp. 237–47.
Bresnahan, T. F.1997. “Comment.” In The Economics of New Goods, NBER Studies in Income and Wealth, edited by T. F.Bresnahan andR. J.Gordon. Chicago, IL: University of Chicago Press, pp. 237–47.)| false
Breton, R.,G.Clews,L.Metcalfe,N.Milliken,C.Payne,J.Winton, andA.Woods. 2016. “Research Indices Using Web Scraped Data.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 2–4. https://www.unece.org.
Breton, R.,G.Clews,L.Metcalfe,N.Milliken,C.Payne,J.Winton, andA.Woods. 2016. “Research Indices Using Web Scraped Data.” Paper presented at the UNECE/ILO Meeting of the Group of Experts on Consumer Price Indices, Geneva, May 2–4. https://www.unece.org.)| false
Burnett-Isaacs, K. 2015. “Using Hedonics to Create Land and Structure Indexes for the Ottawa Condominium Market.” Paper presented at the Fourteen Meeting of the International Working Group on Price Indices, Tokyo, May 20–22. http://www.ottawagroup.org.
Burnett-Isaacs, K. 2015. “Using Hedonics to Create Land and Structure Indexes for the Ottawa Condominium Market.” Paper presented at the Fourteen Meeting of the International Working Group on Price Indices, Tokyo, May 20–22. http://www.ottawagroup.org.)| false
Cage, R.,J. S.Greenlees, andP.Jackman. 2003. “Introducing the Chained Consumer Price Index.” In International Working Group on Price Indices (Ottawa Group): Proceedings of the Seventh Meeting, edited by ThierryLacroix. Paris: INSEE, pp. 213–46.
Cage, R.,J. S.Greenlees, andP.Jackman. 2003. “Introducing the Chained Consumer Price Index.” In International Working Group on Price Indices (Ottawa Group): Proceedings of the Seventh Meeting, edited by ThierryLacroix. Paris: INSEE, pp. 213–46.)| false
Carli, G.-R.1804. “Del valore e della proporzione dei metallic monetati.” In Scrittori classici italiani di economia politica, Vol. 13. Milano: G. G. Destefanis, pp. 297–366; originally published in 1764.
Carli, G.-R.1804. “Del valore e della proporzione dei metallic monetati.” In Scrittori classici italiani di economia politica, Vol. 13. Milano: G. G. Destefanis, pp. 297–366; originally published in 1764.)| false