Assessing Accuracy: Hotel Horizons Forecasts

April 13, 2016

​EXECUTIVE SUMMARY

  • The U.S. hotel industry had another strong year in 2015 with RevPAR up 6.3 percent over the prior year. In this report, we examine how accurately Hotel Horizons® predicted this performance as well as the performance of previous years. Hotel Horizons® econometric forecast models rely on economic forecasts from CBRE Econometric Advisors to guide our hotel market forecasts. There were a small number of errors and economic forecasts did not meaningfully affect hotel market forecasts.
  • For the nation, Hotel Horizons® chain scale and location RevPAR forecasts had a mean absolute error of 1.56 percent and 1.33 percent from 2011-2015. Specifically in 2015, these errors were 1.67 percent and 1.02 percent. Both indicate strong forecast performance for the year.
  • MSA level RevPAR forecasts show mostly similar levels of performance in terms of absolute errors. In 2015, 31 of the 50 markets examined here had an absolute error of less than three percent. Sixteen of these had absolute errors under one percent, indicating extremely high levels of precision in forecasts of those markets.
  • As expected Hotel Horizons® forecast accuracy for 2015 improved over time from 2011 to the final quarter forecast in 2015. The mean absolute error of the one-year ahead forecast across markets was 2.6 percent in 2015, down from 3.3 percent the prior year. Over the course of the year, as new forecasts were released, this error fell to only 0.8 percent by the final forecast produced for 2015.

1. INTRODUCTION AND OVERVIEW OF U.S. HOTEL MARKET FUNDAMENTALS SINCE 2011

This report contains an accuracy self-assessment of the CBRE Hotels|Americas Research Hotel Horizons® forecasts. This assessment is developed from comparisons between forecast changes in hotel market performance and actual changes during the period 2011 through 2015. We also compare our econometric forecasts to forecasts from an intuitive scenario. Previous analysis were conducted during 2005, 2010, 2014, and 2015.1 As part of the 2016 assessment, we examine outcomes from the industry recovery that took hold in 2011 and the impact of economic events on hotel market forecasting accuracy. Specifically, the study analyzes Hotel Horizons® forecasts released during the first quarter of each year from 2011 through 2015. We analyze short-term accuracy defined as rolling one-year forecasts versus actual results for revenue per available room (RevPAR), average daily rate (ADR), occupancy (OCC), demand (i.e., rooms sold), and supply (i.e., rooms available). Long-term accuracy over the five-year interval also is reported. The exhibits contained herein display information about annual forecasting accuracy for all U.S. hotels, hotel chain scales, across locations, and for 50 of the 59 Hotel Horizons® Metropolitan Statistical Area (MSA) markets2. We rely on various statistical measures, such as mean absolute error and Theil's U statistics, to assess accuracy.3

The period covered in this self-assessment was a period of modest, mostly steady economic activity, highlighted by a number of meaningful dips on the road to economic recovery. Changes in general economic conditions brought about coincidental, but not necessarily synchronous, movements in hotel market performance measures. This response is reflected in the performance history for U.S. hotels shown in Exhibit 1. Between 2011 and 2015, RevPAR growth for all U.S. hotels increased each year, but at uneven paces from one year to the next. Both Upper- and Lower-Price hotels experienced these fluctuations through growth deceleration from 2011-2013 followed by reacceleration after 2014 only to slow the pace of growth again in 2015.
 
The implications for forecasting in conditions such as these are twofold. First, entirely unexpected events, such as the 2012-2013 federal budget crises, cannot be woven into forecasts performed prior to such events. While the economy was in a steady recovery, not many experts predicted the political risks created by the U.S. Government flirting with financial default in 2012 and 2013. These political events cannot be forecast from economic data. Second, modern forecasting models perform best during prolonged cyclical up phases and down phases and perform their worst around turning points. The rapid decline in oil prices as well as the sudden rise in financial market volatility during the latter half of 2015 were largely unexpected and may have influenced hotel performance in unanticipated ways. The remainder of this report is organized as follows. Section 2 describes how CBRE Hotels' Americas Research approaches hotel market forecasting including discussions about the data used, the statistical techniques employed, and the potential sources of forecasting errors. In Section III, we provide details about the measures introduced to assess the accuracy of Hotel Horizons® forecasts. Sections 4 and 5 present the results of our assessments for all U.S. hotels, chain scales, locations, and 50 of the 59 Hotel Horizons® MSAs we cover, respectively. The final section gives a summary of our forecasting effort during the period 2011-2015.
 

2. HOW HOTEL HORIZONS® FORECASTS ARE PRODUCED

CBRE Hotels Americas Research prepares forecasts of the hotel markets in the U.S. based on generally accepted econometric procedures and sound judgment regarding fundamental relationships of the economic and behavioral market indicators to hotel financial performance. These relationships have been tracked by CBRE (formerly PKF Hospitality Research) for over 70 years. The models that underlie econometric forecasts rely on statistical linkages estimated with historical data that come from actual market transactions involving individuals and firms interacting in hotel markets.

2.1 Econometric Models

The Hotel Horizons® econometric forecasting models, because they represent an entire sector of the national and MSA economies, fall into the category of multi-equation, demand and supply models. These models have a general structure as defined below but vary in their form for particular market applications. The three estimated equations are:

  1. Demand - The number of rooms occupied (i.e., accommodated demand) is the dependent variable in this equation which is explained by either gross domestic (metropolitan) product, real personal income, or total employment, which serve as the primary independent variables, along with the lagged changes in any of these variables and the lagged demand from the prior year.4
  2. Supply Change - (change in the number of rooms available) is the dependent variable, which is explained by real ADR and OCC, serving as the main independent variables along with the change in supply from the prior period.5
  3. ADR (Real) - is the dependent variable, which is explained by occupancy, the primary independent variable, along with ADR from the previous period.6
These equations are estimated with ordinary least squares in a non-simultaneous fashion using data from STR, Inc. and CBRE Econometric Advisors (CBRE EA) dating back to the late 1980s. The parameters (coefficients on each variable) then are used to forecast demand, supply change, and ADR by multiplying the parameters by CBRE EA forecast of the economic variables and relevant previously estimated values (lagged variables). Three additional calculations are made with these results as follows:

  1. Supply change is added to the previous period number of available rooms to produce an available rooms level in future periods.
  2. Number of rooms sold is divided by number of available rooms to obtain occupancy percent in each future period.
  3. Expected inflation is added to real ADR to convert to nominal ADR.
Regression equation estimations using time-series data, such as the work done to produce Hotel Horizons® forecasts, may have an econometric problem known as autocorrelation. For each of the equations estimated by CBRE Hotels' Americas Research, we run tests to detect the presence of autocorrelation, and if the problem is found, corrective measures are introduced.7

2.2 Judgmental Intervention

The econometric models predict future room supply in small increments (e.g., 50 rooms per quarter). In reality, rooms typically enter and leave markets in larger blocks (e.g., 300 rooms) as new hotels are placed in, and removed from, service. When, for example, it becomes apparent that a new hotel(s) will be put in service within the next 18 months the modeled supply change will be manually adjusted to account for the opening of the new hotel. The reverse also is true when it becomes apparent that a hotel(s) are taken out of service (e.g., demolished or converted to an alternative use). These assessments of near-term supply changes are made by locally based CBRE Hotels consultants working in the various offices across the U.S.

Finally, a committee of hotel experts from CBRE Hotels Americas Research performs a thorough review of each model prediction. Locally based consultants throughout CBRE also participate in these reviews. The quarterly forecasts for the current year, as well as the annual forecasts beyond the current year, are subject to review. This committee modifies predictions from the model when there is compelling evidence that factors have come into play in a market that the model could not possibly foresee. A Katrina-style event, as an extreme example, would cause the committee‟s forecast to differ noticeably from the model's prediction in not only the MSA in which the event occurred but also competing MSAs within the region. In most instances however, the committee either defers to the model prediction or makes modest adjustments.

2.3 Data Sources and Issues

The forecasts utilize historical data from STR beginning in Q1 1987 and involve three performance measures – rooms available, rooms occupied, and rooms revenue. Using these measures, we compute three additional measures - ADR, occupancy percent, and RevPAR. The STR universe, more than five million rooms, represents a majority of the hotel rooms in the U.S. As they modify the census over time, analysts at STR alter the historical record. Hence, in producing this accuracy assessment, we made certain that forecast results and histories are in synchronization. The second important data source for Hotel Horizons® forecasts is CBRE EA. The vast array of economic variables provided by this firm, both at the national and MSA geographic strata, provide a rich testing environment for the development of stable relationships between economic and hotel market experiences. We use the historical information from CBRE EA and STR to build regression equations. Next, we use the forecasts of economic variables from CBRE EA to forecast hotel demand, supply, and ADR; and then compute Occupancy and RevPAR.

An implication of using CBRE EA forecasts is that our forecasts pick up errors from their models along with errors from our models. Exhibit 2 presents errors from CBRE EA's 2015 annual employment forecast of the 50 Hotel Horizons® MSAs included in this report. The impact of these errors on Hotel Horizons® forecasting accuracy is not measured here in a direct way. We see that the forecast errors for CBRE EA MSAs are quite small (within one percent). We believe these minor errors had minimal impact on the Hotel Horizons® forecast error.


 

3. ACCURACY ASSESSMENT METHODOLOGY

Assessing the accuracy of forecasts involves an analysis of errors, and often an examination of the sources of those errors. The 2016 version of CBRE Hotels Americas Research self-assessment of Hotel Horizons® forecasts involves both the investigations of absolute errors – those from taking differences between actual performances realized after forecasting and forecast performance made before realizations – and relative errors – those from taking differences between Hotel Horizons® forecasts and forecasts from an intuitive approach.

3.1 Absolute Measures

The errors generated from Hotel Horizons® forecasts, e, may be defined as the differences between actual hotel market results, A, reported each year by STR and the CBRE Hotels Americas Research forecast numbers, F, such that
An appropriate way to represent these differences involves not allowing negative and positive „misses‟ to cancel each other. The effect of the mathematical signs needs to be removed either by taking the absolute value of e (|e|) or by squaring e (e2). Thus, the mean absolute error (MAE) can be defined as



Also, the mean square error (MSE) can be defined as



Another absolute accuracy assessment measure is Theil's U1 statistic. This statistic ranges between zero and one. The closer the statistic is to zero the better the forecast accuracy (see Appendix A for details).

3.2 Relative Measures

Forecasts are often evaluated in relative terms as well as absolute terms. Typically, the forecast results generated by the theoretically preferred model are evaluated against results from an alternative, more simply conceived set of assumptions about the future referred to as the intuitive model. To construct intuitive forecasts against which to compare Hotel Horizons®, we use rolling three-year historical averages to represent intuition-based expectations.

Theil's U2 statistic provides a measure of relative accuracy. This statistic is centered on 1.0. If the U2 statistic is less than 1.0, the preferred/sophisticated method results in the more accurate forecasts (see Appendix A for details).

4. ACCURACY OF U.S. AND CHAIN SCALE, AND LOCATION FORECASTS

The five panels in Exhibit 3 provide graphical representations of the March 2011 and March 2012 Hotel Horizons® forecasts of the five hotel market performance metrics – RevPAR, ADR, OCC, demand, and supply – for the U.S. along with the actual results. These graphs support the point made above as to the inherent inaccuracy of forecasts produced during cyclical turning points in the market. Our 2012 forecast more closely fit actual performance, as the economic recovery was well in place. The mostly steady, though modest, economic growth that occurred from 2012-2015 allowed our models to work most effectively.

Forecasting demand requires some insight as to the position of the economy in the business cycle. In 2011, our models suggested the economic recovery would not be as long lived as actual experience. Thus, we see overly modest demand forecasts in the out years. By 2012, however, economic fundaments were in place allowing the models to predict a longer expansion period correctly. Forecast and actual data show that models closely track actual performance during the period of economic recovery from 2011-2015. The same story holds true for RevPAR and Occupancy as demonstrated in the first four panels in Exhibit 3.

Hotel supply change is a special case relative to other market measures given the nature of hotels as durable assets.8 Supply additions from construction and reductions from removals create patterns through time that ordinarily do not conform to the business cycle.9 While supply is the most stable of market measures, building supply forecasting models is challenging. Panel E of Exhibit 3 demonstrates this point. Both forecasts of supply were slightly off, predicting more rapid increases in capacity than materialized. Following the real estate boom/bust, financial crises, and Fiscal Cliff crisis in 2013, supply remained below model predictions reflecting a lack of access to financing for construction or caution on the part of real estate investors about committing equity capital to hotel development given the prior pain inflicted on the industry by the 2008 financial crisis.




4.1 Chain Scales

The hotel inventory in the U.S. can be subdivided in many ways. Two common delineations are by market segment for chain-affiliated hotels and by property locations. Each quarter, Hotel Horizons® forecasts are prepared for these two categorizations. Exhibit 4 presents forecast accuracy in two panels – one for chain scales and the other by locations. The hotel populations differ between these two subdivisions in that the chain scale delineation does not include independent hotels while the location subdivision includes independents. The U.S. hotel industry is comprised of approximately 70 percent chain-affiliated hotels and 30 percent independents. Hence, the chain scale subset is somewhat smaller and tends to include larger, higher quality properties than the location subset.


 
The forecast accuracy assessed in Exhibit 4 is for annual RevPAR growth rates at the end of each year from 2011-2015 which were published in March of that year. Each panel includes results from forecast comparisons throughout the analysis period reported using three metrics.
 
Mean absolute error, defined above in Equation (2), is a standard measure of absolute forecast accuracy (i.e., direct comparison of forecast results to actual results). The remaining two measures reported in Exhibit 4, Theil's U1 and U2 statistics, are presented in the text above and formally defined in Appendix A.

For chain scales, the average MAE for the analysis period is quite good at 1.56 percent. This measure varies as expected by year from 2011 through 2015. The small MAE each year indicates a high level of accuracy during the expansion years of 2011-2015. From the U1 analysis of 2011-2015, the order of accuracy is: upper upscale, economy, luxury, upscale, upper midscale, and midscale. The forecast accuracy was high over the periods examined here, yet in 2015 some chain scales did produce increases in U1 as financial volatility began to creep back into the macroeconomy during the second half of the year. From the U2 analysis (i.e., comparing model forecasts to the non-econometric forecasts), the order of forecast accuracy is: economy, upper upscale, upper midscale, midscale, luxury, and upscale. From 2011–2015, almost all Hotel Horizons® forecasts were superior to the non-econometric forecast for all chain scales. Similar to the U1 analysis, the 2015 economic volatility proved the most challenging for modeling future hotel performance as uncertainty crept back into the macro economy. Nevertheless, the non-econometric forecasting approach tested here is not as accurate as the combination of econometric modeling and expert judgment underlying Hotel Horizons®.

4.2 Locations

For locations, the average MAE for the period is good at 1.33 percent, conforming to our chain scale findings. From the U1 analysis of 2011-2015, the order of accuracy is: resort, suburban, small metro/town, airport, interstate, and urban. The relatively stable performance reflects the steady economic conditions which have occurred over this period. From 2011 to 2015, accuracy was sustained at a very high level for all locations.

Almost all Hotel Horizons® forecasts were superior to the naïve forecast from 2011 through 2015 based on the U2 analysis. Over the five year period the order of relative (i.e., to the naïve approach) accuracy is: resort, suburban, small metro/town, airport, interstate, and urban. As in the case of the chain scale forecasts, 2015 proved to be the most challenging for modeling hotel performance, and the Hotel Horizons® forecasts are more accurate than the naïve forecasts for most locations.

5. ACCURACY OF MSA FORECASTS

The Hotel Horizons® universe covers 59 of the largest Metropolitan Statistical Areas (MSA) in the U.S. in terms of size of the hotel market. These area-specific forecasts are generated each quarter and for each market for the aggregate categories of upper-price and lower-price hotels.10 We evaluate the accuracy of 50 of our MSA forecasts along three dimensions – short-term (i.e., one year forward) during the 2011-2015 evaluation period, rolling forecasts starting with a five-year horizon and continuing in one year increments to a one-year horizon, and one-year forecasts for 2015 which corresponds to the a period of relative hotel market stability.

5.1 One-Year MSA Forecast Accuracy

In Exhibit 5 we present evidence of the accuracy of Hotel Horizons® forecasts of MSA RevPARs for one-year periods beginning in 2011 and ending in 2015. Specifically, the evaluation covers forecasts made in March of each year for that calendar year. The Theil U-statistics appearing in this exhibit are averages of Theil U-statistics computed using the single year comparisons of (1) Hotel Horizons® forecasts vs. actual RevPAR (i.e., U1) and, (2) Hotel Horizons® forecast vs. naïve forecast (i.e., U2).



The column averages of U-statistics shown at the bottom of the exhibit indicate that the one-year Hotel Horizons® forecasts over this period closely align with realized RevPARs across the 50 MSAs and that the naïve forecasting idea we tested is not as accurate as the Horizon forecasts. These results are good given the volatile economic recovery experienced during the period. While most of the individual MSA U-statistics resemble the 50 city averages, a few areas proved more problematic than others did from a forecasting perspective. These are:

  • Miami – A naïve forecasting approach turned out to be slightly better than our models over the period, yet the small U1 statistic shows that our forecast produced a reasonably good result.
  • Pittsburgh – The intuitive approach performed as well or better than the Hotel Horizons® forecast for all hotel segments likely driven by the poor performance of the Hotel Horizons® upper-priced segment forecast.
  • Washington DC – The Hotel Horizons® model for all hotels struggled somewhat in this market as indicated by the U statistics. This weakness is likely the results of the large impact the federal government has on the DC metro area economy. Specifically, the DC market suffered during unexpected shifts in federal budgets over the period.

5.2 Rolling Forecast Accuracy for MSAs

Another perspective on MSA accuracy involves an analysis of 2015 forecasts beginning with the publication of RevPAR percent change estimates made in March 2011 for 2015 then rolling forward up to publication in December 2015 (i.e., using data through 2014 Q3) to forecast year 2015 performance. This analysis serves two purposes. First, it allows readers to evaluate our long-term forecast accuracy (i.e., up to five-year forecasts made in 2011 for 2015). Second, this rolling accuracy report indicates how consistent the forecasts were through the study period.

The top row of Exhibit 6 presents the MAEs for each 2015 forecast published at annual intervals from March 2011 through March 2015 and then at quarterly intervals during 2015. As expected, the size of the errors is directly related to the length of the forecast. The MSA forecasts made in March 2011 produced a MAE of 5.3 percent, and the shortest-term forecast for year-end 2014 made in December 2015 (i.e., using data through 2014 Q3) generated an MAE of less than one percent.

The March 2011 MSA forecasts produced some large misses in certain MSAs, for example, Tampa (3.0% vs. 13.8% actual) and Houston (5.2% vs. -3.3% actual). It also resulted in excellent predictions in some large markets, for example, Philadelphia (4.9% vs. 5.5% actual) and New Orleans (4.3% vs. 3.4% actual). Economists at CBRE Hotels' Americas Research have reconfigured the models in MSA markets in which the forecasts produced less than acceptable results. Backtesting indicates that the new models perform far better than those used for this assessment and should result in smaller differences in the future.
 


5.3 MSA Forecast Annual Accuracy

To gauge medium term accuracy, we examine our absolute errors of forecasts produced of the beginning of each year with the actual results that occurred in that year. This gives insight into how accurately forecasts predict the upcoming year. Exhibit 7 shows absolute differences for each MSA examined in this report for each year from 2011-2015. The results show that while some markets may have higher errors in individual years that may result from idiosyncratic shocks, the annual MAE of market forecasts improved in 2015 in comparison with the two prior years. Overall, in the period studied here annual forecasts are accurate within an average range of 2.5%-3.5% with the exception of 2011 in which the macroeconomy was still adjusting from the prior recession. This means Hotel Horizons® forecasts on average are quite accurate in any given year.

In 2015, 31 of the 50 markets examined here had an absolute error of less than three percent. Sixteen of these had absolute errors under one percent, indicating extremely high levels of precision in forecasts of those markets. Relatedly, markets in which we struggled in the prior year we were able to rectify by through changes in our models to improve accuracy. Two notable examples of this are Denver in which our absolute error fell from 10 percent to 0.4 percent in 2015, and similarly our Nashville forecast error declined from 9.7 percent in 2014 to only 0.3 percent in 2015.
 

5.3 2015 MSA Short-Term Forecast Accuracy

Finally, we report MSA forecast accuracy for the year-end RevPAR percent changes realized from the forecast produced in March of 2015 (i.e., using 2014 year-end data). As shown in Exhibit 7, both U1 and U2 statistics provide evidence of accurate forecasting. All of the U1 statistics are less than 0.50 across the 50 MSAs except four cities, and only five of the U2 statistics exceed 1.0. The Hotel Horizons® method was the least effective for this stable-year forecast, mainly in the same MSAs as discovered when assessing the five-year forecasts. By contrast, the simple difference between 2015 forecast and 2015 actual RevPAR change is less than one percent in the 15 cities we cover and less than two percent in 20 cities. Our forecasts mostly proved to err on the side of caution as 35 of the 50 MSAs covered had actual performance that exceeded our forecast.
 








6. FORECAST ACCURACY SUMMARY

The method used to prepare Hotel Horizons® forecasts produces accurate hotel performance forecasts across large geographic areas, market segments, and locations. The accuracy of the forecasts is quite good for the U.S. hotel market and the sub-categories at the national level that the hotel industry uses to identify different hotel types – chain scales and locations. The chart presented in Exhibit 9 demonstrates Hotel Horizons® forecast accuracy for RevPAR from one quarter out for the immediately following quarter. The ninety-three percent r2 indicates that CBRE Hotels' Americas Research's near-term forecasts at the national level are highly accurate throughout the entire period analyzed.
 

Forecasting accuracy is shown to be uneven over the 50 included MSAs in this report, with quite large differences between forecast and actual results in certain cases, and differences well within a tolerable range of error in most cases. Work has begun to improve forecasting methods in those areas. This work will be completed well before our next accuracy report.
 
___________________________________

1 See www.cbrehotels.com
2 Nine of the 59 Hotel Horizons® MSAs are not included in this assessment because they are new markets to the Hotel Horizons® portfolio. Therefore, these markets contain no previous forecasts to compare against their current forecasts.
3 These measures are defined later in this report and in Appendix A.
4 Different numbers of lags are used for independent variables based upon statistical significance.
5 Ibid.
6 Different numbers of lags are used for independent variables based upon statistical significance.
7 We do not test for, nor expect, distortions resulting from another issue sometimes encountered when performing econometric analyses with time series data known as spurious correlation.
8 Durable assets, especially operating real estate, cannot be immediately produced when demand increases (i.e., known as delivery lag) and are not immediately taken out of service when demand shrinks, particularly when a downturn is viewed as temporary.
9 The supply cycles for hotels and some other property types appear to operate independent of the business cycle that determines demand. See William C. Wheaton, 'Real Estate “Cycles”: Some Fundamentals.' Real Estate Economics, 1999, 27: 209-230.
10 A list of these MSA markets can be found at www.pkfc.com. Upper-price hotels include the chain scale divisions luxury, upper- upscale and upscale; lower-price hotels include economy, mid-price, and upper mid-price chain scales.


FOR MORE INFORMATION REGARDING THIS ARTICLE, PLEASE CONTACT:​ 

​Brett Edgerton ​Jamie Lane
​Economist ​Senior Economist
CBRE Hotels’ Americas Research CBRE Hotels’ Americas Research
​brett.edgerton@cbre.com ​jamie.lane@cbre.com
 

APPENDIX A
Econometrician Henri Theil during the 1960s and 1970s developed two statistics for measuring the accuracy of forecasts – U1 and U2. Theil's coefficients are derived using changes rather than levels in order to avoid the inflated view of accuracy. For U1, the values are bounded between 0 and 1, with values closer to 0 representing greater precision. For U2, values less than 1 show that the forecasting technique used is better than the naïve forecast and values greater than 1 demonstrate that the technique is worse than the naïve forecast. When U2 equals 1, there is no difference between the methods.

Equations for Theil's U1 and U2 statistic: