In a new research publication, “Joined at the hip: ETF and index development”, Vanguard claims that an increasing focus on non-capitalisation-weighted benchmark launches and on the use of back-tested data risks painting an overly optimistic picture of the potential performance of new index concepts.
In their study the authors, Joel Dickson, Sachin Padmawar and Sarah Hammer, focused on 637 indices that served as benchmarks for exchange-traded funds and which also “went live” after the year 2000.
Vanguard observes an increasing focus on non-capitalisation-weighted index methodologies for use by ETFs since the turn of the millennium. In 2000, 2001 and 2002 almost all the new indices created for ETFs followed the traditional method for compiling benchmarks, weighting by individual securities’ market size, Vanguard says. However, in recent years “alternative beta” index launches to underlie ETFs have outnumbered cap-weighted index launches for the same purpose, the firm adds: last year, 2011, by almost three to one.
Amongst alternative beta approaches are a number of different index concepts, notes Vanguard: methodologies that weight constituents by momentum, price and volatility; “characteristic-based” weighting schemes based on company earnings, revenues, fundamentals and dividends; and other methods that weight index constituents equally, in tiers, or by proprietary or multi-factor selection processes.
Index providers are also creating new benchmarks at an accelerating rate, and without waiting to see how well a concept performs in practice, says the fund management firm. The median time between index and ETF creation has decreased from almost three years in 2000 to just 77 days in 2011, says Vanguard, meaning that most indices now have little live performance history for investors to assess.
In its study, Vanguard also examines the use by index providers of back-tested performance data. Vanguard notes that while fund management firms are not allowed to use back-filled performance data in their own marketing materials under US regulations, an index provider that is not affiliated with a fund or an ETF tracking its index is typically not subject to such rules and can make more liberal use of hypothetical performance data.
However, this practice presents a serious risk of investors being offered a potentially rose-tinted view of an index’s ability to perform, notes Vanguard, backing up its assertion with a study of performance data, both “live” and back-filled, from a broad selection of indices.
Vanguard says its study shows conclusively that the vaunted superior performance of back-tested indices did not continue after the relevant indices went live.
Of the approximately 1,070 indices that serve as ETF benchmarks, Vanguard says it was able to obtain “index-live” date information for 775 of the indices, either from index providers’ or ETF sponsors’ websites.
Of this total, Vanguard says it isolated 370 indices that had at least six months of back-filled data and six months of live data over the period from 2000 through 2011.
The firm says its analysis shows that, while 87% of the indices outperformed the broad U.S. stock market for the period during which the back-filled data were used, only 51% outperformed the broader market after the index went “live”.
In fact, says Vanguard, if you look at the historical average performance against a broad US equity benchmark of the 370 indices that use back-filled data, for the five years before and five years after the respective indices were launched, you actually find that the vaunted outperformance claimed by index back-tests turns into underperformance once indices see the light of day.
The average annual “outperformance” for back-tested indices of 10.31% during the five years before their respective live dates becomes an underperformance of nearly 1% a year in the five-year post-launch period, Vanguard calculates.
The fund manager’s assertion therefore appears to be that many index firms and ETF marketers are involved in a large-scale data mining process, with the objective of isolating those index concepts which look good in the rear-view mirror and seeing if funds can be launched to track them.
“A major reason for the outperformance of back-filled data is likely that sponsors tend to develop new products based on what has performed well recently,” Vanguard states in its study.
As things stand, many investors are falling for the back-test mirage, Vanguard observes. The firm notes that cash flows into ETFs that are based on indices with back-filled data are bigger if the index has a relatively short live performance history, suggesting that investors have less time to compare hypothetical with actual performance.
All index investors should ask themselves one simple question, suggests Vanguard: “Are back-filled data a reliable indicator of how the index will perform after it is launched?” The fund manager says that the answer to this question is an unequivocal no.
More worryingly, says the firm, index and ETF development may be increasingly dominated by firms’ marketing departments, who are keen to latch onto any index concept that might attract cash, irrespective of its investment merits.
“Index creation activity has been transformed from an exercise of providing investable benchmarks for different asset-class segments to one of providing ETFs a way to market and promote new products with ready-made indices that might jump-start the acceptance and viability of new offerings,” Vanguard argues in the conclusion to its study.
