Trading Model Testing - Time Periods and Markets
by theorangedog on Jan.08, 2008, under Skills
One divide in ideas on trading system development concerns the applicability of trading models to time periods and markets.
The markets consideration is easily explained: Should a model be designed for one product, such as oil, equities, currencies, etc.? That can be drilled down further: Which oil futures (months, products), which equities (sectors, size), and which currencies (USD-pairs, regional)?
The time period consideration is explained in terms of when a model should work: over all market data, a selected time period, a regime based on specified metrics, etc. (For clarity, time period does not refer to time interval, such as 5-minute, day, tick/inconsistent, etc.)
These considerations boil down to this question: Should a trading system work on all time periods and in all markets?
Some say that a truly robust system will have captured an underlying markets law, much like in physics, and will therefore work at all times in all markets. Others argue that a system will work for the regime that it is designed for, and that detecting the regime is part of the system.
I posted this question in a thread in Nuclear Phynance, so you can check there - it is actually a slight threadjack of the “trade optimization” thread in the Trading section.
But, thanks to another Nuclear Phynance member, I came across an article written by Pablo Triana that discusses Jim Simons’ (of RenTec) take on this:
But perhaps the most interesting observation came in response to a question posed by the moderator, Nobel Prize-winner Robert Engle: “Why don’t you publish your research, the theory behind your trading methods? If not while you are active in the markets, perhaps later on.”Simons’ reply – there is nothing to publish. Quantitative investment is not physics. The markets have no fundamental, set-in-stone truths, no immutable laws. Financial “truth” changes constantly, so that a new paper would be needed almost every week.
The implication is that there is no eternal theorem of finance that could serve as an infallible guide through all the ages. Indeed, there can be no Einstein or Newton of finance. Even the math genius raking in $1 billion and consistently generating 30%-plus annual returns wouldn’t qualify. The terrain is just too lawless.
While Simons has a higher level of comprehension on this subject, I would agree with his assessment. Bill Rempel suggested that in this camp, a model may be regime specific, and the system will chose to activate/deactivate the model as needed, in essence making it work across all time periods and all markets. At least, that is what I interpreted from his response at Nuclear Phynance.
Leave a Reply
-
Welcome to theorangedog.net.
This site is my dashboard for links and information on topics related to Skills (Finance), Spirits (Whisky & Beer), and Speed (Races & Training).
-
.
Check these guys out!
January 9th, 2008 on 5:04 am
foquant,
For calrity what are we referring to when we say time period?
January 9th, 2008 on 7:21 am
Refers to the testing period - for example, do you backtest over a 1 year period of data, over 3 years, over the market history, etc.? If a model works over 2 years of data, should it also work over the past 30 in order for it to be deemed “robust”?
While there are heuristics for what is the appropriate amount of data to use, I was very interested in Simons’ response as it accurately sums up the situation - do you believe there is an underlying law in the financials markets?
January 9th, 2008 on 6:15 pm
Yep, you got the interpretation essentially correct. Thanks for the shoutout!
If the timeframe test over is long enough to include all possible regime conditions, and the output of the test is adequate as a gestalt, then it’s truly robust. If one sees on closer examination that the system works better in some regimes than in others, one could overlay a market-timing or regime-switching function to make the system even more robust. A combination of systems is still a system. CANSLIM is an example of this. The “Thermostat” trading system for futures has a trend determinant that switches the system from trend-following to counter-trend based on regime. Etc.
In all markets … it depends on whether the system is fully technical or not. TA-based systems should work in all markets, because they are based on human sentiment, and humans are the opponents in all markets. Systems based on stock market anomalies (valuation, et al) can’t work in bond or commodity markets, although you make parallels to real estate markets.
I think Simons is off base in terms of there being no eternal theorem, I think that the closest thing to an eternal theorem of financial speculation will be found in behavioral finance. His definition of quantitative may be more specific than mine, though, and I doubt that trading spurious regression pairs intraday will ever allow one to see what’s behind broader moves.
January 9th, 2008 on 6:37 pm
Great point Bill,
On an initial read of Simons’ argument, I didn’t think about behavior finance - but systems/models based on that topic would, I think, fall into the quant category.
I agree with the theory that elements of behavioral finance will always be present in the markets, as people will always be present in the markets (even if behind algorithms). Others would argue that because of that reason, it cannot be a “law” in the same sense as Newtonian physics, because it is not derived from “nature” but from “man”.
Still, I have had a good number of trades on market overreactions (e.g. MSFT, MSO, AMR, MRK etc.). How much longer into the future will the ability to profit from overreactions exist? Will the general trading population get to a point where behavioral externalities, such as emotion and an investor’s opinion of a fund manager, no longer impact prices? I think to a degree they will, but those who are fast enough will be able to profit - much like the few news traders that exist today. It is rumored that groups like RenTec and BGI have been working with linguists to parse news feeds for trading - that will make them the fastest (relative to current news traders) and give them the edge, and hence, profit. That could apply to any bias, but will likely always apply - my $0.02.
Thanks for stopping by!
January 15th, 2008 on 12:31 pm
[...] upon what the trader believes they are looking for. Einstein was looking for underlying laws, and from a prior post we already know that Simons doesn’t believe in underlying laws in the financial marketplace, [...]