I have seen quite a few messages here on the dangers of backtesting to develop/prove systems so thought I would share the system I have used which (in my opinion) allows you to backtest with some confidence.

Firstly to backtest with any confidence you need a good sample size - I personally have a database with several years data (although the older data has some limitations so some of my testing is limited to the more recent data). I think you need at least 6 months data and preferably a full year.

Divide your sample data into 2 parts - doesn't really matter how - for simplicity you can put the first half of the data in "Part A" and the other half in "Part B". If you are concerned about seasonal influences then put alternate months in each group (eg. Jan, Mar, May in "A"; Feb, Apr, Jun in "B")

Now you use the "Part A" data to develop and test your ideas - when you have come up with something that seems to work you can then test it against your "Part B" data. When you are doing this testing you MUST apply the rules/system you have come up with as if you are betting with real money and you should make your bet decision based on the form, etc without looking at the results first.

If your system shows a similar profit against the "Part B" data then you should have some confidence that it is valid and hopefully will work in the future as well. If it makes a loss or if the profit is significantly lower than on the "Part A" data then you should go back to the drawing board and revise your system (again using the "A" data for development and the "B" data for testing.)

One other thing to be aware of is that you should always use the same type of data for developing a system as you are going to use when betting. So if you are only going to bet on Saturday Metro meetings then only use Saturday Metro data for development and testing.

As always I am happy to try to answer any questions or comments.

I would like to know more.

Benny

Yeah, some good ideas there.

Ditto.

i think that 6 months data is irrelevant. any thing can happen within that 6 month period. i prefer to use a minimum of 5 years to determine if something is going to work. this way you have looked at many thousands of selections and many, many thousands of horses. what i do agree with is your pattern of checking. i use 1997 to 2002 as my basis data for checking my concepts. if it has worked at say a 15% profit, then i check how that idea worked over the full year of 2003. if it still stacks up over that 12 month period...then i think i may be on to something.

markallan

markallan,

Obviously the more data you have access to, the better - a lot of people don't have 5 years data to work with, so I think becareful was giving a minimum criteria.

One thing I've been using is looking at how filters perform over various systems.
If a filter only works or improves things for one system but not another, then perhaps the increase in strike rate or POT comes from somewhere else.

One thing I have noticed, is that obvious filters and criteria do not boost POT as every "Harry" with a formguide can use this strategy. For example horses that won their last start are far less value than horses that were beaten by less than a length, similarly horses that ran a place are far less value (in general) rhan horses that ran 4th or 5th but beaten by less than 2 lengths etc.

Howdy critters,

I thought I'd wade in with my 2c worth.

For starters, any form that is more than 12mths. old in any horse is pretty much worthless info and meaningless stats, so probably the only stats. worth anything much are those last 12 mths., but only for use over the next 12mths. while most of the horses they were based on are still in the game.

How a system performs over the next 5yrs. will have little relationship to how it performed over the last 5yrs. because all the horses will be newbies.

% variables on a system that won 10/20% over the past 5yrs. will have swings large enough to make even a 10yr. average back-fitted performance, all but next to meaningless as far as it's performance over the next 12mths. goes.

It is worth remembering that this game is 70% [approx] a game of chance that is subject to wild swings of our ability to predict outcomes from available variables.

A back-fitted system result that shows say 15% POT over the last 5yrs. may make 15% [or more] over the next 5yrs., loose 15% [or more], but how it performs over the next 12months [ most important ] could be anyones guess.

The fact that anyone can approach a system that has shown a profit over the last year/5yrs. anyway, with some sort of smug glow of security [ radiating downward toward the hand that removes the wallet, that will then remove the notes that are going to do a magic trick of multiplying themselves based on the secure 'wisdom' gleaned from shonky maths applied to shonky facts ], that they are safe from a financial mauling, beggars belief !!!

Cheers.


[ This Message was edited by: crash on 2004-03-22 07:22 ]

Crash,
We've had our differences, but on this I have to totally agree with you.

Every race is different depending on circumstances.

How can I back horses FOR A PLACE that have never been out of a place at the distance, never been out of a place at the track, and is down in in class from their last start, and the same jockey has ridden it at it's last 5 or so starts, and they run a shocker.

The reason is that this is horse racing and there is ALWAYS the unforseen.








[ This Message was edited by: kenchar on 2004-03-22 19:09 ]

[ This Message was edited by: kenchar on 2004-03-23 08:37 ]

Hi kenchar,

I think that the difference is the expectation of what stats will do.

Stats will not predict the outcome of Race 1 at Flemington with any accuracy, although they might point to value.

As you say, every horse is independant, as is every race, condition jockey and result etc.

However, I can say with some accuracy, that data can predict the OVERALL strike rate or profit given enough bets.

For example, if one in three households have a computer, that's 33% strike rate - this does not mean that if I walk into three households one of them MUST have a computer, it means that if I walk into 100 houses, roughly 33 will have one.

Forgive the crude analogy.

__________________
RaceCensus - powerful system testing software.
Now with over 412,000 Metropolitan, Provincial and Country races!
http://www.propun.com.au/horse_raci...ng_systems.html
*RaceCensus now updated to 31/12/2024
Video overview of RaceCensus here:
http://www.youtube.com/watch?v=W821YP_b0Pg

You have only one problem Chrome,

nobody will give you odds for your money on predicting 'average' outcomes !!!

Ok, so we know that 30% of favorites will win over the next 12 mths. So what ?

No bookie will take your bet on it, but he'll take your money on the favorite of the next race !!!

Cheers.

PS. Hi Kenchar, good to see you around. We probably agree on more things than disagree, so that's all good news.