How to properly test a system
 

These ideas/rules are not mine. I have collected them from this forum and made them concise in this post.

There is more to testing systems then throwing a bunch of rules and then hoping for them to hold up in the future. Look for posts by Angry Pixie and wesmip1. These two seem to know how to analye systems.

A couple of pointers.

1. The number of selections needed to cofirm a system is usually effected by the odds of the selections. A system based on favourites might only need 2000 selections for confirmation. I system on long shots might need 10,000.

2. Use the chi square test to determine how luck has affected the results.

3. Test over at least a years worth of results and check the progression holds especially over the last month or two. If your lost in the last month of testing be very careful with it.

4. When testing over at least a years worth of results you want to see at least 8 or more months showing a profit. Preferably 10 or more.

5. Plot out the bar graph of X bets of the POT and check it looks like a bell curve. If its not showing a bell curve you have not tested enough. If it does show a bell curve check the middle of it sits to the right (positive) of 0. X bets should be at least 30 bets or 5/(1/[average odds]) and no more then 5% of the total number of bets.

If it passes all the above tests then you have found a long term keeper.

Excellent posting UB.

Concise it is however, so 1 or 2 questions if I may...

1. How big a sample with a strike of say 75%-80% ?

2. Do you have an equation for this so that we can check different SR's

3. Please clarify your logic regards point 5 in relation to sample size and odds

X bets should be at least 30 bets or 5/(1/[average odds]) and no more then 5% of the total number of bets.


Cheers LG

PS any chance of a resurrection for the system tester in 2013?

Good post UB.

I used to be a big believer in Bell Curves, but thanks to advice from iirc Angry Pixie & other reading etc, not so much these days - provided the chi-test looks ok with a good sized sample that's probably good enough.

Another approach that may help is to drop the top & bottom 10% of SP's from results & see how it holds up.

I go with gut feel but I would need at least 300 for a 75% strike rate.
You could use at least 200 winners as the general rule. So if your strike rate is expected to be 75% then you get a winner every 1.33 races determined by 1/.75. So in this case you would want at least 266 selections.

See above.


You determine your POT for each expected 5 winners as a minimum. So if you had average odds of $10 then it would require at least 5/(1/10) = 50 selections. If your average odds was $2 then you require at least 5/(1/2) = 10 seelctions.

Personally I use a minimum of 30 bets per sequence. Lets assume I havea system which has 900 bets tested. I can divide this up into 30 bets each sequence. You work out the profit for each 30 bets sequence and plot it on a bar graph. You should see most of the POT's be around a certain figure and show a bell curve. If it doesn't show a bell curve then you have not done enough testing.


I do need to get this going again. I haven't had a chance to set it up though.

I think you need to use the chi test but it is suseptible to back fitting. The bell curve analysis ensures that there are enough results tested. If the bell curve is smooth then you should be right. If its jagged or has no shape you are likely to have backfitted results.

Yes, that's the approach I used as well UB, using the Bell Curve confirm Chi-test results to satisfy myself I hadn't back-fitted, however I found that depending on grouping size the bell curve could vary wildly, & that was with large sample sizes (>5000 races, grouping sizes from 25 to 50), so I lost confidence in it.

Thanks for feedback UB.
Also some good balancing points from you NR.

I will venture further into testing the short end of returns from here.

Cheers LG

Nothing wrong with these ides if you have the data to test on, but what if you have some ides you want to test?

A few years back a post was made for testing systems without a data base.

thanks Shaun. Very useful for those who don't collect the data.

The problem with most punters is trying to gather the the sample size, for which some members will try to compensate by testing back 12 years.

I've tried to explain that in getting a true POT, in reality you need thousands of bets for a system to properly gauge its performance. So does that mean you would need quite a few 1000 years to achieve that with a system like one with 20 bets per year?

This isn't possible, and so many people report that their system didn't work when it might be a winning system.

Even if you have a system with 50% strike rate the possibility of having 10 losses in a row is very real possibility

Most people tend to launch into a system after it has exhibited great performance, given that a system has a stable line of performance any deviance ABOVE this line will be followed by a deviance BELOW this performance line. SO guess what most people cop when they start ? Yep, the natural deviance to below the performance line to return the system to its normal average performance.

There is no answer to help you gauge the performance of your systems so that you can be absolutely certain, unfortunately!

So stick with your systems, if that is your punting method. For me I test over 5 years into portions because at the end of the day - the horses don't understand all the figures and trainers have change of styles, weights change and jockeys have bad runs. Each to their own and don't get convinced by others your methods are incorrect.

That is also an exceptional post - A dose of reality, human nature, mathematics, common sense and generosity, all rolled up in one message.

This one is perhaps the biggest clue to overall success on the punt...

*Do what other punters do and do it better than them.
*Do something other punters aren't doing, exploiting a niche.

Thank you Hermes(and Shaun for unearthing such a beauty)

Cheers LG

System Testing
 

A method I've used with some success is to take my d/b (14 yrs), split it in 2 parts, develop the idea on one lump and then run test the produced system on the other. That avoids testing a system on the data that was used to develop it.

demodocus i've been reading quite a bit on creating systems/models for NBA recently and what you've posted relates to what i've been reading.

The guys on there would use 1 set of data to create the system. They would then use a second set of data to test and make any slight adjustments that may smooth the results to be more aligned etc providing it still shows a profit on this data. If its not showing a profit on this data after it did on the original data you used to create the system, then go straight back to the drawing board.

Once the second lot of data even with some tweaks has validated the system, run a final test on a 3rd set of data without any tweaks at all. If you continue to be profitable on this data without much change to the results (SR, POT etc) then your on a winner, if it falls over at this step then back to the drawing board.

Creating systems, ratings, models is not an easy process, if it was everyone would do it and everyone would be millionaires. I've been lucky enough to have some success with my ratings for Australian horse racing, but trying to crack it with a model for the NBA has been near on impossible for me so far.

I think finding a niche where you have some form of an edge is vital. American sports is very difficult to find an edge because the market is so huge and there are so many people betting on it, identifying the sharp lines and therefore creating lines that are very much a 50/50 spread across the season.

This is where Australian horse racing is different, you can find an edge a lot easier IMO. The public are prone to betting on a 100 rater or the top weight or the TAB's tips etc (sometimes all 3 align for a lot of public money). These horses may very well be the favourites for good reason in a race, but it means the odds available are way under the horses actually chance/ability to win the race. This provides opportunity at the other end of the scale providing you can select the right horses at the right time.

I'll post some stats tonight that i've found over the weekend using my ratings that are particularly interesting to horse numbers 10 and below (i.e. lower weighted horses). It's basically shown that whenever my ratings have had a top rater with a TAB number 10 or higher (11,12,13 etc) the POT is huge as its simply an underbet and underrated part of the market.

Anyway thats a bit off track but hopefully that adds to the discussion.

Where are the best places to get hold of the following ....
A: a database
B: system developing / system testing tools

The 2 i know of are both run by members of this forum UselessBettor has a free service run from a webpage, he would have to post the link as it seams to have changed.

Has a lot of variables but a limited database at this time but he is building on it.


If you are after a more private testing ground with a larger database go here http://www.propun.com.au/horse_raci...ng_systems.html this one is run by Chrome Prince and compared to others around is a very good price.

Another way to test a system is run the system in 5 or more batches of 150 bets.

We can then judge the consistency from there.

Another is to break up the race number into odds or evens or days into odd or evens.
If it is heavily skewed one way, more research will need to be done.

FWIW
this is how i go about it.
1. I have designed my data capture so that it can easily be tested. Each starter holds about 300 pieces of information in its form that can be easily filtered out.
2. To sustain in this business you need a very user friendly system and approach.
3. Believe me the punting gods are very cruel and the minute you can’t be bothered and stop because you have had a bad day is the moment the big results is next in line. It has happened to me more than once where I can’t be stuffed putting the Europe night bets on only to get up in the morning and see 3 x 20-1 shots got up. It happened yesterday – near nil return in the day - $2.5k down – had a selection at Limerick – reluctantly put 120 on it and lo and behold – at 12.50AM it wins at $32.40. So to have something that works easy is imperative.
4. The results are incorporated into the data after the race has been completed. both the SP , the dividends and the beaten lengths – 0 lengths for the winner
5. Every horse race covered by Tattsbet is for the last 2 years is in the data base
6. When I have an idea I want to test I break my data into 3 data sets. I massage data set 1 as much as I like until I get a winning formula
7. I then test with absolutely no changes in data set 2. If it’s OK I go to data set3
8. I am always on the lookout for /wary of long priced winners that make it look good.
9. I want to see firstly a continued strike rate and secondly same or better POT.
10. Even after that I’ll let it run for another 6 months before I use my own money
11. I try to look where no-body else might be looking- to improve my price.
12. The all horserace average SP of the winner is around $7.80 and has been for at least the last 25 years that I can see- thus every horse in every race has some chance – by not concentrating on the obvious you improve your dividend.
13. Understanding of probability and “run of outs (‘ROO’)” is important. If you have a 12% SR a ROO of 20 is NOT uncommon.
14. Your bank needs to be about 5000 divided by your SR. so for 12% you need 5000/12 = 416x etc.

good luck every body

aussielong

Hi Aussielongboat,

Looks like you've got it together!

Thanks for such a detailed insight into both your testing and punting approach.

I wish you the best of luck!!

This seems more then reasonable for a betting system. Whats your thoughts on a laying strike rate ?

probabaly use the same formula.
5000/SR

Not so sure this would work.

Lets say we lay selections in the $50 - $100 range and have a 99% strike rate.

5000/SR = 52 which would fail on 1 loss.

Perhaps if your laying to lose an amount it would work ok.

For example

5000/SR = 52 per 1 unit. So to lay to lose $100 you would need at least 5200 bank.

ODDS
 

If each selection was staked to price say to take $100 50-1 $2 9-1 $10 4-1 $20 and so on and if it still made a profit I could not see why it should fail in the future if back tested for any length of time I would give it a good chance of success.

Am I missing the point.

yes that's fair enough if you lost on the first bet - but with you laying 50 to 100 - it would be expected that you should have very few occurrences of losses - big drama when they did come up - but not too many times hopefully.

however don't forget the old rule - unless you have an edge it will end in tears - all yours.

by laying long shots you can put it off for while - but thats where it ends up - and that's a certainty..


any how each to their own -
cheers and good luck
aussie

only if rigorously back tested on fresh data - bookmakers love people with data bases, computers and back testing.

a nip here a tuck there and lo and behold you have discovered the path to eternal riches - until you do it in real time with your hard earned.

"but it all looked so logical - what could have happened" you say :confused: and then :mad:

I have filing cabinets of "good" logical sensible systems that won for 2-3 years and then started a slide and never came back.

And i mean never came back as in over 10 years of later testing.
:mad:

Giday Aussielongboat.

What if all successful systems have bright patches or clear winning runs now and then (timing being variable and unpredictable) and no amount of back testing can change this.. what then?

If this were the case, could we allow for this via 3 step process...

1. continue to rigorously search for successful systems (rather than test them ad finitum to see if they will run hot until eternity?)
2. track them and understand / detect when they are 'on' and when they are 'off', coming on and coming off etc
3. create a portfolio of these and run them together eg some on, some off, others warming up or cooling down etc to increase activity and decrease likelihood of giving it all back to the bookie, tote, corporate, tom etc etc

Just my thoughts.

LG

my view is that if it back tested over a couple of sets of fresh data and it looks OK then it is probabaly OK - i mean what else could you do



probably do both -search and monitor - main edge here is to build an approach where you can data gather, test and monitor efficiently. i run 180 concurrent on every horse race covered by tattsbet and tabulate the results automatically. 1 button macro.

If you could do that regularly you would be set for life and would avoid the gamblers curse - run of outs.
If you avoid the run of outs - you don't destroy your bank, your POT and/or your sanity.

Probably the most difficult pursuit of all
IMHO


that's pretty similia to what I do now - run about 10 different approaches concurrently - hoping that on average i will be in front.
as far as warming up or cooling down etc. i have no idea - they either win or they lose - doesn't seem to be, as far as i can detect so far, that it is that logical or organised.

It just happens or it doesn't happen.

The only thing i have noticed is that if i am going a bit average i usually get some long priced place getters - i.e $8+ for place dividend before i get some recovery winners. i got 3 yesterday so i am a bit chirpy for today.
LOL

what i have also learned so far is that it is "darkest before the dawn" meaning that just when you give up you are on the eve of a recovery - with one BIG proviso - that you are using a tested and proven approach - otherwise it is just down , down and down again and you can end up in debtors prison or the asylum or both.




no sweat all good :cool:

Lots to chew over here Aussie... some brilliant input, thanks.

You being a 'macro maestro' .. to track when systems are on / off / coming good / turning down etc, could it be a simple ss to auto-track SR and watch / learn from the patterns to detect high / low / turning points etc? Could this assist with avoiding the holy crap of punting = a long series of outs??

You seem to already have an idea of one such turning point eg long price place getters which seems to lead / indicate that recovery winners are close by.

Also, have been meaning to ask regards your 1 button macro that churns and crunches thru 180 concurrent variables - why so many??

Cheers LG

Come on LG it's only one button! LOL

re why so many:
these are separate approaches or systems that I monitor. e.g 1 such systems is choosing last start winners when there is only 1 LSW in the race.
in isolation it does not work but with some tweaking it may improve- so I might try 5 or 6 tweaks or combinations of tweaks - they will become 5 or 6 systems/approaches with their own POT and SR that i will monitor and so on it goes to where no i have about 180 such combination or filters etc.
they have to be easy to monitor and easy to develop.
Trust me day in day out all this stuff can get terribly boring - winning helps -but the work still has to be done.

re your first point - yes i have been thinking about that. i feel there must be some sort of lead indicator that will predict good/bad times to come. i have a feeling that it might be looking at trend POT or trend SR's. so that when i have been above or below trend thats the time to do something.

For me its peculiar about writing code for these type of things. i have to be 100% clear in my mind what i want to do and i can do it quickly - but until i get to that point i cant write the code.

At this time i am not at that point so i am just gathering data in such a fashion that when I am ready it will be there for me.

hopefully I will get fired up soon and get back into it.

[QUOTE=aussielongboat]re why so many:
these are separate approaches or systems that I monitor. e.g 1 such systems is choosing last start winners when there is only 1 LSW in the race.
in isolation it does not work but with some tweaking it may improve- so I might try 5 or 6 tweaks or combinations of tweaks - they will become 5 or 6 systems/approaches with their own POT and SR that i will monitor and so on it goes to where no i have about 180 such combination or filters etc.
they have to be easy to monitor and easy to develop.


Hi Aussieboat - When you pick an approach like above where you take LS Winners with only one LSW and test against various other variables. I understand your testing into three batches. Do you start by testing over the 1st batch of years. If the POT shows promise, run the same system for the next period of years and so on.....

Also lets say your period 2 of testing shows a POT of 25% in the 4 years of testing. A loss of 8% in year one, 56% POT in year two, loss of 18% in year three and a lost of 5% in the fourth year. Are you happy with a system producing a profit in 1 out of 4 years?

Sorry for all the questions

People should take note of this as it's one of the few correct statements on the whole forum.

I lay long shots not on the Exchange but on S.P. which prevents a wipe-out if a selection at massive odds wins the race, and in spite of my first loss yesterday I'm well and truly in profit with my two methods.

As for an edge, I hope I've found one by the selection/filter process. However, I don't know without trying, and if I was to test it on paper-only I'd miss the buzz. I fully accept the success might only be temporary and could end in tears, but not to try would be unacceptable to me.

I think I mentioned it here quite a few, months ago that I transferred a total of $10,000, a profit made by laying outsiders, from my Betfair account to my bank account.

The long shot / fav bias which I read about on here shows that long shots are the worst bets on the tab. Betfairs long shots are "usually" longer then the tab though and counteract this bias. But this doesn't happen in every case and you can find threads on here talking about laying selections less then the tab prices and making a profit.

Its all about having the edge whether that be in the shorter end or the longer end of the market.

Hi, U.B.

The edge you are referring to is price-related. Interestingly, a few years ago someone on one of the forums (could even have been this one) said he was making good money by backing on Betfair the horses that had shorter prices than the TAB. However he didn't mention if they were in the top of the market or with all qualifying horses regardless of price. I remember a couple of times whilst I was at my computer I noticed that there were quite a few winners (and placegetters) that started under $10 where the Betfair price was less than that of the NSWTAB.

To do this one has to be at one's computer or have an automated system, and also there may not be much action.

I prefer to lay S.P. early, and I can only do this not by price but system-related that hopefully produces an edge.

Can I be as bold to suggest that there are at least 2 'elements of edge' required for a profitable strategy i.e. selection & staking.. Perhaps in that order?

e.g. what be the point of being able to consistently pick winners (or loosers) when our money management methods are flawed?

I volunteer this as I have spent the best part of 3 months exploring various successful selection strategies, only to find that various additional elements are required to advance, come out ahead on POT, ROC etc

For example...
1. Record ever bet as it is placed
2. Add results and any learning, discoveries
3. Batch (separately record) specific strategies and performance
4. Repeat what consistently works, eliminate what doesn't
5. Focus on and fine-tune methods for selection and staking

Cheers LG


PS have also found that the fastest way to do one's dough is to ignore some / most of these. Would suggest that I am on the right track here, regards my personal preferences etc Each to their own, I guess?

I don't think theres such a thing as staking "edge" in the classic gaming sense of the word - advantage over the house. The amount you stake is/should be a product of the edge offered by your selection. No amount of imagined "staking edge" can compensate for negative "selection edge".

michaelg,

http://www.fileconvoy.com/dfl.php?i...4ec52df6c2 2dc

Available for the next 7 days. These are the prices I have recorded for a bookie (not saying which one atm) and from betfair 5 secs before the jump.

I have only included the first 13K horses. Its a simple testing ground for others to look into to see if they can find anything useful.

If anyone does find anything i would appreciate if it was shared with the forum or me privately at betfinder@exemail.com.au

Was referring to a selection edge or positive expectancy in the first instance, which is then leveraged on or geared up by an aggressive staking strategy.

Both being part of an overall 'edge' in that the combination of the two will produce a superior outcome to Joe P's plan, or lack of etc i.e. we are not talking about an "imaginary outcome" or some kind of "fools gold" where the staking plan is expected to compensate for an inferior selection method.

Rather, both pull together in the same direction.

Cheers LG

yes that's it - the key thing is that you cannot massage it on the 2nd or 3rd sets of data - if you do that you are data fitting.
data fitting IMHO never really works



probably not as i could imagine the Run of outs would be a big number in there somewhere.
also 1 in 4 means there was probably 1 -2 LS winners in y2 somewhere.

however in more detail - it would depend on the ongoing SRs, the place POT and whether in Y2 there was a few or only one LS winner that bumped up the POT.
i also like to look for winning months greater than 7/12
the higher the winning months % - the higher my confidence.

i am always looking for consistency as i feel that if that exists the results are more likely to be repeated.

4 & 5 above IMHO fall into the category of back fitting data.

using all the logic in the world you can filter out some non performing criteria - but be careful as those "non performers" in the fullness of time can become performers.

filter them out by all means - but then in the future examine them in isolation and i'll bet most of the time they will make a come-back - well that's been my experience.


cheers and good luck