Here is an example of how important it is to have enough data when backtesting systems.
I was trialling a system and looked at the results for 2014.
There are benchmarks I use such as minimum number of selection to base a method on, at the pointy end of the market, I'd suggest a minimum of 2,000 selections, but I now advocate a minimum of 1,000 winners regardless of market price. I feel this gives a much more accurate picture of chances of ongoing performance. Of course, it going to be pretty difficult to get 1,000 winners out of most methods on outsiders. But that's my thoughts on why so many systems collapse even though "I have 3,000 races and it shows a profit."
The answer invariably is that of the 3,000 races, the profit is based on one or two outsiders.
This is why RaceCensus has a System Viability calculation, which is the profit divided by the maximum win dividend. It tells you how many winners make up your profit.
<pre>2014 Only
WIN PLACE
Bets: 263 260
Winners: 98 176
S.R.: 37.3% 67.7%
Outlay: 263.00 260.00
Return : 262.60 264.54
Profit : -0.40 4.54
P.O.T. : -0.2% 1.7% </pre>
<pre>2007 to 2014
WIN PLACE
Bets: 2902 2889
Winners: 960 1857
S.R.: 33.1% 64.3%
Outlay: 2902.00 2889.00
Return : 2580.90 2713.54
Profit : -321.10 -175.46
P.O.T. : -11.1% -6.1% </pre>