Forum Archive :
From: Fredrik Dahl <firstname.lastname@example.org>
Date: Wed, 2 Oct 1996 08:33:03 +0200
To: "Chuck Bower" <email@example.com>
Subject: Re: Systematic Uncertainty of Jellyfish
Saw parts of the discussion on rgb.
The systematic error is rather dependant on the type of positions.
In most middlegame or opening positions the evaluation error,
compared to an infinite level 5 rollout is less than 0.05 in cubeless eq
on level 6 and 7. Level 5 disagrees with level 6 evaluations by 0.023
on average for complete games.
Of course, the relative error for all levels is lower than the absolute,
so normally I consider errors of 0.02 to be meaningful on level 6 and 7.
Deep and well timed backgames are underestimated systematically,
sometimes even giving an absolute error of 0.2 for level 7.
(In fact, for those positions the level does not make a very big difference
in correcting the misevaluations, because it's rooted further down the road,
when the attacker starts leaving shots.)
Even here play vs play equities are usually ok.
Any rollout (with low enough sd) is better than any evaluation,
so even a truncated level 5 rollout is better than level 7 evaluation.
Level 6 rollouts have (obviously) less systematic error than level 5 ones,
but for most 'normal' positions the diff is less than 0.02. In tricky
positions where one side has all the hard plays the diff is bigger, for
example deep backgames are often improved by 0.05 for the backgame side.
Please note that all of this is based upon experience, so anyone who has
tested a lot can have a wellfounded opinion.
Except the 0.023 average diff between level 5 and 6 evaluations;
that I have sampled.
All the best
- Advice (David Montgomery, Apr 1996)
- Cautionary tale (Kit Woolsey, Sept 1995)
- Combining rollouts (Gregg Cattanach+, Dec 2003)
- Confidence intervals (Bob Koca, Nov 2010)
- Confidence intervals (Timothy Chow, May 2010)
- Confidence intervals (Gerry Tesauro, Feb 1994)
- Cubeless vs centered-cube rollouts (Ron Karr, Dec 1997)
- Duplicate dice (David Montgomery, June 1998)
- How reliable are rollouts? (David Montgomery, Aug 1999)
- Level-5 versus level-6 rollouts (Michael J. Zehr, June 1998)
- Level-5 versus level-6 rollouts (Chuck Bower, Aug 1997)
- Positions with inaccurate rollouts (Douglas Zare, Oct 2002)
- Reporting results of rollouts (David Montgomery, June 1995)
- Rollout settings (Lokicol+, Apr 2010)
- Settlement limit (Michael J. Zehr, Apr 1998)
- Settlement limit (Kit Woolsey, Dec 1997)
- Settlement limit in races (Alexander Nitschke, Dec 1997)
- Some guidelines (Kit Woolsey, Apr 1996)
- Standard error and JSD (rambiz+, Feb 2011)
- Standard error and JSD (Stick+, Oct 2007)
- Systematic error (Chuck Bower, Oct 1996)
- Tips for doing rollouts (Douglas Zare, June 2002)
- Truncated rollouts (Gregg Cattanach, Oct 2002)
- Truncated rollouts: pros and cons (Jason Lee+, Jan 2006)
- What is a rollout? (Gregg Cattanach, Dec 1999)