GNU Backgammon

Forum Archive : GNU Backgammon

Interpreting JSD's

From:   Adrian Wright
Date:   19 February 2005
Subject:   Any statisticians out there?
Forum:   GammOnLine

If you do a GNU rollout which terminates once a certain JSD is reached
then what is the likely 'actual' JSD when the rollout stops.

I say this as the JSD seems to be a bit of a 'random walk' thats
ultimately headed to zero (if the outcomes are identical) or infinty if
not. But if I stop the first time I get to say 3 jsds, am I not really
getting the full 3 jsds worth of 'confidence'.

I guess its going to depend somehow on the number of trials / volatility
of the position as well somehow?

Maik Stiebler  writes:

A typical example of a rollout setting is: stop when plays are x JSDs
apart, but do at least 300 trials and not more than 10000.

I simulated this by drawing rollout results from a uniform distribution
(which is probably not a very good model). Instead of JSDs I used
(current deviation of the sample mean from the known population mean)
/(standard error as estimated from the rollout).

With x=2.5 and settings as above, the rollout hit the stopping criteria
before the 10000th trial (i.e., it gave a 'wrong' result) in 15% of the
cases, with x=2.7 it happened 10% of the time.

Then I changed the settings to at least 500 trials, and for x=2.5 the
number of wrong results decreased to 10%. Then I allowed the rollouts to
be extended to 30000 trials and, of course, it got worse again: The
probability of a wrong result was more than 40% for x=2.0, but less than
6% for x=3.0.

I'd conclude from that that a threshold of 3 JSDs is usually very good
in practice and that 2 JSDs may be a bit too low really to get good
statistical significance.

I cannot see how that would depend on the volatility of the position. It
does depend on the number of trials in the sense that the lower the
minimum number of trials and the larger the maximum number, the worse
the confidence gets. If you decide before the rollout that you will do
exactly N trials, then x=2.0 will give you fine confidence as long as N
is sufficiently large (say larger than 100).

Adrian Wright  writes:

Thanks, that's helpful. What I meant by the volatility of the position
was that some situations, such as a last roll position or a relatively
low contact position will converge very rapidly, whereas others my be
inherently much more complex with many possible bi-furcations, say a
prime vs prime or back game. I would have thought the more complex
positions would require more trials to get an equivalent confidence.

If I've understood you correctly then if I'm doing 2ply-2ply rollouts
and stopping when over 3 jsd with minimum 648 trials and maximum 3888
that should produce a high degree of confidence.

Maik Stiebler  writes:

The simpler the position, the better variance reduction will work, i.e.
you get lower standard errors with fewer trials. That is very helpful
and will tend to reduce the number of trials needed to reach the 3 JSDs
difference threshold. It does not affect the relationship between # of
JSDs and confidence levels, however.

The settings you give as an example would indeed give a very high
confidence level in my model.

Robert-Jan Veldhuizen  writes:

I think there's not much need for setting the JSD to stop a move at, to
values like 3 or higher even, when you'd settle for >1.65 JSD otherwise.
As long as you keep the minimum trials value high enough, like 216 or
higher when the moves are close or SE's are high.

Part of the reason is that GNUBG keeps measuring the JSD diff even after
a certain rollout candidate has stopped. Should the JSD drop below your
set value later, GNUBG will re-start the interrupted rollout of the
trial(s) and get them up to the current number of trials, then proceed
as usual.

If you're planning on doing 1296 or maybe 2592 trials maximum, my idea
is that setting it at minimum 324 trials with 2.2 JSD is enough, if
you'd settle for 1.65 JSD after the maximum number of trials.

Very hard to really test or model this though...
Did you find the information in this article useful?          

Do you have any comments you'd like to add?     


GNU Backgammon

Analyzing GamesGrid matches  (Roy Passfield, Dec 2001) 
Batch analysis tool  (Øystein Johansen, June 2004)  [GammOnLine forum]
Cache size  (Ned Cross+, Mar 2004)  [GammOnLine forum]
Compiling for Windows  (Øystein Johansen, Jan 2002) 
Edit mode removing checker from bar  (Scott Steiner+, May 2003) 
Entering an annotated match  (Albert Silver, Dec 2003)  [GammOnLine forum]
Error rates: Gnu vs. Snowie  (Raccoon, Mar 2006)  [GammOnLine forum]
Even-ply/odd-ply effect  (Raccoon, Nov 2004) 
Even-ply/odd-ply effect  (Tom Keith+, Oct 2003) 
Even-ply/odd-ply effect  (Scott Steiner+, Dec 2002) 
Filter settings  (Robert-Jan Veldhuizen, Nov 2004)  [GammOnLine forum]
Gnu 0.13 versus Jellyfish and Snowie  (Torsten Schoop, Aug 2003) 
Gnu 0.13 vs. Snowie 4  (Albert Silver, June 2003) 
Gnu 0.14 vs. Jellyfish  (Michael Howard+, July 2003) 
Gnu versus Snowie and Jellyfish  (Michael Depreli, Oct 2005) 
How luck factor is calculated  (Gregg Cattanach, Aug 2002) 
How rollouts work  (Gary Wong, July 1999) 
How to enter an illegal move  (Øystein Johansen, Aug 2003)  [GammOnLine forum]
Importing .gam files  (PAR+, Mar 2005) 
Importing PartyGammon matches  (rew+, July 2006) 
Improving your game using GnuBG  (D.U.G.+, Nov 2002) 
Installing on Windows  (maareyes, Oct 2001) 
Interpreting JSD's  (Adrian Wright+, Feb 2005)  [GammOnLine forum]
JSD's and confidence intervals  (Daniel Murphy+, Jan 2005) 
Logging rollouts  (Øystein Johansen, Oct 2004)  [GammOnLine forum]
Luck rate  (Kees van den Doel+, May 2002) 
MWC versus Equity (EMG)  (Ken+, Apr 2005)  [GammOnLine forum]
Manually entering first roll  (Andreas Graf+, Apr 2005) 
Match equity tables  (Raccoon, July 2005)  [GammOnLine forum]
Personal reflections  (Louis Nardy Pillards, Sept 2002) 
Playing two computers against each other  (Stanley E. Richards+, Mar 2008)  [GammOnLine forum]
Python scripting  (Øystein Johansen+, Nov 2004) 
Quasi-random dice in rollouts  (Ian Shaw, Mar 2004)  [GammOnLine forum]
Question marks in game list  (Jim Segrave, July 2005) 
Questions and answers  (Jim Segrave+, Jan 2003) 
Questions and answers  (Jørn Thyssen, Aug 2002) 
Restarting a rollout with different settings  (Jim Segrave, Apr 2005)  [GammOnLine forum]
Restarting a rollout with different settings  (Robert-Jan Veldhuizen, Apr 2004)  [GammOnLine forum]
Rollout settings  (geoff arnold+, Apr 2007) 
Rollout settings  (Stick+, Nov 2005)  [GammOnLine forum]
Rollout settings  (Robert-Jan Veldhuizen, Mar 2004)  [GammOnLine forum]
Rollout settings  (Ian Dunstan, Aug 2003)  [GammOnLine forum]
Rollout settings for the impatient  (Robert-Jan Veldhuizen, June 2004)  [GammOnLine forum]
Running rollouts in background  (Bruce+, Apr 2004)  [GammOnLine forum]
Saving rollout results from command-line interface  (Jeremy Bagai+, Apr 2006)  [GammOnLine forum]
Saving rollouts  (Mislav Radica+, May 2006)  [GammOnLine forum]
Setting GnuBG's playing strength  (JP White, Sept 2001) 
Setting skill level  (Jim Segrave, Apr 2004) 
Setting up and saving a rollout  (Albert Silver, Dec 2003)  [GammOnLine forum]
What's GNU?  (Gary Wong, Oct 2001) 
Which player is player 0?  (Neil Kazaross+, Oct 2004)  [GammOnLine forum]

[GammOnLine forum]  From GammOnLine       [Long message]  Long message       [Recommended reading]  Recommended reading       [Recent addition]  Recent addition

  Book Suggestions
Computer Dice
Cube Handling
Cube Handling in Races
Extreme Gammon
Fun and frustration
GNU Backgammon
Luck versus Skill
Magazines & E-zines
Match Archives
Match Equities
Match Play
Match Play at 2-away/2-away
Opening Rolls
Pip Counting
Play Sites
Probability and Statistics
Source Code
Strategy--Bearing Off
Strategy--Checker play


Return to:  Backgammon Galore : Forum Archive Main Page