The errors of a rollout come from four things:
First, the bot may misplay unequally from the different sides.
Second, if the rollout is truncated, the evaluations used at the point of
truncation may be wrong.
Third, there is still luck involved in the rollout, since variance
reduction is not perfect.
Fourth, the results may be misinterpreted.
If you believe that the position is easy to play, 2-ply may even be
overkill. On the other hand, some positions are tricky to play from one
side and easy to play from the other. You need a very high level of play to
get an accurate rollout in a position of one-sided errors. 2-ply might be
sufficient, and it might not, but it should be reasonable if your own play
is comparable to 2-ply's play.
Some positions are easy to evaluate, and some are strange enough to warrant
rollouts. If you don't trust the evaluations of positions with early
primes, and are doing rollouts, truncation might be accurate if you trust
the evaluations of the likely positions after a few rolls: Prime versus
prime, crashed positions, position with several primed checkers, etc. Would
you trust the evaluations then? I'm not sure, so I would tend to avoid
using truncated rollouts then. On the other hand, in some stages of a
holding game, you may know that either a shot has been hit or the game has
turned into a race after a few moves. Then you may be able to trust
Some versions of gnu have had inaccurate error estimates. Snowie apparently
does in match play and when the cube is higher than 1. However, these bugs
aside, you should look at the error estimates and compare them with the
difference between the best action and the second best action. You should
also compare the error estimates with the size of what you consider an
acceptable mistake. For example, I really don't mind if I make a 0.050
take/pass error. If the best plays are too close to distinguish, then if
the confidence interval is wide you should do more rollouts, but if the
confidence interval is small you should accept that there is not a big
difference between the plays, and move on.
Finally, the results of the rollout may be interpreted by applying
something like Janowski's formula. This might give an accurate assessment
of cubeful equity given the cubeless bg/g/w breakdown, but it might be
systematically inaccurate when you have a position with unusually good or
bad opportunities to recube efficiently. In those situations, cubeful
rollouts might be best, assuming that you trust the cube actions during the
rollout... but there are simple positions in which Snowie rollouts reach
the wrong conclusion no matter what settings are used. Also, I am highly
sceptical of Janowski's formula when the position may be too good to
Anyway, remember that rollouts tell you how the bot plays against itself,
not how you play against your opponent, and not how perfect play does