This article originally appeared in the February 2001 issue of GammOnLine.|
Thank you to Kit Woolsey for his kind permission to reproduce it here.
On January 6th I proposed a GammonLine group project via the forum. There
are three parts to this project:
1) Benchmarking CPU's for Snowie rollouts,
This article is a report on the current results of part 1.
A standard position and set of rollout parameters were chosen. The position is the opening setup with a diceroll of 62. The rollouts consisted of 36 trials of each of three candidate plays: 24/18, 13/11; 24/16; and 13/5. The rollout parameters were: cubeless money play, 3-ply truncated at 10 rolls, truncation if bearoff database reached, medium search space, 20% speed, seed = 12345. Several GoL members took on the task with their respective CPU's and they are credited below for their contributions.
Although cputime for a given rollout should roughly correlate with CPU clock speed, different processors handle programs differently, particularly when both integer and floating point arithmetic are used. The table below is a summary of the various results reported both publically on the forum and to me privately. Also included is a plot of Snowie's runspeed (1000/runtime where 1000 is chosen to make the resulting numbers manageable and 'runtime' is given in seconds). The Snowie runspeed is plotted as a function of CPU clockspeed.
Contributor SW ver. Processor Proc. Mem. OS Tot Time Normalized Projected Ratio Speed (sec) Speed Speed Norm/Proj JP White 3.0 P-166 166 64MB Win98 7555 0.13 0.15 0.87 Chuck Bower 3.0 PII-233 233 32MB Win98 4042 0.25 0.25 0.99 Jason Lee 3.0 PII-233 233 64MB Win98 3299 0.30 0.25 1.21 JP White 3.0 AMD K6-300 300 128MB Win95 3917 0.26 0.35 0.73 Gregg Cattanach 3.2 PII-350 350 64MB Win98 2174 0.46 0.42 1.09 David Montgomery 3.2 PII-400 400 Win98 2030 0.49 0.49 1.00 David Montgomery 2.3 PII-400 400 Win98 1948 0.51 0.49 1.04 Robi 3.2 PII-400 400 128MB Win98 2104 0.48 0.49 0.96 Chuck Bower 3.2 PIII-450 450 128MB WinNT4.0 1667 0.60 0.57 1.06 Gregg Cattanach 3.2 PIII-450 450 128MB Win98 1752 0.57 0.57 1.01 JP White 3.2 AMD K6-2 450 450 128MB Win98 2161 0.46 0.57 0.82 JP White 3.2 AMD K6-2 450 450 128MB Win98 2253 0.44 0.57 0.78 Rob Maier 3.2 Celeron-466 466 1810 0.55 0.59 0.94 Chris Yep 3.2 PIII-500 500 96MB Win98 1571 0.64 0.64 1.00 TarHeelFan 2.3 PIII-600 600 256MB Win2000 1236 0.81 0.79 1.03 Bob Stringer 3.2 PIII-1000 1000 256MB Win98 782 1.28 1.37 0.93 Petter Bengsston 3.2 AMD-1000 1000 384MB Win2000 636 1.57 1.37 1.15
You will notice that the data conform roughly to a linear relationship, which should not be surprising. I performed an 'eyeball' linear fit to the data. The second to last column of the table is a predicted speed based upon this fit. The last column is the ratio of the normalized speed (1000/actual runtime) divided by the predicted speed. Although one must be careful about drawing too much from this comparison, a few generalities seem to be evident.
The early AMD processors (K6 models) underperform their Intel cousins by about 20-25%. However the latest generations of processors reverses this trend. In particular the AMD Thunderbird (Athlon?) running at 1 GHz beats the prediction by 15%, while the Intel processor running the same clock speed might even be a few percent below the projection. Both of these occurences were predicted by forum posts prior to the results becoming available and I refer the interested reader to the thread which I initiated on January 6th for more details. Finally, admittedly based upon limited data, the Intel Celeron processor may be a bit slower than it's PII and PIII siblings, though if the effect is real it is only a few percent. I believe all of this is consistent with known performance differences of these processors, particluarly when performing floating point operations.
Again cautioning that the data is limited, there may be a few percent affect due to the version of Snowie being run (2.x or 3.y) with the older version running a bit faster. There is not an obvious connection relating to either operating system nor amount of available memory.
Concerning part II (runtime dependence on Snowie parameter settings), I have compliled quite a bit of data and will report the results next month. There is still considerable help needed for part III (rollout results for different robots and for different Snowie settings). Again I'm asking for help from GoL members. Here the position and candidate plays are the same as in the other parts, but the settings (including number of trials) can and should be different. In particular, since there is the potential of combining results, the use of a random seed would be safest, but in particular DON'T use 12345. If you have a favorite seed, that is probably OK.
Part III (when combined with part II) may help answer a longstanding question: what are the most efficient Snowie settings for getting fast, reliable rollout results? In addition, by comparing rollout results from different robots, we may be able to speculate about the relative strengths of different creations. I plan on setting up the old Expert Backgammon (version I). Gary Wong has submitted data for gnu-backgammon (but more would be welcome). Jellyfish is a must (any volunteers?) and I'm going to ask at least a couple authors of homegrown robots for their input. Again, any contributions you can make would be valuable and appreciated.