A tinyBenchmark
10 febrero 2011 10 comentarios
One of the first things I was shown how to do in a Smalltalk Image was a tinyBenchmark. It is a particular way to get a measure of the performance of Squeak and Pharo System. I thought it would be interesting to compare the results of that between different Smalltalk Systems. I’ll take those that I have most heard of: Pharo 1.2, which uses the COG VM , Squeak 4.2, GNU Smalltalk 3.2.3, VA Smalltalk 8.0.3 and VisualWorks 7.7.1.
However, tinyBenchmarks is not available on every Smalltalk being checked. So we’ll standardize the benchmarking code in every image to be the same (in Integer class, methods tinyBenchmark, benchmark and benchFib). The experiment will be repeated 10 times with each machine to reduce error imposed by GC and OS via issuing
(1 to:10) do: [:e | Transcript show: 1 tinyBenchmarks; cr.]. . All the experiments were run in an Acer Aspire 4540 Notebook with Ubuntu Linux 10.10 32-bit. (AMD Athlon II x2 M300 (2GHz, 1MB L2 Cache), with 2GB RAM).
So… here are the results!
Bytecodes

Message Sends

My impression is that, if you are working on some non commercial effort, you should go for VisualWorks ASAP. If you want to sell sth made on Smalltalk, Pharo is really the best option right now.
I hope this comparison is useful for somebody!
UPDATE There was some discussion about this post on the official Pharo mailing list. I will work on redoing this experiments during the next days, but for now, here are some extra facts that could be useful for the analysis of the results previously shown:
- The Pharo 1.2 benchmarks are run with the following Cog VM: ‘SimpleStackBasedCogit VMMaker-oscog.34 uuid: ec174a95-5f42-4fe0-88f0-4baff13dd3ce Oct 4 2010′. The method to get this information is
Smalltalk getSystemAttribute: 1008(Thanks Elliot!). In Squeak 4.2, this returns nil. TheSmalltalk vmVersionreturns ‘Squeak4.1 of 17 April 2010 [latest update: #9957]‘. - Several people have pointed out the comparison between Squeak and Pharo as unfair, since the COG VM is mostly responsible for the performance difference, and it is also available for Squeak. However, the official distribution of Squeak does NOT still come with the COG VM, but the Squeak VM. Using that VM you get the results previously shown.
I repeated the experiment using the same VM (the COG that came with Pharo 1.2) and the Squeak Image. With the same VM, the results are very similar in performance between Squeak and Pharo. However, Pharo has, in average, 0,14% more bytecodes and 3.89% more message sends/second.





