A tinyBenchmark

One of the first things I was shown how to do in a Smalltalk Image was a tinyBenchmark. It is a particular way to get a measure of the performance of Squeak and Pharo System. I thought it would be interesting to compare the results of that between different Smalltalk Systems. I’ll take those that I have most heard of: Pharo 1.2, which uses the COG VM , Squeak 4.2, GNU Smalltalk 3.2.3, VA Smalltalk 8.0.3 and VisualWorks 7.7.1.

However, tinyBenchmarks is not available on every Smalltalk being checked. So we’ll standardize the benchmarking code in every image to be the same (in Integer class, methods tinyBenchmark, benchmark and benchFib). The experiment will be repeated 10 times with each machine to reduce error imposed by GC and OS via issuing
(1 to:10) do: [:e | Transcript show: 1 tinyBenchmarks; cr.]. . All the experiments were run in an Acer Aspire 4540 Notebook with Ubuntu Linux 10.10 32-bit. (AMD Athlon II x2 M300 (2GHz, 1MB L2 Cache), with 2GB RAM).

So… here are the results!
Bytecodes

Message Sends

My impression is that, if you are working on some non commercial effort, you should go for VisualWorks ASAP. If you want to sell sth made on Smalltalk, Pharo is really the best option right now.

I hope this comparison is useful for somebody!

UPDATE There was some discussion about this post on the official Pharo mailing list. I will work on redoing this experiments during the next days, but for now, here are some extra facts that could be useful for the analysis of the results previously shown:

  • The Pharo 1.2 benchmarks are run with the following Cog VM: ‘SimpleStackBasedCogit VMMaker-oscog.34 uuid: ec174a95-5f42-4fe0-88f0-4baff13dd3ce Oct 4 2010’. The method to get this information is Smalltalk getSystemAttribute: 1008 (Thanks Elliot!). In Squeak 4.2, this returns nil. The Smalltalk vmVersion returns ‘Squeak4.1 of 17 April 2010 [latest update: #9957]’.
  • Several people have pointed out the comparison between Squeak and Pharo as unfair, since the COG VM is mostly responsible for the performance difference, and it is also available for Squeak. However, the official distribution of Squeak does NOT still come with the COG VM, but the Squeak VM. Using that VM you get the results previously shown.

    I repeated the experiment using the same VM (the COG that came with Pharo 1.2) and the Squeak Image. With the same VM, the results are very similar in performance between Squeak and Pharo. However, Pharo has, in average, 0,14% more bytecodes and 3.89% more message sends/second.

  • Anuncios

Acerca de fbanados
CS Student at University of Chile.

11 Responses to A tinyBenchmark

  1. lemuus says:

    You could try squeak 4.2 on cog, just for a fair comparison with pharo 😉

  2. Pingback: Tweets that mention A tinyBenchmark « A Cardboard Box -- Topsy.com

  3. Paolo Bonzini says:

    A couple remarks on GNU Smalltalk: 1) the bytecodes benchmark is very heavily penalized by the presence of an object table; 2) Were you running gst under 32- or 64-bit? 64-bit is _much_ faster.

    (And of course, thanks for giving us reason to go hunting more performance! We already started: http://permalink.gmane.org/gmane.comp.lang.smalltalk.gnu.general/6763)

    • Paolo Bonzini says:

      ah, of course you stated it was 32-bit. 🙂

      • fbanados says:

        Yes. I didn’t choose 64-bit because i’m almost sure that not all of the mentioned Smalltalk Systems have a 64-bit version.

    • fbanados says:

      It is true that only one benchmark is not enough to compare, but is a start, and it was what i had at hand. When I have the time, i’ll re-run the experiment with other benchmarks. (Added to the ones mentioned on the GNU-Smalltalk mailing list posts you linked, Elliot Miranda added some on Cog.)

  4. Hi, which version of Cog didd you run tinyBenchmarks on, SimpleStackBasedCogit or StackToRegisterMappingCogit. there is a marked difference, e.g. on my Mc Book Pro 2.66GHz Intel Core i7:

    old:
    Smalltalk getSystemAttribute: 1008
    ‘SimpleStackBasedCogit VMMaker-eem.679 uuid: 85144ac7-9aa4-4c72-8b5e-eedc0effcf0d Dec 14 2010’
    0 tinyBenchmarks
    ‘753495217 bytecodes/sec; 63921780 sends/sec’

    new:
    Smalltalk getSystemAttribute: 1008
    ‘StackToRegisterMappingCogit VMMaker-eem.715 uuid: 707ba1f8-c822-4204-82b6-1c7e7af64117 Feb 1 2011’
    0 tinyBenchmarks
    ‘931756141 bytecodes/sec; 128318689 sends/sec’

    • fbanados says:

      Hi Elliot! I updated the post. The Cog version was ‘SimpleStackBasedCogit VMMaker-oscog.34 uuid: ec174a95-5f42-4fe0-88f0-4baff13dd3ce Oct 4 2010′ It is what comes bundled with Pharo 1.2 One-Click.

      I’m working now with the Cog-Benchmarks packag, got it already running in the 3 open source systems. I’ll also use the new Cog VM to compare. However, it’s sure that my results will be worse than yours, because of the machine I run them on.

      After that I’ll redo tinyBenchmarks. GNU-Smalltalk also made some optimizations.

  5. Travis Kay says:

    Pharo 1.2rc
    MacBook Pro i7 2.66 Ghz
    Smalltalk getSystemAttribute: 1008
    StackToRegisterMappingCogit VMMaker-oscog.47′
    0 tinyBenchmarks ‘1047034764 bytecodes/sec; 134650203 sends/sec’

  6. clementbera says:

    Hello,

    Do you have an order of idea of the smalltalk/X vm speed ? I wonder if it is faster than cog.

Responder

Introduce tus datos o haz clic en un icono para iniciar sesión:

Logo de WordPress.com

Estás comentando usando tu cuenta de WordPress.com. Cerrar sesión / Cambiar )

Imagen de Twitter

Estás comentando usando tu cuenta de Twitter. Cerrar sesión / Cambiar )

Foto de Facebook

Estás comentando usando tu cuenta de Facebook. Cerrar sesión / Cambiar )

Google+ photo

Estás comentando usando tu cuenta de Google+. Cerrar sesión / Cambiar )

Conectando a %s

A %d blogueros les gusta esto: