Willus.com Home   |   Archive   |   About  

CONTENTS

I. BACKGROUND
    1. Overview
    2. The Compilers
    3. The Programs
    4. The Systems
    5. Other notes

II. RESULTS
    1. BW1D (C)
    2. BW1D (FOR)
    3. BW2D
    4. FEM2D
    5. LAME

III. SUMMARY
    (User Comments)


Willus.com's 2002 Win32 Compiler Benchmarks:
III. SUMMARY

Clearly, the Intel compiler shines, even for Athlon chips. The Intel compiler's LAME Pentium 4 results are especially impressive. The only case where an Intel-compiled executable does not have the fastest run time is the BW1D FORTRAN code on an Athlon system, where a MinGW-compiled executable just edges it out. Kudos to the Gnu C team for that result. [See 10/02 update below.]

The other commercial compiler, Microsoft, is certainly respectable, typically losing out to Intel by only 11-13% (except for that P4 LAME result) in run-time speed.

After that, MinGW probably gets the blue ribbon among the free compilers for overall performance and flexibility, plus, it's generally the most up-to-date port of GNU-C to the Win32 platform. The other offerings: Digital Mars, Borland, RSXNT, and lcc, perform admirably as well, given that they are all free distributions.

In the end, picking the right Win32 compiler will involve more than just a few benchmark results. I'd guess that for 90% of the progams I write, they run so fast that I don't care about the run time execution. So you'll want to find out what you are comfortable with, much of which is tied up in how easy the compiler is to use, what extra goodies it comes with, and how good the documentation is. Even though RSXNT didn't perform especially well in these benchmarks, I've stuck with it for several years now because the documentation is good, it has never met a C or C++ program it didn't like, and the performance is respectable. But MinGW is clearly beckoning. If you are a serious programmer, I highly recommend that you try more than one of the free distributions. All it will cost you is some download time and some play time. In that sense, they are all winners. Again, you can find links to these compilers on my Win32 C/C++ Compilers web page.

[Update 8/02:] See the first page of this article for info on a C++ compiler benchmark performed by HP World Magazine in July 2002.

[Update 10/02:] With MinGW32 2.0 (based on gcc 3.2) now released, I did some quick benchmarks comparing it to RSXNT based on gcc 2.7.x. Surprisingly, with -O2, it was slower than RSXNT on a P-III on bw1d.c. But if I use -ffast-math (uses non-standard IEEE math shortcuts), the results are dramatic, without seeming to affect the accuracy of the simulation results. The run time with MinGW -ffast-math -O3 on bw1d.c on the 1 GHz P-III system used in these benchmarks is 176.9 seconds. This is only 12% slower than the best Intel compiler time on the same system and code. Compare this to the 240 second run time of MinGW without the -ffast-math switch. Also, MinGW with the fast-math flag is faster (by about 7%) than RSXNT with the -ffast-math flag (189.3 s), which makes me think gcc 2.7.x took some math shortcuts even without the -ffast-math flag. I also suspect that Intel takes lots of math shortcuts, so I if I do another compiler comparison in the future, I'll definitely include the -ffast-math flag as an option. [See also the 10/03 update below.]

[Update 2/03:] Be sure and see the "Other Links" section on page 1 of this benchmark to see some links to other, more recent compiler benchmarks.

[Update 10/03:] See my MinGW/Gnu C Tips page for some in-line x87 math routines (sincos, pow, exp, and atan2) which boost the performance of the MinGW/Gnu C compilers on i386 architectures.

Also see my updates on the Overview page.

I'd like to thank all of my friends and colleagues who helped me generate the data for this article by patiently running (and sometimes re-running!) my benchmarks on their machines. I'd also like to thank my wife for putting up with a few lost nights while I worked on this review. I have a new appreciation for the articles put up by places like Tom's Hardware, Ace's, and Real World Technologies. If you'd like to respond to this article, you can post a comment below or send me an e-mail.


Comments are below. They aren't quite formatted how I'd prefer, but it's a start. Ability to add new comments was terminated in 2007 due to spam.

* NEW COMMENT **

Thu Jan 31 17:55:09 2002 the author Test post This is just a test to see if my user comment posting is working correctly.

* NEW COMMENT **

Fri Feb 1 13:38:35 2002 Mike Groff Thank you I found you site to be an extremely valuable resource. Thank you for preparing this information and sharing it with the world.

* NEW COMMENT **

Mon Feb 11 07:56:37 2002 gswork Great evaluation This is a great site. Your commentary on compilers and this benchmark can help isolate compiler characteristics for certain types of task. A good reason to keep several compilers handy.

* NEW COMMENT **

Thu Feb 21 02:10:02 2002 dhd Thanks Very helpful, thank you.

* NEW COMMENT **

Thu Feb 21 12:47:44 2002 Marco Montero Interesting site Very interesting site, is a good place to find usefull information about free compilers.

* NEW COMMENT **

Thu Feb 21 17:03:57 2002 Christof Meerwald Latest Digital Mars Version It would be interesting to see the results of the benchmark compiled with the latest Digital Mars C++ compiler. According to www.digitalmars.com, DMC++ 8.27.2 now "aligns double autos for faster floating point execution".

* NEW COMMENT **

Mon Feb 25 16:29:55 2002 Dmitriy Iassenev I got another results with my reversi program I tried two compilers Visual C++ 6.0 SP5 and Intel 4.5 one. So, Visual compiled faster code for my reversi programs and it reaches 875-1060 thousands of nodes per scond on my P-III 1000MHz, while Intel compiled 4-6% slower code. I tried a lot of opt ions but all results were close to this proportion. What am I doing wrong?

* NEW COMMENT **

Fri Mar 1 17:51:59 2002 Dave Snowdon Visual C++ vs CodeWarrior Many thanks for this very useful information. I currently use Metrowerks CodeWarrior (mostly because I have CodeWarrior for the Mac too) but would love to know how CodeWarrior compares to Visual C++.
Any chance of including CodeWarrior in the benchmarks?
many thanks
Dave

* NEW COMMENT **

Tue Mar 12 06:42:16 2002 Guti Watcom C++ What about the new Watcom C++ 11.0c Beta?
Also Codewarrior is impresive in AMD CPUs!

* NEW COMMENT **

Sat Mar 23 10:01:06 2002 Thomas Nichols OpenWatcom - going O/S? Thanks for this very useful work. I've seen no mention here of OpenWatcom - Sybase have committed to releasing Watcom C++ 11.0c under an open-source license, though they haven't decided which. Watch http://www.openwatcom.org for status.
Supports DOS16, DOS32 (incl. free Causeway extender with source), Win16, Win32 and I think OS/2 and Novell NLM. I'd love to see some metrics on this one.
Many thanks!

* NEW COMMENT **

Thu Mar 28 18:48:49 2002 Anonymous Good job. I was all over the search engines trying to find some win32 compiler benches and its nice to see one with most of the popular free and commercial ones..

Anyway, I would have loved to see Codewarrior and get an idea of how it competes with the other commercial options..

Well, the evaluation was a job well done.. Nice to see lame (which I use quite often) benched.

Hmm..Dmitriy Iassenev, how long ago was the Intel 4.5 compiler released?

* NEW COMMENT **

Sat Mar 30 13:47:43 2002 Divide thanks thanks, this comparison REALLY helped me. (although i was mainly interested in the filesizes, as i don't really care about speed in my programs)

* NEW COMMENT **

Sun Mar 31 05:58:44 2002 Guti To Dmitriy Iassenev Intel C++ 4.5 is not the latest release. As far as I know 5.01 is available, featuring great improvements in code generation.

* NEW COMMENT **

Sat Apr 27 02:34:14 2002 M. Domeika To Dmitriy Iassenev Intel C++ 6.0 has recently been released. If I had to guess, reversi is probably like SPEC95/099.go. go is very sensitive to code size. You might try some code size optimizations. -Os or even using profile guided and IPO.

Max
Not speaking for Intel

* NEW COMMENT **

Sun May 12 15:19:54 2002 Justus Intel c++ Regarding this one, you all may read http://www.open-mag.com/features/Vol_15/IntelC/intelc.htm

* NEW COMMENT **

Thu May 16 07:33:34 2002 Marcus MinGW optimizations The MinGW compiler has quite a few optimization options. At a minimum, I would like to see -O3 too (not only -O2). It generally does a good job.

* NEW COMMENT **

Thu Jun 13 09:28:54 2002 the author Marcus Sorry for the late response--I haven't checked this page in a while. I tried -O3 on most of these benchmarks with MinGW and found insignificant improvement.

* NEW COMMENT **

Fri Jun 28 03:16:06 2002 Steve What about the VC options?? Thanks for running the test – the results are intriguing. However, I find that you have run an experiment with a bias towards the Intel compiler. I am a graphics and visualization programmer, and as such deal with huge data and expensive numerica l computations. In my experience, you can almost always get the best performance from the VC++ compiler. That said, I am not usually a proponent of Microsoft products. So I’d be interested to see more VC++ compiler options in your benchmarks.

* NEW COMMENT **

Sat Jun 29 09:17:39 2002 Guti To Steve Under Windows enviroments, VC++ .NET generates the fastest code for x86 CPU.
But if you want the fastest speed under Intel CPU then Intel C++ 6 is your choice (about 20% faster than VC).
Under AMD CPU, Codewarrior 8, is the winner (about 25% faster than VC++ .NET).

* NEW COMMENT **

Sat Jun 29 09:19:40 2002 Guti GCC 3.1 vs OpenWatcom 1.0 I am most interested in comparing portable compilers.
For my choice, Watcom/OpenWatcom is better than its competitors, since one machine can target different plattforms.
About speed, Watcom seems to be in 4th or 5th position, just after GCC 3.1 and before Borland C++ 5.6.0.

* NEW COMMENT **

Thu Jul 4 00:08:36 2002 the author Steve and Guti Steve, you're right--I was probably a bit biased towards the Intel compiler, but like I said in the article, I tried the numerous Intel compiler options because it had them. VC++ 6.0 did not have as many interesting optimizations, especially for th e P4. Also note that the Intel compiler is used heavily by both AMD and Intel in the Spec benchmarks, indicating that the chip engineers favor it. Guti, do you have links to benchmarks that back your claims?

* NEW COMMENT **

Fri Jul 5 05:55:14 2002 To Author License Microsoft, Borland, and Metrowerks does not allow to publish any benchmarks about its compilers.

The Codewarrior conclusion is based in my own experience. The fact is that it is the only compiler capable of optimize specially for AMD chips.

* NEW COMMENT **

Mon Jul 29 06:48:46 2002 the author Test Test to see if comments work on the new server.

* NEW COMMENT **

Wed Sep 4 14:15:53 2002 slim_shady fem2d (fortran vs c) how come bw1d for fortran & c have such a huge runtime discreancy?
on mingw 1 ghz PIII = 26 sec (fortan) & 240sec (c). Anyone knows why? are they same source code? I know fortran is blazingly fast but all benchmark online seems to show a factor of 2 only & not 10 as in this case... very confuse...

* NEW COMMENT **

Wed Sep 4 14:27:51 2002 slim_shady borland bw2d compilation did the 2nd update patch fix borland's problem to compile bw2d?

* NEW COMMENT **

Tue Sep 24 16:37:34 2002 the author slim_shady Slim, sorry for the delay--you could have e-mailed me for a faster response. I don't check this comment page every day. I probably should not have named the BW1D C and FORTRAN codes the same. They have similar physics but are actually very differ ent codes, and they are running entirely different input problems, so do not draw conclusions by comparing their run times. Regarding the 2nd update patch for Borland (I assume you meant FEM2D and not BW2D), I have not had the time to test any patc hes or upgrades to any of the compilers, but perhaps I'll do another benchmark in 2003.

* NEW COMMENT **

Sun Sep 29 20:54:53 2002 slim_shady Thanks for response Thanks for the response. Ahh.. that explains it.

It would be great if these numerical language shootouts can be referenced to the numerical leader FORTRAN. Without these shootout, I would have never know that there are such huge performance gap between C++ compilers.

* NEW COMMENT **

Sat Oct 5 08:16:38 2002 author slim_shady Slim, I zapped your mistaken comment. See my 10/02 update above on the summary page. As for FORTRAN being a numerical leader, I think C can get equivalent numerical performance to FORTRAN when coded well, but that's a whole different debate topic! Later.

* NEW COMMENT **

Wed Nov 6 12:05:21 2002 John MS compilers I've created a really simple hello world console application and compiled and linked it with Microsofts CL.EXE and LINK.EXE. To my suprise when I compiled with the tools for Visual Studio 6.0 the file size was 28K. When I compiled it with Visual St udio .NETs tools it was 36K. I know for sure that I am not including the CLR component in .NET. I wonder why the differences?

* NEW COMMENT **

Tue Nov 12 01:11:19 2002 Guti To John Maybe this size increase is due to different executable alignments.
Or maybe version7 includes a bigger version info resource.

* NEW COMMENT **

Fri Jan 3 14:57:00 2003 Ken Roger Lccwin32 Adding the -p6 switch will improve execution times for Lccwin32.

* NEW COMMENT **

Thu Jan 30 09:22:23 2003 Guti Open Watcom 1.0 RC2 released! Open Watcom 1.0 RC2 has released!
Seems that the final 1.0 version is very near.
What about including it in your benches? I am sure it is at the level of Borland C++ compiler, or even a bit better!

* NEW COMMENT **

Sat Feb 8 14:34:15 2003 Azar Updated versions and new compilers Thanks so much for putting this together. It's useful to see where the different compilers excel. Do you have any plans to keep this site updated? I understand if it's too much work, but it would be nice to see "Willus.com's 2003 Win32 Compiler B enchmarks". Also, it would be great if could get ahold of even more compilers (both free and commercial).

New/updated compilers:
1) Open Watcom 1.0 (Officially released)
2) Codewarrior 8 (if possible)
3) Intel 7
4) MSVC 7 (Part of Platform SDK)
5) Digital Mars 8.32
6) MinGW 2.0.0 (Using GCC 3.2.1)
7) LCC 3.8
8) DJGPP (Using GCC 3.2.1)

Thanks!

* NEW COMMENT **

Mon Feb 10 01:23:16 2003 Guti That would be nice Azar I think so too.
This could be nice, specially for the hight expectatives with Codewarrior 8 and Intel 7, for me the best eficiency targeting AMD CPU the first, and Intel the second.
OpenWatcom should be in the test, since it is and old friend!
About Visual C++ 7, I will suggest to wait until 7.1 shocks the market, in 2 months or so, because it improves the standard, and C++ compatibility.

* NEW COMMENT **

Tue Feb 18 23:49:29 2003 Alex Codewarrior vs. Visual Studio I just tried Codewarrior 8 for the first time. For what it's worth, on my Athlon XP 1800, it generated an executable that executed twice as fast as the one generated by VC++ .NET, and I hadn't even instructed it to optimize for an Athlon chip!

* NEW COMMENT **

Wed Feb 19 03:04:11 2003 Guti CW and VC++ In my opinion, CW is a lot more effective than VC++.
It is only comparable targeting AMD machines with Intel C++ targeting Intel ones.

* NEW COMMENT **

Sun Feb 23 22:03:11 2003 author Azar and Guti I would love to do an update with all new compilers, but the codes I run are time consuming to get compiled for each platform (each compiler almost always has a couple issues to work out when you compile 150K lines of code), and my free time is in s hort supply these days. So don't hold your breath. Meanwhile, see the updates on this column for a link comparing Intel v7 to gcc 3.2.1.

* NEW COMMENT **

Tue Mar 4 09:52:03 2003 Stig 20x slower! I've used a regular C program (drawing a lot of random numbers from rand()) for radiation transport. Moving from my laptop (766MHz; Windows ME; Microsoft C/C++ 2.0) to the desktop (766MHz; Windows NT 4.0; Microsoft C/C++ 4.0) decreases the running t ime by a FACTOR OF 20 !! I'm running the program as a console application, but I assume I have some setting wrong on my desk-top. Any ideas as to what is wrong?

* NEW COMMENT **

Thu Mar 6 03:01:37 2003 Guti To Stig Can you post the source or a link to it?
NT is fair more efficient than ME, so your results are very strange. Anyway, impossible to judge without seeing the source.

* NEW COMMENT **

Fri Apr 18 20:03:55 2003 author Stig Sorry to be replying so late. I'd be happy to look at you code if you e-mail it to me. My suggestion would be to "profile" your code and see where it is spending its time.

* NEW COMMENT **

Thu May 8 16:48:02 2003 Fang Jin Arh Thanks for putting these together. I am not sure if all the compiler's getting better, but i just compiled one of my c++ console program using VC, BC and Intel. It was written in BC and ported to VC later. The result is 16s(BCB6),11s(VC++.net),22s(I ntelC++). I am confused by the result. I optimized them as much as i can.

* NEW COMMENT **

Fri May 9 06:02:53 2003 Guti

Arh, probably you are running your tests in a AMD machine, or has not set the compiler switches correctlly for ICL.
In the case of AMD architectures, try Codewarrior.
ICL 7.1 is a bit faster than VC 7.1.
The code generator for BCB is a bit bad.

* NEW COMMENT **

Tue Sep 2 10:24:08 2003 JimP Template Support and STL How well do these compilers support the ANSI/ISO C++ standard, in part. Templates and the STL?

* NEW COMMENT **

Thu Sep 18 04:53:46 2003 Wee Jin Goh More appropriate compiler flags For GCC, instead of just using -mcpu=i686, in my experience, it has always been better to put supply the proper processor name.
So you should have -mcpu=pentium3, -mcpu=pentium4 and -mcpu=athlon-xp. Of course, you'll end up with 3 binaries, but they should be more optimized (due to instruction scheduling).
You might even want to try enabling sse with the -msse & -msse2 switches. It would be good to see what that does to performance.

* NEW COMMENT **

Sat Sep 27 05:47:41 2003 author Wee Jin Goh In my experience with gcc on PC's (RSXNT and MinGW), -O2 and -ffast-math are the two most important compile flags. Everything beyond that (-O3, -funroll-all-loops, -mcpu=) seems to add only very small improvements (0-3%). That's just my experience. I notice that AMD used gcc 3.3 for their latest SPEC benchmark on the Athlon64 FX-51, and their base flags were -O3 and -funroll-all-loops. You can see the flags they used here.

* NEW COMMENT **

Sat Sep 27 05:54:29 2003 author JimP Sorry for the late reply. As I've said before in this section, I do not check this page very often any more, so it's better to e-mail me questions if you want a fast response.
I'm not sure how well any of the compilers that I tested support the complete ANSI/ISO C++ standard. I didn't write the C++ code that I used in my benchmark, but I know that the code I compiled uses templates, and I would assume that the more widel y used compilers (gcc or MS) do a good job of conforming to the latest standards. To be sure, you can check out the home page for each compiler you are interested in. My compiler links are here.

* NEW COMMENT **

Mon Dec 15 06:22:24 2003 Guti Intel C++ 8 has been released Hope this will be included in the next update of Willus benches.

* NEW COMMENT **

Tue Dec 16 21:56:46 2003 author Guti I have updated my compilers page about the Intel release. I still have no immediate plans to do more benchmarks, but you can see Intel v8 vs. v7 on the spec benchmark (look at Intel's AA- 206 P4EE results). Send me an e-mail sometime, Guti.

* NEW COMMENT **

Fri Mar 5 07:33:56 2004 Guti Did you received my email? (nt)

* NEW COMMENT **

Mon Mar 8 10:55:10 2004 author yes Yes, and I wrote back. I will try writing back one more time.

* NEW COMMENT **

Mon Apr 12 15:06:35 2004 Osman M. for I don't know if anyone else here has noticed this , but the for loop in MS VC++ (Enterprise edition) is interesting: the middle slot (continuations) allows for both statements and conditionals i.e. allows for: (cout << variable && variable < y) . H owever, it doesn't seem to execute incrementation (x++) commands in the same section, for some odd reason (the programmers didn't want people making fun of their for loops?). Who here knows if there's a study done on compilers & their for loops?

* NEW COMMENT **

Mon Apr 19 10:12:48 2004 author Osman M. I'm not sure what you're getting at, but this really isn't the right place for your post. Perhaps you should try Google Groups under comp.compilers, or some other similar forum.

* NEW COMMENT **

Tue May 11 13:59:50 2004 Mr Undisclosed Which is best to start off with? Hi,
I am looking to start compiling and i dont know what compiler to use. I am running Win XP Pro, AMD Athlon Xp 2800+, 256 MB Ram and plenty of space. If anyone can suggest a Compiler and or visual dewlopment program as well for me to use i would be gr ateful! Thanks

* NEW COMMENT **

Sat Jun 26 06:59:03 2004 Guti

If you want a good compiler, and easy to use, get Microsoft Visual C++ 2003.

If you need visual development, then your best choice is the discotinued Borland C++ Builder 6.

* NEW COMMENT **

Tue Nov 9 14:51:58 2004 Ken E Looking for advice I'm looking for some help selecting a "free" C compiler for an embedded 186 processor and to work with an SigNum 186 emulator. I currently have a copy of the Paradigm x86 compiler but I'm not happy with it's optimizer for the '186. The embedded de sign is for a micro controller w/o a lot of computation work - mostly floging I/O bits and table lookup. Any suggestions will be appreciated.

* NEW COMMENT **

Thu Dec 23 14:49:11 2004 Guti Ken E, Take a look at Digital Mars (nt)

* NEW COMMENT **

Tue Jul 12 12:45:59 2005 Pranav Swaroop Great Job Your site is a great place for anyone looking to start some serious C programming. Keep it coming... cheers!

* NEW COMMENT **

Sun Jun 11 17:18:28 2006 Geek-kit Nice site! I think the best thing about all of your work is that you've approached it with an open mind and no preconceptions nor bias towards anyone particular compiler. This is hard to do!

Keep up the good work!


<<  Previous: LAME

 
This page last modified
Thursday, 29-Dec-2011 09:25:00 MST