CONTENTS
I. BACKGROUND
1. Overview
2. The Compilers
3. The Programs
4. The Systems
5. Other notes
II. RESULTS
1. BW1D (C)
2. BW1D (FOR)
3. BW2D
4. FEM2D
5. LAME
III. SUMMARY
(User Comments)
|
| |
Willus.com's 2002 Win32 Compiler Benchmarks: III. SUMMARY
Clearly, the Intel compiler shines, even for Athlon chips. The
Intel compiler's LAME Pentium 4 results are especially impressive. The
only case where an Intel-compiled executable does not have the fastest
run time is the
BW1D
FORTRAN code
on an Athlon system, where a MinGW-compiled executable just edges it out.
Kudos to the Gnu C team for that result.
[See 10/02 update below.]
The other commercial compiler, Microsoft, is certainly respectable, typically
losing out to Intel by only 11-13% (except for that P4 LAME result) in
run-time speed.
After that, MinGW probably gets the blue ribbon among the free compilers for
overall performance and flexibility, plus, it's generally the most
up-to-date port of GNU-C to the Win32 platform. The other offerings:
Digital Mars, Borland, RSXNT, and lcc, perform admirably as well,
given that they are all free distributions.
In the end, picking the right Win32 compiler will involve
more than just a few benchmark results. I'd guess that for 90% of the
progams I write, they run so fast that I don't care about the run time
execution. So you'll want to find out what you are
comfortable with, much of which is tied up in how easy the compiler
is to use, what extra goodies it comes with, and how good the documentation
is. Even though RSXNT didn't perform especially well in these benchmarks,
I've stuck with it for several years now because the documentation
is good, it has never met a C or C++ program it didn't like, and
the performance is respectable. But MinGW is clearly beckoning.
If you are a serious programmer, I highly recommend that you try more
than one of the free distributions. All it will cost you is some download
time and some play time. In that sense, they are all winners.
Again, you can find links to these compilers on my
Win32 C/C++ Compilers web page.
[Update 8/02:] See the first page of this article for info on a
C++ compiler benchmark performed by HP World Magazine in July 2002.
[Update 10/02:] With
MinGW32 2.0 (based on gcc 3.2) now
released, I did some quick benchmarks comparing it to RSXNT based on
gcc 2.7.x. Surprisingly, with -O2, it was slower than RSXNT on a P-III
on bw1d.c. But if I use -ffast-math (uses non-standard IEEE math shortcuts),
the results are dramatic, without seeming to affect the accuracy of
the simulation results. The
run time with MinGW -ffast-math -O3 on bw1d.c on the 1 GHz P-III system
used in these benchmarks is 176.9 seconds. This is
only 12% slower than the best Intel compiler time on the same system
and code. Compare this to the 240 second run time of MinGW without
the -ffast-math switch.
Also, MinGW with the fast-math flag is faster (by about 7%) than RSXNT
with the -ffast-math flag (189.3 s), which makes me think gcc 2.7.x took some
math shortcuts even without the -ffast-math flag. I also suspect that
Intel takes lots of math shortcuts, so I if I do another compiler
comparison in the future, I'll definitely include the -ffast-math flag
as an option. [See also the 10/03 update below.]
[Update 2/03:] Be sure and see the "Other Links"
section on page 1 of this
benchmark to see some links to other, more recent compiler benchmarks.
[Update 10/03:] See my
MinGW/Gnu C Tips page for some in-line x87 math
routines (sincos, pow, exp, and atan2) which boost the performance
of the MinGW/Gnu C compilers on i386 architectures.
Also see my updates on the Overview
page.
I'd like to thank all of my friends and colleagues who helped me generate
the data for this article by patiently running (and sometimes re-running!)
my benchmarks on their machines. I'd also like to thank my wife for
putting up with a few lost nights while I worked on this review. I have
a new appreciation for the articles put up by places like
Tom's Hardware,
Ace's, and
Real World Technologies.
If you'd like to respond to this article, you can post a comment below
or send me an e-mail.
Comments are below. They aren't quite formatted how I'd prefer, but it's a
start. Ability to add new comments was terminated in 2007 due to spam.
* NEW COMMENT **
Thu Jan 31 17:55:09 2002
the author
Test post
This is just a test to see if my user comment posting is working correctly.
* NEW COMMENT **
Fri Feb 1 13:38:35 2002
Mike Groff
Thank you
I found you site to be an extremely valuable resource. Thank you for preparing this information and sharing it with the world.
* NEW COMMENT **
Mon Feb 11 07:56:37 2002
gswork
Great evaluation
This is a great site. Your commentary on compilers and this benchmark can help isolate compiler characteristics for certain types of task. A good reason to keep several compilers handy.
* NEW COMMENT **
Thu Feb 21 02:10:02 2002
dhd
Thanks
Very helpful, thank you.
* NEW COMMENT **
Thu Feb 21 12:47:44 2002
Marco Montero
Interesting site
Very interesting site, is a good place to find usefull information about free compilers.
* NEW COMMENT **
Thu Feb 21 17:03:57 2002
Christof Meerwald
Latest Digital Mars Version
It would be interesting to see the results of the benchmark compiled with the latest Digital Mars C++ compiler. According to www.digitalmars.com, DMC++ 8.27.2 now "aligns double autos for faster floating point execution".
* NEW COMMENT **
Mon Feb 25 16:29:55 2002
Dmitriy Iassenev
I got another results with my reversi program
I tried two compilers Visual C++ 6.0 SP5 and Intel 4.5 one. So, Visual compiled faster code for my reversi programs and it reaches 875-1060 thousands of nodes per scond on my P-III 1000MHz, while Intel compiled 4-6% slower code. I tried a lot of opt
ions but all results were close to this proportion. What am I doing wrong?
* NEW COMMENT **
Fri Mar 1 17:51:59 2002
Dave Snowdon
Visual C++ vs CodeWarrior
Many thanks for this very useful information. I currently use Metrowerks CodeWarrior (mostly because I have CodeWarrior for the Mac too) but would love to know how CodeWarrior compares to Visual C++.
Any chance of including CodeWarrior in the benchmarks?
many thanks
Dave
* NEW COMMENT **
Tue Mar 12 06:42:16 2002
Guti
Watcom C++
What about the new Watcom C++ 11.0c Beta?
Also Codewarrior is impresive in AMD CPUs!
* NEW COMMENT **
Sat Mar 23 10:01:06 2002
Thomas Nichols
OpenWatcom - going O/S?
Thanks for this very useful work. I've seen no mention here of OpenWatcom - Sybase have committed to releasing Watcom C++ 11.0c under an open-source license, though they haven't decided which. Watch http://www.openwatcom.org for status.
Supports DOS16, DOS32 (incl. free Causeway extender with source), Win16, Win32 and I think OS/2 and Novell NLM. I'd love to see some metrics on this one.
Many thanks!
* NEW COMMENT **
Thu Mar 28 18:48:49 2002
Anonymous
Good job.
I was all over the search engines trying to find some win32 compiler benches and its nice to see one with most of the popular free and commercial ones..
Anyway, I would have loved to see Codewarrior and get an idea of how it competes with the other commercial options..
Well, the evaluation was a job well done.. Nice to see lame (which I use quite often) benched.
Hmm..Dmitriy Iassenev, how long ago was the Intel 4.5 compiler released?
* NEW COMMENT **
Sat Mar 30 13:47:43 2002
Divide
thanks
thanks, this comparison REALLY helped me. (although i was mainly interested in the filesizes, as i don't really care about speed in my programs)
* NEW COMMENT **
Sun Mar 31 05:58:44 2002
Guti
To Dmitriy Iassenev
Intel C++ 4.5 is not the latest release. As far as I know 5.01 is available, featuring great improvements in code generation.
* NEW COMMENT **
Sat Apr 27 02:34:14 2002
M. Domeika
To Dmitriy Iassenev
Intel C++ 6.0 has recently been released. If I had to guess, reversi is probably like SPEC95/099.go. go is very sensitive to code size. You might try some code size optimizations. -Os or even using profile guided and IPO.
Max
Not speaking for Intel
* NEW COMMENT **
Sun May 12 15:19:54 2002
Justus
Intel c++
Regarding this one, you all may read http://www.open-mag.com/features/Vol_15/IntelC/intelc.htm
* NEW COMMENT **
Thu May 16 07:33:34 2002
Marcus
MinGW optimizations
The MinGW compiler has quite a few optimization options. At a minimum, I would like to see -O3 too (not only -O2). It generally does a good job.
* NEW COMMENT **
Thu Jun 13 09:28:54 2002
the author
Marcus
Sorry for the late response--I haven't checked this page in a while. I tried -O3 on most of these benchmarks with MinGW and found insignificant improvement.
* NEW COMMENT **
Fri Jun 28 03:16:06 2002
Steve
What about the VC options??
Thanks for running the test the results are intriguing. However, I find that you have run an experiment with a bias towards the Intel compiler. I am a graphics and visualization programmer, and as such deal with huge data and expensive numerica
l computations. In my experience, you can almost always get the best performance from the VC++ compiler. That said, I am not usually a proponent of Microsoft products. So Id be interested to see more VC++ compiler options in your benchmarks.
* NEW COMMENT **
Sat Jun 29 09:17:39 2002
Guti
To Steve
Under Windows enviroments, VC++ .NET generates the fastest code for x86 CPU.
But if you want the fastest speed under Intel CPU then Intel C++ 6 is your choice (about 20% faster than VC).
Under AMD CPU, Codewarrior 8, is the winner (about 25% faster than VC++ .NET).
* NEW COMMENT **
Sat Jun 29 09:19:40 2002
Guti
GCC 3.1 vs OpenWatcom 1.0
I am most interested in comparing portable compilers.
For my choice, Watcom/OpenWatcom is better than its competitors, since one machine can target different plattforms.
About speed, Watcom seems to be in 4th or 5th position, just after GCC 3.1 and before Borland C++ 5.6.0.
* NEW COMMENT **
Thu Jul 4 00:08:36 2002
the author
Steve and Guti
Steve, you're right--I was probably a bit biased towards the Intel compiler, but like I said in the article, I tried the numerous Intel compiler options because it had them. VC++ 6.0 did not have as many interesting optimizations, especially for th
e P4. Also note that the Intel compiler is used heavily by both AMD and Intel in the Spec benchmarks, indicating that the chip engineers favor it. Guti, do you have links to benchmarks that back your claims?
* NEW COMMENT **
Fri Jul 5 05:55:14 2002
To Author
License
Microsoft, Borland, and Metrowerks does not allow to publish any benchmarks about its compilers.
The Codewarrior conclusion is based in my own experience. The fact is that it is the only compiler capable of optimize specially for AMD chips.
* NEW COMMENT **
Mon Jul 29 06:48:46 2002
the author
Test
Test to see if comments work on the new server.
* NEW COMMENT **
Wed Sep 4 14:15:53 2002
slim_shady
fem2d (fortran vs c)
how come bw1d for fortran & c have such a huge runtime discreancy?
on mingw 1 ghz PIII = 26 sec (fortan) & 240sec (c). Anyone knows why? are they same source code? I know fortran is blazingly fast but all benchmark online seems to show a factor of 2 only & not 10 as in this case... very confuse...
* NEW COMMENT **
Wed Sep 4 14:27:51 2002
slim_shady
borland bw2d compilation
did the 2nd update patch fix borland's problem to compile bw2d?
* NEW COMMENT **
Tue Sep 24 16:37:34 2002
the author
slim_shady
Slim, sorry for the delay--you could have e-mailed me for a faster response. I don't check this comment page every day. I probably should not have named the BW1D C and FORTRAN codes the same. They have similar physics but are actually very differ
ent codes, and they are running entirely different input problems, so do not draw conclusions by comparing their run times. Regarding the 2nd update patch for Borland (I assume you meant FEM2D and not BW2D), I have not had the time to test any patc
hes or upgrades to any of the compilers, but perhaps I'll do another benchmark in 2003.
* NEW COMMENT **
Sun Sep 29 20:54:53 2002
slim_shady
Thanks for response
Thanks for the response. Ahh.. that explains it.
It would be great if these numerical language shootouts can be referenced to the numerical leader FORTRAN. Without these shootout, I would have never know that there are such huge performance gap between C++ compilers.
* NEW COMMENT **
Sat Oct 5 08:16:38 2002
author
slim_shady
Slim, I zapped your mistaken comment. See my 10/02 update above on the summary page. As for FORTRAN being a numerical leader, I think C can get equivalent numerical performance to FORTRAN when coded well, but that's a whole different debate topic!
Later.
* NEW COMMENT **
Wed Nov 6 12:05:21 2002
John
MS compilers
I've created a really simple hello world console application and compiled and linked it with Microsofts CL.EXE and LINK.EXE. To my suprise when I compiled with the tools for Visual Studio 6.0 the file size was 28K. When I compiled it with Visual St
udio .NETs tools it was 36K. I know for sure that I am not including the CLR component in .NET. I wonder why the differences?
* NEW COMMENT **
Tue Nov 12 01:11:19 2002
Guti
To John
Maybe this size increase is due to different executable alignments.
Or maybe version7 includes a bigger version info resource.
* NEW COMMENT **
Fri Jan 3 14:57:00 2003
Ken Roger
Lccwin32
Adding the -p6 switch will improve execution times for Lccwin32.
* NEW COMMENT **
Thu Jan 30 09:22:23 2003
Guti
Open Watcom 1.0 RC2 released!
Open Watcom 1.0 RC2 has released!
Seems that the final 1.0 version is very near.
What about including it in your benches? I am sure it is at the level of Borland C++ compiler, or even a bit better!
* NEW COMMENT **
Sat Feb 8 14:34:15 2003
Azar
Updated versions and new compilers
Thanks so much for putting this together. It's useful to see where the different compilers excel. Do you have any plans to keep this site updated? I understand if it's too much work, but it would be nice to see "Willus.com's 2003 Win32 Compiler B
enchmarks". Also, it would be great if could get ahold of even more compilers (both free and commercial).
New/updated compilers:
1) Open Watcom 1.0 (Officially released)
2) Codewarrior 8 (if possible)
3) Intel 7
4) MSVC 7 (Part of Platform SDK)
5) Digital Mars 8.32
6) MinGW 2.0.0 (Using GCC 3.2.1)
7) LCC 3.8
8) DJGPP (Using GCC 3.2.1)
Thanks!
* NEW COMMENT **
Mon Feb 10 01:23:16 2003
Guti
That would be nice Azar
I think so too.
This could be nice, specially for the hight expectatives with Codewarrior 8 and Intel 7, for me the best eficiency targeting AMD CPU the first, and Intel the second.
OpenWatcom should be in the test, since it is and old friend!
About Visual C++ 7, I will suggest to wait until 7.1 shocks the market, in 2 months or so, because it improves the standard, and C++ compatibility.
* NEW COMMENT **
Tue Feb 18 23:49:29 2003
Alex
Codewarrior vs. Visual Studio
I just tried Codewarrior 8 for the first time. For what it's worth, on my Athlon XP 1800, it generated an executable that executed twice as fast as the one generated by VC++ .NET, and I hadn't even instructed it to optimize for an Athlon chip!
* NEW COMMENT **
Wed Feb 19 03:04:11 2003
Guti
CW and VC++
In my opinion, CW is a lot more effective than VC++.
It is only comparable targeting AMD machines with Intel C++ targeting Intel ones.
* NEW COMMENT **
Sun Feb 23 22:03:11 2003
author
Azar and Guti
I would love to do an update with all new compilers, but the codes I run are time consuming to get compiled for each platform (each compiler almost always has a couple issues to work out when you compile 150K lines of code), and my free time is in s
hort supply these days. So don't hold your breath. Meanwhile, see the updates on this column for a link comparing Intel v7 to gcc 3.2.1.
* NEW COMMENT **
Tue Mar 4 09:52:03 2003
Stig
20x slower!
I've used a regular C program (drawing a lot of random numbers from rand()) for radiation transport. Moving from my laptop (766MHz; Windows ME; Microsoft C/C++ 2.0) to the desktop (766MHz; Windows NT 4.0; Microsoft C/C++ 4.0) decreases the running t
ime by a FACTOR OF 20 !! I'm running the program as a console application, but I assume I have some setting wrong on my desk-top. Any ideas as to what is wrong?
* NEW COMMENT **
Thu Mar 6 03:01:37 2003
Guti
To Stig
Can you post the source or a link to it?
NT is fair more efficient than ME, so your results are very strange. Anyway, impossible to judge without seeing the source.
* NEW COMMENT **
Fri Apr 18 20:03:55 2003
author
Stig
Sorry to be replying so late. I'd be happy to look at you code if you e-mail it to me. My suggestion would be to "profile" your code and see where it is spending its time.
* NEW COMMENT **
Thu May 8 16:48:02 2003
Fang Jin
Arh
Thanks for putting these together. I am not sure if all the compiler's getting better, but i just compiled one of my c++ console program using VC, BC and Intel. It was written in BC and ported to VC later. The result is 16s(BCB6),11s(VC++.net),22s(I
ntelC++). I am confused by the result. I optimized them as much as i can.
* NEW COMMENT **
Fri May 9 06:02:53 2003
Guti
Arh, probably you are running your tests in a AMD machine, or has not set the compiler switches correctlly for ICL.
In the case of AMD architectures, try Codewarrior.
ICL 7.1 is a bit faster than VC 7.1.
The code generator for BCB is a bit bad.
* NEW COMMENT **
Tue Sep 2 10:24:08 2003
JimP
Template Support and STL
How well do these compilers support the ANSI/ISO C++ standard, in part. Templates and the STL?
* NEW COMMENT **
Thu Sep 18 04:53:46 2003
Wee Jin Goh
More appropriate compiler flags
For GCC, instead of just using -mcpu=i686, in my experience, it has always been better to put supply the proper processor name.
So you should have -mcpu=pentium3, -mcpu=pentium4 and -mcpu=athlon-xp. Of course, you'll end up with 3 binaries, but they should be more optimized (due to instruction scheduling).
You might even want to try enabling sse with the -msse & -msse2 switches. It would be good to see what that does to performance.
* NEW COMMENT **
Sat Sep 27 05:47:41 2003
author
Wee Jin Goh
In my experience with gcc on PC's (RSXNT and MinGW), -O2 and -ffast-math are the two most important compile flags. Everything beyond that (-O3, -funroll-all-loops, -mcpu=) seems to add only very small improvements (0-3%). That's just
my experience. I notice that AMD used gcc 3.3 for their latest SPEC benchmark on the Athlon64 FX-51, and their base flags were -O3 and -funroll-all-loops. You can see the flags they used here.
* NEW COMMENT **
Sat Sep 27 05:54:29 2003
author
JimP
Sorry for the late reply. As I've said before in this section, I do not check this page very often any more, so it's better to e-mail me questions if you want a fast response.
I'm not sure how well any of the compilers that I tested support the complete ANSI/ISO C++ standard. I didn't write the C++ code that I used in my benchmark, but I know that the code I compiled uses templates, and I would assume that the more widel
y used compilers (gcc or MS) do a good job of conforming to the latest standards. To be sure, you can check out the home page for each compiler you are interested in. My compiler links are here.
* NEW COMMENT **
Mon Dec 15 06:22:24 2003
Guti
Intel C++ 8 has been released
Hope this will be included in the next update of Willus benches.
* NEW COMMENT **
Tue Dec 16 21:56:46 2003
author
Guti
I have updated my compilers page about the Intel release. I still have no immediate plans to do more benchmarks, but you can see Intel v8 vs. v7 on the spec benchmark (look at Intel's AA-
206 P4EE results). Send me an e-mail sometime, Guti.
* NEW COMMENT **
Fri Mar 5 07:33:56 2004
Guti
Did you received my email? (nt)
* NEW COMMENT **
Mon Mar 8 10:55:10 2004
author
yes
Yes, and I wrote back. I will try writing back one more time.
* NEW COMMENT **
Mon Apr 12 15:06:35 2004
Osman M.
for
I don't know if anyone else here has noticed this , but the for loop in MS VC++ (Enterprise edition) is interesting: the middle slot (continuations) allows for both statements and conditionals i.e. allows for: (cout << variable && variable < y) . H
owever, it doesn't seem to execute incrementation (x++) commands in the same section, for some odd reason (the programmers didn't want people making fun of their for loops?). Who here knows if there's a study done on compilers & their for loops?
* NEW COMMENT **
Mon Apr 19 10:12:48 2004
author
Osman M.
I'm not sure what you're getting at, but this really isn't the right place for your post. Perhaps you should try Google Groups under comp.compilers, or some other similar forum.
* NEW COMMENT **
Tue May 11 13:59:50 2004
Mr Undisclosed
Which is best to start off with?
Hi,
I am looking to start compiling and i dont know what compiler to use. I am running Win XP Pro, AMD Athlon Xp 2800+, 256 MB Ram and plenty of space. If anyone can suggest a Compiler and or visual dewlopment program as well for me to use i would be gr
ateful! Thanks
* NEW COMMENT **
Sat Jun 26 06:59:03 2004
Guti
If you want a good compiler, and easy to use, get Microsoft Visual C++ 2003.
If you need visual development, then your best choice is the discotinued Borland C++ Builder 6.
* NEW COMMENT **
Tue Nov 9 14:51:58 2004
Ken E
Looking for advice
I'm looking for some help selecting a "free" C compiler for an embedded 186 processor and to work with an SigNum 186 emulator. I currently have a copy of the Paradigm x86 compiler but I'm not happy with it's optimizer for the '186. The embedded de
sign is for a micro controller w/o a lot of computation work - mostly floging I/O bits and table lookup. Any suggestions will be appreciated.
* NEW COMMENT **
Thu Dec 23 14:49:11 2004
Guti
Ken E, Take a look at Digital Mars (nt)
* NEW COMMENT **
Tue Jul 12 12:45:59 2005
Pranav Swaroop
Great Job
Your site is a great place for anyone looking to start some serious C programming. Keep it coming... cheers!
* NEW COMMENT **
Sun Jun 11 17:18:28 2006
Geek-kit
Nice site!
I think the best thing about all of your work is that you've approached it with an open mind and no preconceptions nor bias towards anyone particular compiler. This is hard to do!
Keep up the good work!
| |