[1][LWN Logo]

   [2]Linux Weekly News
   [3]Daily news
   [4]Contact us

   A Japanese translation of this page is being prepared at [5]ChangeLog.net.

   Rik van Riel has made [6]a Dutch version of this page.

   Here is [7]a French translation of this page.

   See also: [8]On Mindcraft's April 1999 Benchmark by Dan Kegel.

A look at the Mindcraft report

   On Tuesday, April 13, Mindcraft released [9]a report claiming to be a comparison
   of Windows NT and Linux in an enterprise server environment. The summary of this
   report reads:

     Microsoft Windows NT Server 4.0 is 2.5 times faster than Linux as a File
     Server and 3.7 times faster as a Web Server.

   Needless to say, this report has drawn some attention. Since their results differ
   so strongly from those of other, similar studies, it is normal to want to
   understand what was different this time around.

   This document is a summary of the information that has been gleaned from their
   report. Thanks are due to the unbelievable number of people who have sent us mail
   on the topic. We have long since lost our ability to credit everybody
   individually; to list some names would do a disservice to those that got dropped.
   So, suffice to say that the following represents the work of a great many people;
   we have mostly just served as organizers of the information.

Some history

   Mindcraft is a company which specializes in testing and benchmarking systems.
   Their testing is done for paying clients (as opposed to, say, Ziff-Davis, which
   tests for their publication). Their [10]services page sums things up very well:

     With our custom performance testing service, we work with you to define test
     goals. Then we put together the necessary tools and do the testing. We report
     the results back to you in a form that satisfies the test goals.

   We may never know what the stated "goals" of this test were, but the client was
   Microsoft. Microsoft also paid for some similar studies:
     * [11]A comparison between NT and Solaris. "Windows NT Server 4.0 Is Four Times
       Faster as a Web Server Than Solaris 2.6 with Sun Web Server 1.0 and Has 10.3
       Times Better Price/Performance".
     * [12]A comparison between NT and Novell Netware. "Microsoft Windows NT Server
       4.0 Is 25.5% Faster Than Novell NetWare 5 as a File Server and Has 2.7 Times
       Better Price/Performance".

   These other tests have brought out charges of unfair practices. See, for example,
   [13]Novell's response to the Netware test (Mindcraft responded [14]thusly).

   It is also interesting to note that, while the other tests include
   price/performance comparisons, the Linux test omitted them.

Configuration of the Linux server

   A number of problems have been found in the way the Linux server was configured
   in this test. These include:
     * The 2.2 kernel supports a number of tunable parameters in the file system and
       the buffer cache. Adjustments to the "bdflush" and file system cache size
       parameters have been known to double performance of high-stress Samba
       systems. This tuning was not performed.
     * Mindcraft used the 2.2.2 kernel for their tests, even though 2.2.3 was
       available at that time. 2.2.2 had some well known and well documented TCP
       problems, particularly relating to interoperability with Windows clients.
     * The server they tested was set up with both NT and Linux on the same disks.
       Bill Henning (of [15]CPU Review fame) points out that whichever system ended
       up on the outer part of the disk (where there are more sectors per cylinder)
       will have a 1.5 to 2-times transfer rate advantage over the other. Mindcraft
       does not specify which system was installed where, so it is not possible to
       really know what the effect of placement really was. (We have received a
       report that the test was actually performed with two separate OS disks, that
       it was not truly a multiboot system. There is still the question of the data
       disks, however).
     * The test was performed using a RAID controller which is not well supported
       with Linux, using version 0.92 of the driver, which had a known problem on
       SMP systems. A different controller would have likely yielded better results.
     * Several people have pointed out that, in the list of "processes running
       immediately before" the tests, both Apache and Samba are absent. This is
       either an oversight on their part, or these servers were running out of
       inetd. If the latter case holds, it is amazing they got the performance they
       did. It actually seems unlikely that they were this badly off; an error in
       the report itself seems more probable.
     * They went to considerable effort to optimize the performance of the network
       cards under NT. The TCP window was increased to 65536, something which can
       speed transfers in some situations on a protected net, but is not normally
       done in real situations. The same optimizations were not performed on Linux.
     * They set up a 1012 MB swap file for NT, but do not mention anything about any
       swapping arrangements for Linux.

Configuration of Apache

   The Apache setup used by Mindcraft does not well match what a real-world web
   server would use.
     * The Apache configuration is also not suited to large loads. It initially
       starts 10 servers, and MinSpareServers is set to 1. In particular, quick
       response to sudden, heavy loads will be reduced by this configuration. It is
       not an "enterprise" setup.
     * Some questions have been raised about how the Apache logging was set up. It
       appears that Apache was logging to the same drive the OS was on, which could
       have hurt performance (NT/IIS was logging to the RAID array). IIS was also
       apparently configured such that no actual logging would happen at all until
       after the completion of the test, while Apache was logging every hit.
     * Their Apache configuration disables KeepAlive, an important real-world
       optimization. (It has been pointed out that the tests do not use KeepAlive in
       any case).

Configuration of Samba

     * Their Samba configuration sets the widelinks parameter to "no". This setting
       increases the system call overhead for file name lookups considerably. The
       penalty is especially severe on SMP systems.
     * All 144 clients used in the tests were Windows 95 and 98 systems. For reasons
       known best to the Samba folks, Samba performs better with NT clients than
       with Windows 95 and 98 clients.
     * It does not appear that Samba was set up to use all of the (multiple)
       ethernet controllers on the system.

Non-issues

   A few complaints that have been sent to us probably do not figure into the test
   results. We list them here in the hopes of helping to slow their propagation and
   improve the quality of information out there.
     * Some complaints have been raised about the test being run on a 4GB server,
       even though the Linux kernel, in its default form, can only use 960M of that.
       Patches can be applied to make 2GB available fairly easily. But, in any case,
       they claim that NT was limited (with the maxmem parameter) to 1G of memory,
       so this aspect of the test was fair. It would have been more straightforward
       of them, however, to have simply remove the other 3G from the system.
     * It has been noted that the test system was running kerneld, portmap and NFS,
       both of which should have been unnecessary in this situation. But their
       presence should not have caused problems either.
     * There have been speculations that the test system may have been running out
       of file handles, but further research suggests this is not likely to have
       been the case. The 2.2 kernel has a default limit of 4096 - rather higher
       than previous versions had - and that should have been sufficient in this
       case.

Summary

   It seems clear that the Linux system in this test was not performing up to its
   full capability. Giving Mindcraft the benefit of the doubt, it could be said
   that, while they clearly had an NT expert present, they were lacking in Linux
   expertise and failed to set up the system in an optimal way. A rerun of this test
   - with suitable Linux expertise at hand - would seem to be in order.

   [16]Eklektix, Inc. Linux powered! Copyright 1999 [17]Eklektix, Inc. all rights
   reserved.
   Linux ® is a registered trademark of Linus Torvalds

References

   1. https://static.lwn.net/
   2. https://static.lwn.net/
   3. https://static.lwn.net/daily/
   4. https://static.lwn.net/op/Contact.html
   5. http://www.ChangeLog.net/log/1999/special/mindcraft/
   6. http://www.reseau.nl/news/article/mindcraft.html
   7. http://perso.wanadoo.fr/sebastien.tanguy/mindcraft.html
   8. http://www.kegel.com/mindcraft_redux.html
   9. http://www.mindcraft.com/whitepapers/nts4rhlinux.html
  10. http://www.mindcraft.com/company/services.html
  11. http://www.mindcraft.com/whitepapers/nts4sol26web.html
  12. http://www.mindcraft.com/whitepapers/nts4nw5filesvr.html
  13. http://www.novell.com/advantage/nw5/nw5-mindcraftcheck.html
  14. http://www.mindcraft.com/whitepapers/response-novell-nts4nw5filesvr.html
  15. http://www.cpureview.com/
  16. http://www.eklektix.com/
  17. http://www.eklektix.com/index.html