Ergebnis für URL: http://www.kegel.com/mindcraft_redux.html On Mindcraft's April 1999 Benchmark
"First they ignore you, then they laugh at you, then they fight you, then you
win". -- [1]Gandhi (?)
Executive summary: The Mindcraft benchmark proved to be a wake-up call, and the
Linux community responded effectively. Several problems which caused Apache to
run slowly on Linux were found and resolved.
* As of mid-1999, Linux/Apache performance was identical to NT/IIS performance
under light load, and respectable under heavy load.
* As of May 2000, performance on Mindcraft-like benchmarks [2]finally may be
substantially improved on 4 CPU tests.
* As of February 2001, [3]performance on the SAP database benchmark on a 4 CPU
machine is dramatically better with 2.4 compared to 2.2.
* In January 2002, [4]Ingo Molnar's 0(1) scheduler was added to the
[5]2.5.2-pre10 kernel. This should help Linux scale better to many processors
and many running threads. (The new scheduler will probably not be included by
default in Linux distributions until 2003, but it is already available as a
patch to the 2.4.17 kernel for the few people who might need it.)
* By early 2003, the NGPT and NPTL projects have succeeded in bringing
high-performance POSIX-compliant threading to Linux. The first commercial
distribution to include NPTL will likely be Red Hat 8.1 in April 2003.
* Graphs showing remarkable progress in performance of 8-CPU SMP Linux systems
are online at [6]www-106.ibm.com/developerworks/linux/library/l-kperf.
* In May 2003, [7]Microsoft released file serving benchmarks showing Windows
2003 outperforming Linux. Here we go again!
I've split the summary of recent benchmark results (including the Mindcraft
benchmarks) off into a separate page, "[8]NT vs. Linux Server Benchmark Graphs",
at the request of readers who liked the data but weren't interested in the
history of the Linux kernel hacking community's response to the benchmarks.
* [9]Introduction
* [10]Previous Benchmarks
* [11]Kernel issue #1: TCP bug
* [12]Kernel issue #2: Wake-One vs. the Thundering Herd Updated 3 Oct 2000
* [13]Kernel issue #3: SMP Bottlenecks in 2.2 Kernel
* [14]Other Apache users getting help solving performance problems
* [15]Kernel issue #4: Interrupt Bottlenecks Updated 2 March 2000
* [16]Kernel issue #5: Mysterious network slowdown
* [17]Kernel issue #6: 2.2.x/NT TCP slowdown
* [18]Kernel issue #7: Scheduler Updated 27 Jan 2000
* [19]Kernel issue #8: SMP Bottlenecks in 2.4 kernel Updated 19 May 2000
* [20]Kernel issue #9: csum_partial_copy_generic Updated 17 Sept 2000
* [21]Kernel issue #10: getname(), poll() optimizations
* [22]Kernel issue #11: Reducing lock contention, poll overhead in 2.4 4 June
2000
* [23]Kernel issue #12: Poor disk seek behavior in 2.2, new elevator code in
2.4 Updated 4 Oct 2000
* [24]Kernel issue #13: Fast Forwarding / Hardware Flow Control 18 Sept 2000
* [25]Kernel tuning issue: hitting TIME_WAIT 31 March 2000
* [26]Suggestions for future benchmarks
* [27]Related reading
____________________________________________________________________________
Introduction
In March 1999, Microsoft commissioned [28]Mindcraft to carry out a [29]comparison
between NT and Linux showing that NT was 2 to 3 times faster than Linux. This
provoked responses from [30]Apacheweek.org, [31]Eric Lee Green, [32]Jeremy
Allison, and [33]Linux Weekly News, among others. Microsoft [34]posted a rebuttal
to the responses (evidently Microsoft takes this very [35]seriously).
The responses generally claimed that Mindcraft had not configured Linux properly,
and gave specific examples. Both Mindcraft and the Linux community agree that
good tuning information for Linux is hard to find.
Why the Outcry?
The Linux community responded to Mindcraft's announcements with hostility, at
least partly because of Mindcraft's attitude. Mindcraft stated "We posted notices
on various Linux and Apache newsgroups and received no relevant responses". and
concluded that the Linux 2.2 kernel wasn't as well supported as NT.
Mindcraft did not seem to take the time to become familliar with all the
appropriate forums for discussion, and apparantly did not respond to requests for
further information (see section III of [36]Eric Green's response). Others have
had better success; in particular, [37]three key kernel improvements all came
about in the course of normal support activities on Usenet, the linux-kernel
mailing list, and the Apache bug tracking database. I believe the cases
illustrated below indicate that free 2.2.x kernel support is better than
Mindcraft concluded.
Also, Mindcraft's [38]April 13th press release neglected to mention that
Microsoft sponsored the Mindcraft benchmarks, that the tests were carried out at
a Microsoft lab, and that Mindcraft had access to the highest level of NT support
imaginable.
Finally, Mindcraft did not try to purchase a support contract for Linux from e.g.
[39]LinuxCare or [40]Red Hat, both of whom were offering commercial support at
the time of Mindcraft's tests.
Mindcraft [41]proposed repeating the benchmarks at an independant testing lab to
address concerns that their testing was biased, but they have not yet addressed
concerns that their conclusions about Linux kernel support are biased.
Truth or FUD?
Mindcraft probably did tune NT well and Linux poorly -- but rather than assume
this fully accounts for Linux's poor showing, let's look for other things that
could have contributed. I'm going to focus on the web tests, since that's what
I'm familliar with.
Previous Benchmarks
Although Apache was designed for flexibility and correctness rather than raw
performance, it has done quite well in benchmarks in the past. In fact,
[42]Ziff-Davis's January 1999 benchmarks showed that "Linux with Apache beats NT
4.0 with IIS, hands down". (Also, [43]Apache currently beats Microsoft IIS at the
single processor SPECWeb96 benchmark, but this is with special caching software.)
Yet [44]Mindcraft found that Apache's performance falls off dramatically when
there are more than 160 clients ([45]see graph). Is this a contradiction?
Not really. [46]The benchmarks done by Jef Poskanzer, the author of the
high-performance server 'thttpd', showed that Apache 1.3.0 (among other servers)
has trouble above 125 connections on Solaris 2.6. The number of clients served by
Apache in the [47]Ziff-Davis tests above was 40 or less, below the knee found by
Poskanzer. By contrast, in the Mindcraft tests (and in the IIS [48]SPECWeb96
results), the server was asked to handle over 150 clients, above the point where
Poskanzer saw the dropoff.
Also, the January Ziff-Davis benchmarks used [49]a server with only 64 megabytes
of RAM, not enough to hold both the server code and the 60 megabyte [50]WebBench
2.0 document tree used by both Mindcraft and Ziff-Davis, whereas Mindcraft used
960 megabytes of RAM.
So it's not suprising that the Jan '99 Ziff-Davis and April '99 Mindcraft tests
of Apache got different results.
Does it matter?
These benchmarks are done on static pages, using very little of Apache's dynamic
page generation power. [51]Christopher Lansdown points out that the performance
levels reached in the test correspond to sites that receive over a million hits
per day on static pages. It's not clear that the results of such a test have much
relevance to typical big sites, which tend to use a lot of dynamically generated
pages.
Another objection to these benchmarks is that they don't accurately reflect the
real world of many slow connections. A realistic benchmark for a heavily-
trafficked Web server would involve 500 or 1000 clients all restricted to 28.8 or
56 Kbps, not 100 or so clients connected via 100baseT.
A benchmark that aims to deal with both of these concerns is the new
[52]SPECWeb99 benchmark. When it becomes available, it looks like it will set the
standard for how web servers should be benchmarked.
Nevertheless, [53]Linus seems to feel that until more realistic benchmarks (like
SPECWeb99) become available, benchmarks like the one Mindcraft ran are an
understandable if dirty compromise.
Kernel issue #1: TCP bug
Why does Apache fall off in the above tests above 160 active connections? It
appears the steep falloff may have been due to a TCP stack problem reported by
ariel@sgi.com and later by Karthik Prabhakar:
* ariel@sgi.com (Ariel Faigon) reported on 3 May 1999 (updates added):
"A couple of items you may find interesting.
1. For a long time the web performance team at SGI has noted that among the
three web servers we have been benchmarking: Apache, Netscape (both
enterprise and fasttrack), and Zeus, Apache is (by far) the slowest. In
fact an SGI employee (Mike Abbott) has done some optimizations which
made Apache run 3 (!) times faster on SPECWeb 96 on IRIX. It is our
intention to make these patches public soon. [They are now online at
[54]oss.sgi.com/projects/apache.]
2. When we tried to test our Apache patches on IRIX the expected 3x speedup
was easy to achieve. However when we ported our changes to Linux
(2.2.5), we were surprised to find that we don't even get into the
scalability game. A 200ms delay in connection establishment in the
TCP/IP stack in Linux 2.x was preventing Apache to respond to anything
more than 5 connections per second. We have been in touch with David
Miller on this and sent him a patch by Feng Zhou which eliminates this
bottleneck. This patch ... has made it into the [2.2.7] kernel [after
some modifications by David Miller]. So now we are back into optimizing
Apache. ..". [The patch affected the files [55]tcp.h and [56]tcp_ipv4.c,
e.g. it notes that Nagle shouldn't be used for the final FIN packet.]
(6 May 1999): "As for our changes to Apache. They are much more
significant and make Apache run 3 times faster on SPECWeb 96. I just
talked to the author and made sure we are releasing them to the Apache
group when we're ready. We just don't want to be too hasty in this. We
want to make it right, clean and accepted by the Apache guys. The
'patch' here is pretty big. ... It includes:
o A page cache
o Performance tunables adjusted to the max
o Changed some critical algorithms with faster ones (this is long, I
hope to have more details when we release).
o Eliminated system calls where they weren't needed (so Apache is
less dependent on the underlying OS)"
* Karthik Prabhakar reports on a problem with Apache 1.3.4 and 1.3.6 on Linux
kernel 2.2.5:
"I've seen load fall off well below 160 clients (for eg., 3 specweb clients
with 30 processes each). I can't explain it yet, especially the fact that the
performance stays low even after the test concludes. This behavior seems
limited to apache".
He has reported this as a bug to the Apache bug tracking system; see
[57]Apache bug #4268.
"The mystery continues. I got round to trying out 1.3.6 again this evening,
this time on 2.2.7. I did _not_ see the performance drop off. Just to verify,
I rechecked on the stock 2.2.5 kernel, and the drop off is there.
So _something_ has been fixed between 2.2.5 and 2.2.7 that has made this
problem go away. I'll keep plugging away as I get spare time to see if I can
get the problem to occur. ...
Compiling 1.3.6 in a reasonable way, along with a few minor tweaks in linux
2.2.7 gives me about 2-4 times the peak performance of the default 1.3.4 on
2.2.5. I simply compiled [Apache] with pretty much all modules disabled....
I'm using the highperformance-conf.dist config file from the distribution".
See also [58]Karthik's post on linux-kernel and its followups.
This sounds rather like the behavior Mindcraft reported ("After the restart,
Apache performance climbed back to within 30% of its peak from a low of about
6% of the peak performance").
Kernel issue #2: Wake-One vs. the Thundering Herd
(Note: According to the Linux Scalability Project's [59]paper on the thundering
herd problem, a "task exclusive" wake-one patch is now integrated into the 2.3
kernel; however, according to [60]Andrea, as of 2.4.0-test10, it still wakes up
processes in same order they were put to sleep, which is not optimal from a
caching point of view. The reverse order would be better.
See also Nov 2000 measurements by Andrew Morton (andrewm@uow.edu.au); [61]post 1,
[62]post 2, and [63]Linus' reply.)
* Phillip Ezolt, 5 May 1999, in linux-kernel ( [64]"Overscheduling DOES happen
with high web server load".):
"When running a SPECWeb96 strobe run on Alpha/linux, I found that when the CPU
is pegged, 18% of the time is spent in the scheduler".
(Russinovich [65]talked about something like this in his critique of Linux.)
This post started a [66]very lively thread in linux-kernel (now on its
[67]second week). Looks like the scheduler (and possibly Apache) are in for
some changes.
* Rik van Riel, 6 May 1999, in linuxperf [68](Re: [linuxperf] Possible fix for
Mindcraft Apache problem):
... The main bug with the web benchmark remains. The way Linux and Apache
'cooperate', there's a lot of trouble with the 'thundering herd' problem. That
is, when a signal comes in, all processes are woken up and the scheduler has
to select one from the dozens of new runnable processes.... The real solution
is to go from wake-all semantics to a wake-one style so we won't have the
enormous runqueues the guy at DEC [Phillip Ezolt] experienced. The good news
is that it's a simple patch that can probably be fixed within a few days...
* Tony Gale, 6 May 1999, in linuxperf ( [69]Re: [linuxperf] Possible fix for
Mindcraft Apache problem):
Apache uses file locking to serialise access to the accept call. This can be
very expensive on some systems. I haven't found the time to run the numbers on
Linux yet for the 10 or so different server models that can be employed to see
which is the most efficient. Check [70]Stephens UNPv1 2nd Edition Chapter 27
for details.
* Andrea Arcangeli, May 12th, 1999, in linux-kernel ( [71][patch] wake_one for
accept(2) [was Re: Overscheduling DOES happenwith high web server load.] and
[72]2.2.8_andrea1.bz2):
I released a new andrea-patch against 2.2.8. This new one has my new wake-one
on accept(2) strightforward code (but to get the improvement you must make
sure that your apache tasks are sleeping in accept(2), a strace -p `pidof
apache` should tell you that).
The patch is linked to from [73]here.
* David Miller's [74]reply to the above:
... on every new TCP connection, there will be 2 spurious and unnecessary
wakeups, and these originate in the write_space socket callback because as we
free up the SYN frames we wakeup listening socket sleepers. I've been working
today on solving this very issue.
* Ingo Molnar, May 13th, 1999, in linux-kernel ( [75]Re: [RFT] 2.2.8_andrea1
wake-one [Re: Overscheduling DOES happen with high web server load.]):
note that pre-2.3.1 already has a wake-one implementation for accept() ... and
more coming up.
* Phillip Ezolt (ezolt@perf.zko.dec.com), May 14th, 1999, in linux-kernel (
[76]Great News!! Was: [RFT] 2.2.8_andrea1 wake-one ):
I've been doing some more SPECWeb96 tests, and with Andrea's patch to 2.2.8
([77]ftp://ftp.suse.com/pub/people/andrea/kernel/2.2.8_andrea1.bz)
**On identical hardware, I get web-performance nearly identical to Tru64!**
...
Tru64 ~4ms
2.2.5 ~100ms
2.2.8 ~9ms
2.2.8_a ~4ms
... Time spent in schedule has decreased, as shown by this Iprobe data: ...
The number of SPECWeb96 MaxOps per second have jumped has well.
**Please, put the wakeone patch into the 2.2.X kernel if it isn't already. **
Larry Sendlosky tried this patch, and [78]says:
Your 2.2.8 patch really helps apache performance on a single cpu system, but
there is really no performance improvement on a 2 cpu SMP system.
The latest version of the wake-one patch is listed [79]below.
See also:
* Dimitris Michailidis (dimitris@sgi.com), 14 May 1999, in linux-kernel (
[80][PATCH] scheduler fixes and improvements) -- several improvements on the
2.2.8 scheduler
* Andrea Arcangeli (andrea@suse.de), 15 May 1999, in linux-kernel ( [81][patch]
andrea's cleanedup buffer.c for 2.2.9 ) -- several improvements on the 2.2.9
buffers and scheduler, and
Andrea Arcangeli (andrea@suse.de), 21 May 1999, in linux-kernel ( [82]Re:
andrea buffer code (2.2.9-C.gz) ) -- update of same. Might have some SMP
bottleneck fixes, too.
Kernel issue #3: SMP Bottlenecks in 2.2 Kernel
* Juergen Schmidt, May 19th, 1999, in linux-kernel ( [83]Bad apache perfomance
wtih linux SMP), asked what could make Apache do poorly under SMP.
Andi Kleen [84]replied:
One culprit is most likely that the data copy for TCP sending runs completely
serialized. This can be fixed by replacing the
skb->csum = csum_and_copy_from_user(from, skb_put(skb, copy), copy, 0, &err);
in tcp.c:tcp_do_sendmsg with
unlock_kernel();
skb->csum = csum_and_copy_from_user(from, skb_put(skb, copy), copy, 0, &err);
lock_kernel();
The patch does not violate any locking requirements in the kernel...
[To fix your connection refused errors,] try:
echo 32768 > /proc/sys/fs/file-max
echo 65536 > /proc/sys/fs/inode-max
Overall it should be clear that the current Linux kernel doesn't scale to CPUs
for system load (user load is fine). I blame the Linux vendors for advertising
it, although it is not true. ... Work to fix all these problems is underway
[2.3 will be fixed first, then the changes will be backported to 2.2].
[Note: Andi's TCP unlocking fix appears to be in [85]2.2.9-ac3.]
Andrea Arcangeli [86]responded describing his own version of this fix (
[87]ftp://ftp.suse.com/pub/people/andrea/kernel/2.3.3_andrea2.bz2 ) as less
cluttered:
If you look at my patch (the second one, in the first one I missed the
reaquire_kernel_lock done before returning from schedule, woops :) then you'll
see my approch to address the unlock-during-uaccess. My patch don't change
tcp/ip ext2 etc... but it touch only uaccess.h and usercopy.c. I don't like to
put unlock_kernel all over the place.
Juergen Schmidt, 26 May 1999, on linux-kernel and new-httpd, ( Linux/Apache
and SMP - my fault ), retracted his earlier problem report:
I reported "disastrous" performance for Linux and Apache on a SMP system.
To doublecheck, I've downloaded a clean kernel source (2.2.8 and 2.2.9) and
had to realize, that those do *not* show the reported penalty when running on
SMP systems.
My error was to use the installed kernel sources (which I patched from 2.2.5
to 2.2.8 - after seeing the first very bad results). But those sources already
had been modified before the machine came to me. Should have thrown them away
in the first place :-( ...
Please take my excuses for this confusion.
Others have [88]reported modest performance gains (20% or so) with Andrea's
SMP fix, but only when serving largish files (100 kilobytes).
Juergen has now [89]finished his testing. Unfortunately, he neglected to
compile Apache with [90]-DSINGLE_LISTEN_UNSERIALIZED_ACCEPT, which ([91]
according to Andrea) significantly hurt Apache performance.
If Juergen missed that, it means it's too hard to figure out. To make it
easier to get good performance in the future, we need the wake-one patch
added to a stable kernel (say, 2.2.10), and we need Apache's configuration
script to notice that the system is being compiled for 2.2.10 or later, and
automatically select SINGLE_LISTEN_UNSERIALIZED_ACCEPT.
Other Apache users getting help solving performance problems
* Mike Whitaker (mike@altrion.org), 22 May 1999, in linuxperf ( [92]High load
under Apache1.3.3/mod_perl 1.16/Linux 2.2.7 SMP ), described an interesting
performance problem:
Our typical webserver is a dual PII450 with 1G, and split httpd's, typically
300 static to serve the pages and proxy to 80-100 dynamic to serve the
mod_perl adverts. Unneeded modules are diabled and hostname lookups turned
off, as any sensible person would.
There's typically between one and three mod_perl hits/page on top of the usual
dozen or so inline images...
The kernel (2.2.7) has MAX_TASKS upped to 4090, and the
unlock_kernel/lock_kernel around csum_and_copy_from_user() in tcp_do_sendmsg
that Andi Kleen suggested.
Performance is .. interesting. Load on the machine fluctuates between 10 and
120, while the user CPU goes from 15% (80% idle) to 180% (0% idle, machine
*crawling*), about once every minute and a half. vmstat shows the number of
processes in a run state to range from 0 (when load is low) to 30-40, and the
static servers manage a mighty 60-70 peak hits/sec. Without the dynamic
httpd's everything *flies*...
After being advised to try a kernel with wake-one support, [93]he wrote back:
We're up with 2.3.3 plus Andi Kleen's tcp_do_sendmsg patch plus Apache
sleeping in accept() on one production server, and comparing it against a
2.2.7 plus tcp_do_sendmsg patch plus Apache sleeping in flock(). Identical
systems (dual PII450, 1G, two disk controllers).
As far as I can *tell*, the wake-one patch is definitely doing its stuff: the
2.2.7 machine still has cycles of load into three figures, and the 2.3.3
machine hasn't actully managed a load of 1 yet.
UNFORTUNATELY, observation suggests that the 2.3.3 machine/Apache combination
is dropping/ignoring about one connection in ten, maybe more. (Network error:
connection reset by peer.)
[94]His next update, on May 25th, reads:
More progress from the bleeding edge:
(Reminder: the config here is split static/mod_perl httpd's, with a pretty
CPU-intensive mod_perl script serving ads as an SSI as the probable
bottleneck)
Linux kernel 2.2.9 plus the [95]2.2.9_andrea3 (wake-one) patch seems to do the
trick: can handle hits at a speed which suggests it's pushing the adverser to
close to its observed maximum. (As I said in a previous note, avoid 2.2.8 like
the plague: it trashes HDs - see threads on linux-kernel for details.)
However... When it *does* get overstressed, BOY does it get overstressed. Once
the idle CPU drops to zero (i.e. its spending most of its time processing
advert requests, everything goes unpleasantly pearshaped, with a load of 400+,
and the number of httpd's on both types of server *well* above MaxClients (in
fact, suspiciously close to MaxClients + MinSpareServers). Spikes in demand
can cause this, and once you get into this state, getting out again under the
load of prgressively more backlogging requests is not easy: in fact from
experience the only way is to which the machine out of the (hopefully short
TTL) DNS round-robin while it dies down.
The potentially counterintuitive step at this point is to *REDUCE* MaxClients,
and hope that the tcp Listen queue will handle a load surge. Experience
suggests this does in fact work.
(Aside: this is a perfect case for using something like [96]Eddieware's load
balancing DNS).
* Eric Hicks, 26 May 1999, in linux-kernel ( [97]Apache/kernel problem? ):
... I'm having some big problems in which it appears that a single PII 400Mhz
or a single AMD 400 will outrun a dual PII 450 at http requests from Apache.
...
Data for HTTP Server Tests: 100 1MByte mpeg files stored on local disks.
Results:
+ AMD 400Mghz K6, 128MB, Linux 2.0.36; handles 1000 simultaneous clients @
57.6Kbits/sec.
+ PII 400Mghz, 512MB, Linux 2.0.36; handles 1000 simultaneous clients @
57.6Kbits/sec.
+ Dual PII/450Mghz, 512MB, Linux 2.0.36 and 2.2.8; handles far less than
300 simultaneous @57.6Kbits/sec [and even then, clients were seeing 5
second connection delays, and 'reset by peer' and 'connection time out'
errors]
I advised him to try [98]2.2.9_andrea3; he said he'd try it and report back.
Kernel issue #4: Interrupt Bottlenecks
[99]According to Zach, the Mindcraft benchmark's use of four Fast Ethernet cards
and a quad SMP system exposes a bottleneck in Linux's interrupt processing; the
kernel spent a lot of time in synchronize_bh(). (A single Gigabit Ethernet card
would stress this bottleneck much less.) [100]According to Mingo, TCP throughput
scales much better with number of CPUs in 2.3.9 than it did in 2.2.10, although
he hasn't tried it with multiple Ethernets yet.
See also comments on reducing interrupts under heavy load by [101]Steve Underwood
and [102]Steven Guo.
See also [103]Linus's "State of Linux" talk at Usenix '99 where he talks about
the Mindcraft benchmark and SMP scalability.
See also [104]SCT's Jan 2000 comments about progress in scalability.
Softnet is coming! Kernel 2.3.43 adds the new [105]softnet networking changes.
Softnet changes the interface to the networking cards, so every single driver
needs updating, but in return network performance should scale much better on
large SMP systems. (For more info, see [106]Alexy's readme.softnet, [107]his
softnet-howto, or [108]his Feb 15 post about how to convert old drivers.
)
The Feb '00 thread [109]Gigabit Ethernet Bottlenecks (especially its [110]second
week) has lots of interesting tidbits about how what interrupt (and other)
bottlenecks remain, and how they are being addressed in the 2.3 kernel.
Ingo Molnar's [111]post of 27 Feb 2000 describes interrupt-handling improvements
in the IA32 code in great detail. These will be moving into the core kernel in
2.5, it seems.
Kernel issue #5: Mysterious network slowdown This one is a bug, not a scalability issue.
Several 2.2 users have reported that sometimes networking slows down to 1 to 10% of
normal, with high ping times, and that cycling the interface fixes the problem
temporarily.
* [112]Øystein Svendsen reported on 29 June 1999:
After upgrading to the 2.2 series we have from time to time experienced severe
slow-downs on the TCP performance... The performance goes back to normal when
I take down the interface and reinsert the eepro100 module into the kernel.
After I've done that, the performance is fine for a couple of days or maybe
weeks.
* [113]David Stahl reported on 29 June 1999:
I've got 3 machines running 2.2.10 [with multiple] 3COM 3C905/905b PCI
[cards]... After approximately 2 days of uptime, I will start to see ping
times on the local lan jump to 7-20 seconds. As others have noticed, there is
no loss -- just some damn high latency. ... It seems to be dependant upon the
network load -- lighter loads lead to longer periods between problems. The
problem ALSO is gradual -- it'll start at 4 second pings, then 7 second pings
about 20 minutes later, than 30 minutes later it's up to 12-20 seconds.
* [114]Another eepro100 report.
* [115]A tulip report. Less repeatable.
* David Stahl wrote on 13 July 1999:
What DID fix the problem was a private reply from someone elese (sorry about
the credit, but i'm not in the mood to sieve 10k emails right now), to try the
alpha version of the latest 3c59x.c driver from Donald Becker
([116]http://cesdis.gsfc.nasa.gov/linux/drivers/vortex.html).
3c59x.c:v0.99L 5/28/99 is the version that fixed it, from
[117]ftp://cesdis.gsfc.nasa.gov/pub/linux/drivers/test/3c59x.c
* On 23 Sep 1999, [118]Alexey posted a one-line patch that clears up a similar
mysterious slowdown. 2.2.13 and Red Hat 6.1 already have this patch applied.
On three Red Hat 6.0 systems I know of with Masq support compiled in,
connected to cable modems, this patch fixed a bug which caused very high
pings after even short bursts of heavy TCP transfers to distant hosts.
* [119]Rickard Cedergren and [120]Michael Brown reported about October 21st on
linux-kernel that that although Alexey's patch greatly improved the problem,
it is not totally gone.
* [121]Tony Hoyle is also seeing occasional long delays with 2.2.13.
* [122]Jeremy Fitzhardinge reported another big delay; the replies say it's
likely caused by a particular Tulip driver.
Kernel issue #6: 2.2.x/NT TCP slowdownPetru Paler, July 10 1999, in linux-kernel (
[123][BUG] TCP connections between Linux and NT ) reported that any kind of TCP connection
between Linux (2.2.10) and a NT Server 4 (Service Pack 5) slows down to a crawl. The
problem was much milder (6kbytes/sec) with 2.0.37. He included a log of a slow connection
made with tcpdump, which helped Andi Kleen see [124]that NT was taking a long time to ACK
a data packet, which was causing Linux to throttle back..
Solved: false alarm! It wasn't Linux' fault at all. Turns out NT needed to be told to not
use full duplex mode on the ethernet card.
Kernel issue #7: Scheduler
Phil Ezolt, 22 Jan 2000, in linux-kernel ( [125]Re: Interesting analysis of linux
kernel threading by IBM):
When I run SPECWeb96 tests here, I see both a large number of running process
and a huge number of context switches. ... Here's a sample of the vmstat data:
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
...
24 0 0 2320 2066936 590088 1061464 0 0 0 0 8680 7402 3 96 1
24 0 0 2320 2065752 590664 1061464 0 0 0 1095 11344 10920 3 95 1
Notice. 24 running process and ~7000 context switches.
That is a lot of overhead. Every second, 7000*24 goodnesses are calculated.
Not the (20*3) that a desktop system sees. This is a scalability issue. A
better scheduler means better scalability.
Don't tell me benchmark data is useless. Unless you can give me data using a
real system and where it's faults are, benchmark data is all we have.
SPECWeb96 pushes Linux until it bleeds. I'm telling you where it bleeds. You
can fix it or bury your head in the sand. It might not be what your system is
seeing today, but it will be in the future.
Would you rather fix it now or wait until someone else how thrown down the
performance gauntelet?
...
Here's a juicy tidbit. During my runs, I see 98% contention on the [2.2.14]
kernel lock, and it is accessed ALOT. I don't know how 2.3.40 compares,
because I don't have big memory support for it. Hopefully, Andrea will be kind
enough give me a patch, and then I can see if things have improved.
[Phil's data is for the web server undergoing the SPECWeb96 test, which is an
ES40 4 CPU alpha EV6 running Redhat 6.0 w/kernel v2.2.14 and Apache-v1.3.9 w/SGI
performance patches; the interfaces receiving the load are two ACENic gigabit
ethernet cards.]
Kernel issue #8: SMP Bottlenecks in 2.4 kernel
Manfred Spraul, April 21, 2000, in linux-kernel ( [126][PATCH] f_op->poll()
without lock_kernel()):
kumon@flab.fujitsu.co.jp noticed that select() caused a high contention for
the kernel lock, so here is a patch that removes lock_kernel() from poll().
[tested] with 2.3.99-pre5.
There was some discussion about whether this was wise at this late date, but
Linus and David Miller were enthusiastic. Looks like one more bottleneck bites
the dust.
On 26 April 2000, kumon@flab.fujitsu.co.jp [127]posted benchmark results in
Linux-Kernel with and without the lock_kernel() in poll(). The followups included
a kernel patch to improve checksum performance and [128]a patch for Apache 1.3 to
force it to align its buffers to 32-word boundaries. The latter patch, by Dean
Gaudet, earned praise from Linus, who relayed rumors that this can speed up
SPECWeb results by 3%. This was an interesting thread.
See also [129]LWN's coverage, and the [130]paragraph below, in which Kumon
presents some benchmark results and another patch.
Kernel issue #9: csum_partial_copy_generic
kumon@flab.fujitsu.co.jp, 19 May 2000, in linux-kernel ( [131][PATCH] Fast
csum_partial_copy_generic and more ) reports a 3% reduction in total CPU time
compared to 2.3.99-pre8 on i686 by optimizing the cache behavior of
csum_partial_copy_generic. The workload was [132]ZD's WebBench. He adds
The benchmark we used has almost same setting as the MINDCRAFT ones, but the
apache setting is [changed] slightly not to use symlink checking.
We used maximum of 24 independent clients and number of apache processes is
16. A four-way XEON procesor system is used, and the performance is twice and
more than a single CPU performance.
Note that in [133]ZD's benchmarks with 2.2.6, a 4 CPU system only achieved a 1.5x
speedup over a single CPU. Kumon is reporting a > 2x speedup. This appears to be
about the same speedup NT 4.0sp3 achieved with 4 CPUs at that number of clients
(24). It's encouraging to hear that things may have improved in the 11 months
since the 2.2.6 tests. When I asked him about this, Kumon said
Major improvement is between pre3 and pre5, poll optimization. Until pre4 (I
forget exact version), kernel-lock prevents performance improvement.
If you can retrieve l-k mails around Apr 20-25, the following mails will help
you understand the background.
[134]subject: namei() query
[135]subject: [PATCH] f_op->poll() without lock_kernel()
[136]subject: lockless poll() (was Re: namei() query)
[137]subject: "movb" for spin-unlock (was Re: namei() query)
On 4 Sept 2000, kumon [138]posted again, noting that his change still hadn't made
it into the kernel.
Kernel issue #10: getname(), poll() optimizations
On 22 May 2000, Manfred Spraul [139]posted a patch on linux-kernel which
optimized kmalloc(), getname(), and select() a bit, speeding up apache by about
1.5% on 2.3.99-pre8.
Kernel issue #11: Reducing lock contention, poll overhead in 2.4
On 30 May 2000, [140]Alexander Viro posted a patch that got rid of a big lock in
close_flip() and _fput(), and asked for testing. [141]kumon ran a benchmark, and
reported:
I measured viro's ac6-D patch with WebBench on 4cpu Xeon system. I applied to
2.4.0-test1 not ac6. The patch reduced 50% of stext_lock time and 4% of the
total OS time. ... Some part of kmalloc/kfree overhead is come from do_select,
and it is easily eliminated using small array on a stack.
kumon then [142]posted a patch that avoids kmalloc/kfree in select() and poll()
when # of fd's involved is under 64.
Kernel issue #12: Poor disk seek behavior in 2.2, new elevator code in 2.4
On 20 July 2000, Robert Cohen (robert@coorong.anu.edu.au) [143]posted a report in
Linux-kernel listing netatalk (appletalk file sharing) benchmarks comparing 2.0,
2.2, and several versions of 2.4.0-pre. The elevator code in 2.4 seems to help
(some versions of 2.4 can handle 5 benchmark clients instead of 2) but ...
The more recent test4 and test5pre2 don't fair quite so well. They handle 2
clients on a 128 Meg server fine, so they're doing better than 2.2 but they
choke and go seek bound with 4 clients. So something has definitely taken a
turn for the worse since test1-ac22.
Here's [144]an update. The *only* 2.4 kernel versions that could handle 5 clients
were 2.4.0-test1-ac22-riel and 2.4.0-test1-ac22-class 5+; everything before and
after (up to 2.4.0-test5pre4) can only handle 2.
On 26 Sept 2000, Robert Cohen posted [145]an update which included a simple
program to demonstrate the problem, which appears to be in the elevator code.
[146]Jens Axboe (axboe@suse.de) responded that he and Andrea had a patch almost
ready for 2.4.0-test9-pre5 that fixes this problem.
On 4 Oct 2000, Robert Cohen posted [147]an update with benchmark results for many
kernels, showing that the problem still exists in 2.4.0-test9.
Kernel issue #13: Fast Forwarding / Hardware Flow Control
On 18 Sept 2000, Jamal (hadi@cyberus.ca) [148]posted a note in Linux-kernel
describing proposed changes to the 2.4 kernel's network driver interface; the
changes add hardware flow control and several other refinements. He says
Robert Olson and I decided after the OLS that we were going to try to hit the
100Mbps(148.8Kpps) routing peak by year end. I am afraid the bar has been
raised. Robert is already hitting with 2.4.0-test7 ~148Kpps with a ASUS CUBX
motherboard carrying PIII 700 MHZ coppermine with about 65% CPU utilization.
With a single PII based Dell machine i was able to get a consistent value of
110Kpps.
So the new goal is to go to about 500Kpps ;-> (maybe not by year end, but
surely by that next random Linux hacker conference)
A sample modified tulip driver (hacked by Alexey for 2.2 and mod'ed by Robert
and myself over a period of time) is supplied as an example on how to use the
feedback values. ...
I believe we could have done better with the mindcraft tests with these
changes in 2.2 (and HW FC turned on).
[update] BTW, I am informed that Linux people were _not_ allowed to change the
hardware for those tests, so I dont think they could have used these changes
if they were available back then.
Kernel tuning issue: hitting TIME_WAIT
On 30 March 2000, Takashi Richard Horikawa [149]posted a report in Linux-Kernel
listing SPECWeb96 results for both the 2.2.14 and 2.3.41. Performance between a
2.2.14 client and a 2.2.14 server was poor because few enough ports were being
used that ports were not done with TIME_WAIT by the time that port number was
needed again for a new connection. The moral of the story may be to tune the
client and servers to use as large a port range as possible, e.g. with
echo 1024 65535 > /proc/sys/net/ipv4/ip_local_port_range
to avoid bumping into this situation when trying to simulate large numbers of
clients with a small number of client machines.
On 2 April 2000, Mr. Horikawa [150]confirmed that increasing the local port range
with the above command solved the problem.
Suggestions for future benchmarks
Become familliar with [151]linux-kernel and the [152]Apache mailing lists as well
as the Linux newsgroups on Usenet (try [153]DejaNews power searches in forums
matching '*linux*').
Post your proposed configuration and see whether people agree with it. Also, be
open about your benchmark; post intermediate results, and see if anyone has
suggestions for improvements. You should probably expect to spend a week or so
mulling over ideas with these mailing lists during the course of your tests.
If possible, use a modern benchmark like [154]SPECWeb99 rather than the simple
ones used by Mindcraft.
It might be interesting to [155]inject latency into the path between the server
and the clients to more realistically model the situation on the Internet.
Benchmark both single and multiple CPUs, and single and multiple Ethernet
interfaces, if possible. Be aware that the networking performance of version
2.2.x of the Linux kernel does not scale well as you add more CPUs and Ethernet
cards. This applies mostly to static pages and cached dynamic pages; noncached
dynamic pages usually take a fair bit of CPU time, and should scale very well
when you add CPUs. If possible, use a cache to save commonly generated pages;
this will bring the dynamic page speeds closer to the static page speeds.
When testing dynamic content: Don't use the old model of running a separate
process for each request; nobody running a big web site uses that interface
anymore, as it's too slow. Always use a modern dynamic content generation
interface (e.g. mod_perl for Apache).
Configuring Linux
Tuning problems probably resulted in less than 20% performance decrease in
Mindcraft's test, so as of 3 October 1999, most people will be happy with a stock
2.2.13 kernel or whatever comes with Red Hat 6.1. The 2.4 kernel, when it's
available, will help with SMP performance.
Here are some notes if you want to see what people going for the utmost were
trying in June:
* As of June 1, Linux kernel 2.2.9 plus [156]2.2.9_andrea3 have been mentioned
as performing well on a dual-processor task (see above). (2.2.9_andrea3 seems
to include both a wake-one scheduler fix as well as an SMP unlock_kernel
fix.) (andrea3 only works on x86, I hear, so people with Alphas or PPC's will
have to apply some other wake-one and tcp copy kernel_unlock patch.)
Jan Gruber writes: "the 2.2.9_andrea3-patch doesn't compile with SMP Support
disabled. Andrea told me to use
[157]ftp://ftp.suse.com/pub/people/andrea/kernel-patches/2.2.9_andrea-VM4.gz
instead".
* On 7 June, Andrea Arcangeli asked:
If you are going to do bench I would like if you would bench also the patch
below.
[158]ftp://e-mind.com/pub/andrea/kernel-patches/2.2.9_andrea-perf1.gz
* On 11 Oct 1999, [159]Andrea Arcangeli posted his list of pending 2.2.x
patches, waiting to go into 2.2.13 or so. This includes several that might
help performance of SMP systems and systems undergoing heavy I/O. You might
consider trying these if you run into bottlenecks.
* The truly daring may wish to try using the kernel-mode http server,
[160]khttpd, as a front-end for Apache. It accellerates static web page
fetches greatly. It's at version 0.1, so use caution.
* linux-kernel ( [161]week 1, [162]week 2 ) is currently (8 June 1999)
discussing benchmarking Apache. [163]Linus Torvalds is in principle bullish
on using khttpd or something like it, and points out that NT is doing the
same kind of thing.
Configuring Apache
* The usual optimizations should be applied (all unused modules should be left
out when compiling, host name lookup should be disabled, and symbolic links
should be followed; see
[164]http://www.apache.org/docs/misc/perf-tuning.html)
* Apache should be compiled to block in accept, e.g.
env CFLAGS='-DSINGLE_LISTEN_UNSERIALIZED_ACCEPT' ./configure
* The [165]http://www.arctic.org/~dgaudet/apache/1.3/top_fuel.patch may be
worth applying. PC Week used top_fuel in their recent benchmarks. (See also
interesting comments by Dean Gaudet in [166]linux-kernel and [167]new-httpd.)
Supposedly, applying top_fuel.patch and using mod_mmap_static on a set of
documents can reduce the number of syscalls per request from 18 to 9.
* For static file benchmarks, try compiling mod_mmap_static into Apache (see
[168]http://www.apache.org/docs/mod/mod_mmap_static.html) and configuring
Apache to memory-map the static documents, e.g. by creating a config file
like this:
find /www/htdocs -type f -print | sed -e 's/.*/mmapfile &/' > mmap.conf
and including mmap.conf in your Apache config file.
* Several people have mentioned that using Squid as a front-end to Apache would
greatly accellerate static web page fetches.
Related reading
* A few Usenet posts showing people experiencing slowness with Apache or Linux:
+ [169]"Apache is not as fast as people claim??", 1999/04/05,
comp.infosystems.www.servers.unix "...when we run WebBench to test the
requests/sec and total throughput, Microsoft IIS 4 is 3 times faster for
both Linux and Mac OS X".
+ [170]"Re: Apache vs IIS 4: IIS 4 3 times faster", 1999/04/02,
comp.infosystems.www.servers.unix "Why are you surprised? I thought it
was common knowledge that Apache is slow. I haven't tested IIS, but I
did compare Apache against a number of other servers last year and found
a bunch that were three or four times faster".
* Ways to profile the kernel:
+ [171]Kernel Spinlock Metering for Linux IA32 - tools to measure SMP
spinlock contention. See also [172]some test results comparing 2.2 to
2.3.
+ [173]An example of someone using spinlock metering to find and fix a
kernel bottleneck in 2.3.19.
+ [174]Andrea Arcangeli's ikd
+ [175]sgi's gprof kernel profiling patch ([176]original announcement)
+ [177]Ingo Molnar's ktracer - for 2.1.x
[178]Example of ktracer use
[179]Example of both ktracer and ikd profile output
+ Christoph Lameter's perfstat patch, at Captech's [180]Linux Performance,
Stability and Scalability Project -- see also their 25 Oct 99 post on
linuxperf
* Ways to profile user programs:
+ The old favorite: compile with -pg, and analyze gmon.out with gprof.
+ [181]Mikael Pettersson's x86 performance-monitoring counters patch.
Supports 2.3.22 and 2.2.13. Includes list of other related tools.
+ [182]Using hardware performance counters with Linux by David Mentré
+ [183]PCL - The Performance Counter Library -- supports many
architectures.
+ [184]Stephan Meyer's MSR patch -- only supports up to 2.2.6. No longer
actively developed.
+ [185]Richard Gooch's MSR and PTC patch -- only supports 2.2. Requires
devfs.
* A few linux-kernel posts:
+ [186]"2.2.5 optimizations for web benchmarks?", 16 Apr 1999 -- Karthik
Prabhakar, about to do serious SPECWeb96 benchmarking, asks the right
questions. The followups are interesting.
+ [187]"Re: 2.2.5 optimizations for web benchmarks?", 16 Apr 1999 -- Dean
Gaudet's response. Interesting insights from an Apache insider.
+ [188]"[patch] new scheduler", 9 May 1999 -- the thread started by Rik
van Riel about possible scheduler changes
* [189]The smbtorture benchmark, which lets you test an SMB server like the big
boys
* Rik van Riel's [190]Linux Performance Tuning site
* [191]The Linux Scalability Project
* [192]The C10K problem - Why can't Johnny serve 10000 clients?
* [193]Banga and Druschel's paper on web server benchmarking
* [194]Linus's "State of Linux" talk at Usenix '99 where he talks about the
Mindcraft benchmark and SMP scalability.
* my [195]NT vs. Linux Server Benchmark Graphs page
* [196]A post on comp.unix.bsd.freebsd.misc from June '99 which mentions that
FreeBSD also has similar SMP scaling properties as Linux on tests like those
run by Mindcraft.
* [197]Mike Abbott of SGI's performance patches for Apache 1.3.9
* Note: Apache 2.0 supports sendfile(), which ought to help its flat file
performance.
____________________________________________________________________________
Copyright 1999-2003 Dan Kegel
dank@alumni.caltech.edu
[198]Last updated: 16 Jan 2003
[199][Return to www.kegel.com]
References
1. http://www.cbu.edu/Gandhi/
2. http://www.kegel.com/mindcraft_redux.html#csum_partial_copy_generic
3. http://boudicca.tux.org/hypermail/linux-kernel/2001week08/0779.html
4. http://marc.theaimsgroup.com/?l=linux-kernel&m=101042477122459&w=2
5. http://lwn.net/daily/2.5.2-pre10.php3
6. http://www-106.ibm.com/developerworks/linux/library/l-kperf/
7. http://www.veritest.com/clients/reports/microsoft/ms_netbench.pdf
8. http://www.kegel.com/nt-linux-benchmarks.html
9. http://www.kegel.com/mindcraft_redux.html#intro
10. http://www.kegel.com/mindcraft_redux.html#apache
11. http://www.kegel.com/mindcraft_redux.html#tcp
12. http://www.kegel.com/mindcraft_redux.html#wakeone
13. http://www.kegel.com/mindcraft_redux.html#smp
14. http://www.kegel.com/mindcraft_redux.html#cases
15. http://www.kegel.com/mindcraft_redux.html#interrupt
16. http://www.kegel.com/mindcraft_redux.html#slowdown
17. http://www.kegel.com/mindcraft_redux.html#tcp2
18. http://www.kegel.com/mindcraft_redux.html#specweb96
19. http://www.kegel.com/mindcraft_redux.html#smp24
20. http://www.kegel.com/mindcraft_redux.html#csum_partial_copy_generic
21. http://www.kegel.com/mindcraft_redux.html#optimizations
22. http://www.kegel.com/mindcraft_redux.html#optimizations2
23. http://www.kegel.com/mindcraft_redux.html#elevator
24. http://www.kegel.com/mindcraft_redux.html#hfc
25. http://www.kegel.com/mindcraft_redux.html#tcp3
26. http://www.kegel.com/mindcraft_redux.html#tips
27. http://www.kegel.com/mindcraft_redux.html#links
28. http://www.mindcraft.com/
29. http://www.mindcraft.com/whitepapers/nts4rhlinux.html
30. http://www.apacheweek.com/issues/99-04-16#mindcraft
31. http://www.linux-hw.com/~eric/mindcraft.html
32. http://www.linuxworld.com/linuxworld/lw-1999-04/lw-04-mindcraft.html
33. http://lwn.net/1999/features/MindCraft1.0.phtml
34. http://www.microsoft.com/ntserver/nts/news/msnw/nt4vLinux.asp
35. http://www.zdnet.com/zdnn/stories/news/0,4586,2263211,00.html?chkpt=hpqs014
36. http://members.tripod.com/~e_l_green/mindcraft.html
37. http://www.kegel.com/mindcraft_redux.html#tcp
38. http://www.mindcraft.com/company/nts4rhlinux-news.html
39. http://www.linuxcare.com/
40. http://store.redhat.com/commerce/store.cgi?page=/support.html
41. http://www.mindcraft.com/openbenchmark.html
42. http://www.zdnet.com/sr/stories/issue/0,4537,2196115,00.html
43. http://www.spec.org/osg/web96/results/res99q2/
44. http://www.mindcraft.com/whitepapers/nts4rhlinux.html
45. http://www.mindcraft.com/whitepapers/nts4rhlinux-webrps.gif
46. http://www.acme.com/software/thttpd/benchmarks.html
47. http://www.zdnet.com/sr/stories/issue/0,4537,2196115,00.html
48. http://www.spec.org/osg/web96/results/res99q2/
49. http://www.zdnet.com/sr/stories/issue/0,4537,387506,00.html
50. http://www.zdnet.com/zdbop/webbench/1main/1wrktree.htm
51. http://cs.alfred.edu/~lansdoct/mstest.html
52. http://www.spec.org/osg/web99/
53. http://www.linuxhq.com/lnxlists/linux-kernel/lk_9906_02/msg00000.html
54. http://oss.sgi.com/projects/apache/
55. http://kernelnotes.org/v22patch/patch-2.2.7/linux_include_net_tcp.h.html
56. http://kernelnotes.org/v22patch/patch-2.2.7/linux_net_ipv4_tcp_ipv4.c.html
57. http://bugs.apache.org/index/full/4268
58. http://kernelnotes.org/lnxlists/linux-kernel/lk_9905_02/msg00056.html
59. http://www.citi.umich.edu/projects/linux-scalability/reports/accept.html
60. http://boudicca.tux.org/hypermail/linux-kernel/2000week45/0178.html
61. http://marc.theaimsgroup.com/?l=linux-kernel&m=97331452723977&w=2
62. http://marc.theaimsgroup.com/?l=linux-kernel&m=97331452823984&w=2
63. http://marc.theaimsgroup.com/?l=linux-kernel&m=97331912328602&w=2
64. http://kernelnotes.org/lnxlists/linux-kernel/lk_9905_01/msg00655.html
65. http://winntmag.com/Magazine/Article.cfm?ArticleID=5048
66. http://kernelnotes.org/lnxlists/linux-kernel/lk_9905_01/index.html#00655
67. http://kernelnotes.org/lnxlists/linux-kernel/lk_9905_02/index.html#00010
68. http://mail.nl.linux.org/lists/linuxperf/1999-05/msg00100.html
69. http://mail.nl.linux.org/lists/linuxperf/1999-05/msg00101.html
70. http://www.amazon.com/exec/obidos/ASIN/013490012X/
71. http://kernelnotes.org/lnxlists/linux-kernel/lk_9905_02/msg00357.html
72. http://kernelnotes.org/lnxlists/linux-kernel/lk_9905_02/msg00618.html
73. http://kernelnotes.org/lnxlists/linux-kernel/lk_9905_02/msg00654.html
74. http://kernelnotes.org/lnxlists/linux-kernel/lk_9905_02/msg00740.html
75. http://kernelnotes.org/lnxlists/linux-kernel/lk_9905_02/msg00813.html
76. http://kernelnotes.org/lnxlists/linux-kernel/lk_9905_03/msg00069.html
77. ftp://ftp.suse.com/pub/people/andrea/kernel/
78. http://www.deja.com/getdoc.xp?AN=479687275
79. http://www.kegel.com/mindcraft_redux.html#tips
80. http://kernelnotes.org/lnxlists/linux-kernel/lk_9905_03/msg00027.html
81. http://lwn.net/1999/0520/a/andrea.html
82. http://linuxwww.db.erau.edu/mail_archives/linux-kernel/May_99/3381.html
83. http://x34.deja.com/viewthread.xp?AN=479309440
84. http://www.deja.com/threadmsg_ct.xp?AN=479923569
85. http://linuxtoday.com/stories/6537.html
86. http://linuxwww.db.erau.edu/mail_archives/linux-kernel/May_99/3277.html
87. ftp://ftp.suse.com/pub/people/andrea/kernel/
88. http://linuxwww.db.erau.edu/mail_archives/linux-kernel/May_99/3793.html
89. http://kernelnotes.org/lnxlists/linux-kernel/lk_9906_01/msg00411.html
90. http://www.kegel.com/mindcraft_redux.html#tips
91. http://kernelnotes.org/lnxlists/linux-kernel/lk_9906_01/msg00451.html
92. http://mail.nl.linux.org/lists/linuxperf/1999-05/msg00225.html
93. http://mail.nl.linux.org/lists/linuxperf/1999-05/msg00235.html
94. http://mail.nl.linux.org/lists/linuxperf/1999-05/msg00251.html
95. ftp://ftp.suse.com/pub/people/andrea/kernel/2.2.9_andrea3.bz2
96. http://www.eddieware.org/
97. http://linuxwww.db.erau.edu/mail_archives/linux-kernel/May_99/4022.html
98. ftp://ftp.suse.com/pub/people/andrea/kernel/2.2.9_andrea3.bz2
99. http://kernelnotes.org/lnxlists/linux-kernel/lk_9906_04/msg01100.html
100. http://kernelnotes.org/lnxlists/linux-kernel/lk_9906_05/msg00094.html
101. http://kernelnotes.org/lnxlists/linux-kernel/lk_9906_04/msg01518.html
102. http://kernelnotes.org/lnxlists/linux-kernel/lk_9906_04/msg01523.html
103. http://www.technetcast.com/tnc_program.html?program_id=17
104. http://lwn.net/2000/0203/a/tweedie.html
105. http://boudicca.tux.org/hypermail/linux-kernel/2000week07/0667.html
106. http://lwn.net/2000/0217/a/README.softnet.html
107. http://lwn.net/2000/0217/a/softnet-howto.html
108. http://boudicca.tux.org/hypermail/linux-kernel/2000week08/0308.html
109. http://kernelnotes.org/lnxlists/linux-kernel/lk_0002_01/msg00263.html
110. http://kernelnotes.org/lnxlists/linux-kernel/lk_0002_02/msg00017.html
111. http://www.lwn.net/2000/0302/a/irq.html
112. http://kernelnotes.org/lnxlists/linux-kernel/lk_9906_04/msg01439.html
113. http://kernelnotes.org/lnxlists/linux-kernel/lk_9906_04/msg01494.html
114. http://kernelnotes.org/lnxlists/linux-kernel/lk_9906_05/msg00288.html
115. http://kernelnotes.org/lnxlists/linux-kernel/lk_9906_05/msg00306.html
116. http://cesdis.gsfc.nasa.gov/linux/drivers/vortex.html
117. ftp://cesdis.gsfc.nasa.gov/pub/linux/drivers/test/3c59x.c
118. http://kernelnotes.org/lnxlists/linux-kernel/lk_9909_04/msg00406.html
119. http://kernelnotes.org/lnxlists/linux-kernel/lk_9911_03/msg00166.html
120. http://kernelnotes.org/lnxlists/linux-kernel/lk_9911_03/msg00968.html
121. http://kernelnotes.org/lnxlists/linux-kernel/lk_9911_04/msg00155.html
122. http://kernelnotes.org/lnxlists/linux-kernel/lk_9912_01/msg00426.html
123. http://kernelnotes.org/lnxlists/linux-kernel/lk_9907_02/msg00354.html
124. http://kernelnotes.org/lnxlists/linux-kernel/lk_9907_02/msg00359.html
125. http://boudicca.tux.org/hypermail/linux-kernel/2000week04/1196.html
126. http://boudicca.tux.org/hypermail/linux-kernel/2000week17/0964.html
127. http://boudicca.tux.org/hypermail/linux-kernel/2000week18/0122.html
128. http://boudicca.tux.org/hypermail/linux-kernel/2000week18/0229.html
129. http://www.lwn.net/2000/0427/bigpage.phtml#kernel
130. http://www.kegel.com/mindcraft_redux.html#csum_partial_copy_generic
131. http://boudicca.tux.org/hypermail/linux-kernel/2000week21/0841.html
132. http://www.zdnet.com/zdbop/webbench
133. http://www.kegel.com/nt-linux-benchmarks.html#bench14june1999
134. http://boudicca.tux.org/hypermail/linux-kernel/2000week17/0811.html
135. http://www.kegel.com/mindcraft_redux.html#smp24
136. http://boudicca.tux.org/hypermail/linux-kernel/2000week18/0122.html
137. http://boudicca.tux.org/hypermail/linux-kernel/2000week17/1085.html
138. http://boudicca.tux.org/hypermail/linux-kernel/2000week37/0180.html
139. http://boudicca.tux.org/hypermail/linux-kernel/2000week22/0244.html
140. http://boudicca.tux.org/hypermail/linux-kernel/2000week23/0715.html
141. http://boudicca.tux.org/hypermail/linux-kernel/2000week23/1028.html
142. http://boudicca.tux.org/hypermail/linux-kernel/2000week24/0048.html
143. http://boudicca.tux.org/hypermail/linux-kernel/2000week30/0567.html
144. http://boudicca.tux.org/hypermail/linux-kernel/2000week31/0353.html
145. http://boudicca.tux.org/hypermail/linux-kernel/2000week40/0135.html
146. http://boudicca.tux.org/hypermail/linux-kernel/2000week40/0202.html
147. http://boudicca.tux.org/hypermail/linux-kernel/2000week41/0555.html
148. http://boudicca.tux.org/hypermail/linux-kernel/2000week39/0375.html
149. http://boudicca.tux.org/hypermail/linux-kernel/2000week14/1119.html
150. http://boudicca.tux.org/hypermail/linux-kernel/2000week15/0132.html
151. http://www.tux.org/lkml/
152. http://dev.apache.org/mailing-lists.html
153. http://www.deja.com/home_ps.shtml
154. http://www.spec.org/osg/web99/
155. http://www.deja.com/viewthread.xp?AN=518197876&search=thread
156. ftp://ftp.suse.com/pub/people/andrea/kernel/2.2.9_andrea3.bz2
157. ftp://ftp.suse.com/pub/people/andrea/kernel-patches/2.2.9_andrea-VM4.gz
158. ftp://e-mind.com/pub/andrea/kernel-patches/2.2.9_andrea-perf1.gz
159. http://kernelnotes.org/lnxlists/linux-kernel/lk_9910_02/msg00668.html
160. http://www.fenrus.demon.nl/
161. http://www.linuxhq.com/lnxlists/linux-kernel/lk_9906_01/index.html#00311
162. http://www.linuxhq.com/lnxlists/linux-kernel/lk_9906_02/index.html#00000
163. http://www.linuxhq.com/lnxlists/linux-kernel/lk_9906_02/msg00000.html
164. http://www.apache.org/docs/misc/perf-tuning.html
165. http://www.arctic.org/~dgaudet/apache/1.3/top_fuel.patch
166. http://kernelnotes.org/lnxlists/linux-kernel/lk_9904_03/msg00309.html
167. http://www.humanfactor.com/cgi-bin/cgi-delegate/apache-ML/nh/1999/May/2nd-half/0090.html
168. http://www.apache.org/docs/mod/mod_mmap_static.html
169. http://www.dejanews.com/getdoc.xp?AN=463005092
170. http://www.dejanews.com/getdoc.xp?AN=462046707
171. http://oss.sgi.com/projects/lockmeter
172. http://www.progressive-comp.com/Lists/?l=linux-smp&m=93914383715173&w=2
173. http://kernelnotes.org/lnxlists/linux-kernel/lk_9910_03/msg01049.html
174. ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/ikd/
175. http://oss.sgi.com/projects/kernprof
176. http://gwyn.tux.org/hypermail/linux-kernel/1999week20/1079.html
177. http://kernelnotes.org/patch/21-p0376.html
178. http://www.uwsg.indiana.edu/hypermail/linux/kernel/9711.2/0270.html
179. http://www.uwsg.indiana.edu/hypermail/linux/kernel/9807.1/0496.html
180. http://opensource.captech.com/lpss/
181. http://www.csd.uu.se/~mikpe/linux/perfctr/
182. http://www.irisa.fr/prive/mentre/linux-counters/
183. http://www.fz-juelich.de/zam/PCL/
184. http://pobox.com/~smeyer/msr.html
185. ftp://ftp.atnf.csiro.au/pub/people/rgooch/linux/kernel-patches/v2.2/msr.desc
186. http://kernelnotes.org/lnxlists/linux-kernel/lk_9904_03/msg00146.html
187. http://kernelnotes.org/lnxlists/linux-kernel/lk_9904_03/msg00309.html
188. http://kernelnotes.org/lnxlists/linux-kernel/lk_9905_02/index.html#00163
189. http://kernelnotes.org/lnxlists/linux-kernel/lk_9905_03/msg00248.html
190. http://linuxperf.nl.linux.org/
191. http://www.citi.umich.edu/projects/citi-netscape/
192. http://www.kegel.com/c10k.html
193. http://www.cs.rice.edu/CS/Systems/Web-measurement/paper/paper.html
194. http://www.technetcast.com/tnc_program.html?program_id=17
195. http://www.kegel.com/nt-linux-benchmarks.html
196. http://x27.deja.com/getdoc.xp?AN=495336865
197. http://reality.sgi.com/mja/apache/
198. http://www.kegel.com/Changes-mindcraft_redux.html.txt
199. http://www.kegel.com/
Usage: http://www.kk-software.de/kklynxview/get/URL
e.g. http://www.kk-software.de/kklynxview/get/http://www.kk-software.de
Errormessages are in German, sorry ;-)