_________________________________________________________________ IRQTUNE -- A LINUX X86 IRQ PRIORITY OPTIMIZER Copyright 1996 by Craig Estey. This FAQ is revision 0.2 Last updated: Mon Aug 26 04:53:01 PDT 1996 See the Changes section at the bottom of this document. irqtune changes the IRQ priority of devices to allow devices that require high priority and fast service (e.g. serial ports, modems) to have it. With irqtune, a 3X speedup of serial/modem throughput is possible. _________________________________________________________________ WHERE DO I GET IRQTUNE? irqtune is free software under the terms and conditions of the GNU Public License. See the file COPYING, included in the distribution, for details. * It's available via anonymous ftp from: ftp://www.best.com/pub/cae/irqtune.tgz (gzipped tar archive) * The author is Craig Estey, (cae@best.com). * This FAQ file is available both in html and text versions included in the tar file. This FAQ is also available online via http://www.best.com/~cae/irqtune _________________________________________________________________ HOW DO I KNOW IF I NEED IRQTUNE? You are running Linux on an x86 PC, other architectures to be implemented later--Sorry. You probably need irqtune, if you are experiencing any of the following: * SLIP/PPP transfers seem slow. For example, using a 28.8 (or better) modem, the effective throughput is approximately 700 bytes/second instead of the expected 2500 bytes/second. * A running SLIP/PPP dies at random times. * Serial connections are slow or drop data. * Netscape hangs mysteriously or stalls when trying to access a web page. * Equivalent serial/PPP programs under Windoze run much faster than under Linux. * Disk accesses seem to interfere with SLIP/PPP. * Interrupt handlers for specialized, time critical devices don't get control when they need to. _________________________________________________________________ WHAT IS ACTUALLY HAPPENING TO CAUSE THESE PROBLEMS? When the PC boots Linux, the timer is given, by default, the highest IRQ priority in the system (it's IRQ 0 and thus, priority 0). On a standard configuration, the serial ports are priority 11 and 12!!! This means that 10 other devices have higher priority. Q: So what does IRQ priority do? When multiple devices are in contention to interrupt the CPU, their priority decides which interrupts will occur in what order. Q: When does this contention occur? After an arbitrary period of having interrupts disabled (e.g after a cli), at the point where they're reenabled (sti). This can happen in several places: * In an ISR that runs with interrupts locked, it happens in the epilog, just before attempting to execute the bottom-half. * The bottom-half itself may do a lock and unlock. * When a task that enters the kernel to do a system call, the system call handler may lock and unlock interrupts briefly. * Almost anywhere in the kernel are brief periods where it does a cli to lock interrupts and then an sti to unlock them again. Q: If there are multiple interrupts now pending,which one gets the service, the serial or some other? In the default configuration, the serial ISR will usually lose as it's priority 11. _________________________________________________________________ HOW DOES IRQTUNE HELP THIS? irqtune gives priority 0 to the whatever device you specify. If we specify a serial device, irqtune guarantees that the serial ISR gets control whenever a contention occurs. _________________________________________________________________ WHY DOES THE SERIAL INTERRUPT SERVICE REQUIRE THE HIGHEST PRIORITY? Q: Why does the serial device merit such special treatment? Serial devices are somewhat unique. Even though they have one of the slowest data rates (relative to a disk), they are the largest consumer of interrupts and are extremely sensitive to interrupt latency (the time from when a device yanks the IRQ line until its ISR is executed). Q: Could you give a concrete example of this? * For a modem running at 33.6, we can have a peak maximum (with compression) of 6700 bytes/second. The serial driver programs the silo to interrupt after 8 bytes, leaving a latency window of 8 bytes. This means that when the serial port yanks its IRQ line, it can still absorb 8 more characters before its buffer will overflow and data will be dropped * In terms of time, this means that the maximum that the serial ISR may be delayed is (8 / 6700) seconds or 1194 microseconds. It also means that the serial ISR will require 838 interrupts/second. * Currently there are 52 devices that install their ISR with the SA_INTERRUPT option. This means that they wish to run with interrupts disabled. I have not looked at all the devices, but just assume for the moment that all will run with interrupts locked for an arbitrary period. * Assume we've got 4 of them pending (remember, at higher priority). Assume that they will lock interrupts individually for 320, 300, 190, and 500 microseconds, respectively. Assume that the serial ISR also wants an interrupt. * This means that the serial ISR will have to wait for (320 + 300 + 190 + 500) or 1310us for these other higher priority ISR's to run to completion. This is greater than the maximum of 1194 us. * With internal modems, we probably just see start/stop behavior causing a slowdown. With an external modem, data will be dropped. This will cause PPP to see a CRC error and request transmission. Remember, the entire packet must be retransmitted, which means we just wasted 296-1500 bytes (depending on the selected MRU). With an MRU of 1500 bytes, we just wasted 23.4% of the bandwidth in a given second--all for the loss of a single byte! * Worse yet, consider a 115 Kbps ISDN card that is masquerading as a serial device (Yes, I know it should be a DMA device). In this case, the maximum delay the serial ISR can wait is 695 us. And that's just one ISDN channel. Try adding the other and, well, golly :-). Q: In this example, how would boosting serial IRQ priority help? * If the serial IRQ had priority 0, it would get in before the others. It could also re-interrupt after any individual lockout window. Thus, the maximum it would be forced to wait would be largest individual time, not the summation of all of these times. In this example, this means 500 us. which is now within the 1194 us. maximum for the serial ISR. * It has been my experience that the serial device must always win these battles for contention. When serial devices don't get what they need, when they need it, they slow horribly or drop data outright. _________________________________________________________________ DOESN'T THIS HURT THE PERFORMANCE OF OTHER DEVICES? Not really. In actual practice, most devices don't even notice the difference. Most other devices (e.g. disks, tape, ethernet) are DMA devices. The DMA does most of the work, thus greatly reducing their need for interrupts. If the device allows a request queue, it may function autonomously on several requests, producing only one interrupt for the entire batch. Furthermore, serial interrupt services are, themselves, very fast. They slam their data as quickly as possible and get out ASAP. No fancy calculations, just the minimum, mindless data transfer. Almost everything else is handled later, in the bottom-half with interrupts enabled. In fact, a serial ISR may have to re-interrupt it's own bottom-half several times. Those devices that do experience some slight slowdown are more likely to have long interrupt disable windows themselves. Having several smaller cli/sti windows is much better than one large cli/sti window--It's just harder to program. Q: But suppose I really want both fast serial and fast disk? Ultimately, it's a bit of a compromise. Which is better: * Reliable serial and slightly slower disk. * Slightly faster disk and unreliable serial/modem support. When paying an ISP for Internet access in $$$/hour, it's an easy decision :-). _________________________________________________________________ ISN'T THIS IRQ PRIORITY THING A BIT OF A NEW IDEA? No. It's actually an old idea. I've been doing device drivers since 1977 and Unix kernel work since 1981. I've personally written 8 serial drivers have used this many times commercially. Giving the serial device the highest priority is actually standard practice in many systems. With a 4Mhz CPU, these problems used to occur at 1200 baud :-) _________________________________________________________________ HOW DO I INSTALL IRQTUNE? * Unpack the tar file: gzip -d It will create a directory, irqtune in whatever is the current directory. Go to that directory: cd ./irqtune To install irqtune, enter: make install This will install the files: /sbin/irqtune /sbin/irqtune_mod.o _________________________________________________________________ HOW DO I USE IRQTUNE? DON'T I HAVE TO REBUILD MY KERNEL? No, you do not have to rebuild your kernel. irqtune uses insmod and rmmod to dynamically load and unload a kernel module. But you are correct in sensing that irqtune is a kernel patch. Q: Ok, if it's a kernel patch, why not just issue a kernel patch like everybody else does (e.g. diff -u output)? irqtune will work even if you don't have the kernel source loaded. It uses insmod to load the patch, invoke it, and then unload it. The IRQ priority changes will last so long as the kernel is booted. Q: How do I invoke it? irqtune takes two arguments optional arguments: * irqtune master slave The only caveat is that you must specify the full pathname, evenif irqtune is placed in a directory that is in $PATH. This is required because irqtune uses argv[0] to locate its irqtune_mod.o file. The default is 3 14 which will work for many standard configurations. More on this later. Q: Could I do this from my /etc/rc.d/rc.local file? Yes. Just add a /sbin/irqtune line to this file and you're in business. You may also issue another irqtune command at any time. Q: But aren't kernel patches dependent on the particular revision of the kernel? What if my kernel revision is different from the kernel revision that you built it on? irqtune is 99.44% kernel revision independent. It is built using ELF binaries, so your insmod must understand them. Also, make sure that you have the correct insmod for your kernel. But that's really about it. Q: But what if I don't have ELF binary support, how can I still do things? Well, I'd recommend that you upgrade your kernel as ELF binaries are cool :-) But if you insist, you'll just have to recompile irqtune. Just be sure that /usr/src/linux/include is installed. The exact procedure for building a.out binaries can vary with compiler revision, so be sure to check your documentation on this (You may need to add a parameter or two). * Then, just type: make clean make sbin make install That should do the trick. But, if it doesn't, look at revisionnumbers on insmod, gcc, ld, and the kernel for incompatibilities. Upgrade as necessary. _________________________________________________________________ WHAT ABOUT MY NON-STANDARD HARDWARE CONFIGURATION? irqtune defaults for a standard IRQ configuration. It assumes that the highest priority device should be on IRQ 3. This is normally the first serial port on standard configurations, which is what you want. Q: How do I determine what my IRQ configuration is? Just type cat /proc/interrupts and you'll get something like: 0: 8578913 timer 1: 109547 keyboard 2: 0 + cascade 3: 86470 + serial 4: 197648 + serial 13: 1 math error 14: 93123 + Ux4F Note that /proc/interrupts only reports on active devices. So to scope out the serial IRQ's ideally you'd have X Windows up with your serial mouse and be connected via PPP to the net. Q: OK, I've got the output from /proc/interrupts, what do I do with it? The leftmost number is the IRQ number. The rightmost column is the internal device name (not to be confused with /dev names). In the above case, the two serial ports are on IRQ 3 and IRQ 4. Just use the lower number, in this case 3: * /sbin/irqtune 3 This sets IRQ 3 to the highest priority. In fact, before we invoked irqtune, the IRQ number was also its priority: IRQ PRIOR 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 After this command, the IRQ priorities are now: IRQ PRIOR 0 5 1 6 2 7 3 0 4 1 5 2 6 3 7 4 Q: BTW, What's the cascade device I saw in the output of /proc/interrupts? Glad you asked. There are actually two interrupt controllers, a master and a slave. The slave is cascaded to the master via its IRQ 2. The master controls IRQ's 0-7 and the slave controls IRQ's 8-15. You actually may select two high IRQ priorities, one for the master and one for the slave. irqtune defaults the slave to IRQ 14, which is normally the disk controller. Although the normal notation is to refer to IRQ's as 0-15, it may be easier to understand if we refer to the master IRQ's as M0-M7 and the slave IRQ's as S0-S7. Q: But I've also got an Ethernet controller on IRQ 12. What about that? In other words, your configuration might look something like this: 0: 8578913 timer 1: 109547 keyboard 2: 0 + cascade 3: 86470 + serial 4: 197648 + serial 12: 17968 + eth 13: 1 math error 14: 93123 + Ux4F In this case, you might want to use: * /sbin/irqtune 3 12 because you want your ethernet card to have a higher priority than the diskcontroller. Actually if you did have this configuration, setting 3 14 (the default) would make the ethernet card, the lowest priority device in the system. In our new notation IRQ 12 is S4, and the resulting priority would be: IRQ M/S PRIOR 0 M0 5 1 M1 6 2 M2 7 3 M3 0 4 M4 1 5 M5 2 6 M6 3 7 M7 4 8 S0 12 9 S1 13 10 S2 14 11 S3 15 12 S4 8 13 S5 9 14 S6 10 15 S7 11 Q: Suppose I also had a serial multiplexer card on IRQ 11? Once again, your configuration might look something like this: 0: 8578913 timer 1: 109547 keyboard 2: 0 + cascade 3: 86470 + serial 4: 197648 + serial 11: 197648 + sermux 12: 17968 + eth 13: 1 math error 14: 93123 + Ux4F This configuration is a bit tricky because now we've got a serial device on the slave controller. It would be much better to put all serial cards on the master controller. Things would stay much simpler. In this case you would want to use: * /sbin/irqtune 2 11 The resulting priorities would be more complex and would result in somethinglike: IRQ M/S PRIOR 0 M0 13 1 M1 14 2 M2 0 3 M3 8 4 M4 9 5 M5 10 6 M6 11 7 M7 12 8 S0 5 9 S1 6 10 S2 7 11 S3 0 12 S4 1 13 S5 2 14 S6 3 15 S7 4 The reason things would be better if all serial devices were on the master is that now you have serial devices at priorities 0, 8, and 9. Q: So what's wrong with that? Well, we boosted the priority of the serial multiplexer at the expense of the regular serial ports. The only way to allow all serial ports equally high priority is to group them on consecutive IRQ's and set the high priority for the lowest of those IRQ's. _________________________________________________________________ HOW CAN I TELL IF IRQTUNE ACTUALLY DID ANYTHING FOR ME? Well, first off, if PPP/SLIP was dying mysteriously, it will probably be more reliable. Secondly, run without it and get a feel for the transfer rate: * Hit many favorite web sites and note the transfer rates in bytes/second. Make life easy. Netscape is at least one browser that reports transfer rates in bytes/second in the status line. * FTP reports the transfer time of a file in bytes/second. Download (or upload) a few files (300K or greater to smooth out the benchmark) and note the transfer rates. * Try several things of varying duration, different times of day, different sites to accomodate variations in network loading. Don't stop until there is an average set of numbers that are more or less repeatable. Repeat this using irqtune and note the transfer times again. NOTE: IRQTUNE just won't quit--if you want to test in the original mode again, reboot the system first. _________________________________________________________________ WHAT IF I STILL DON'T SEE ANY REAL IMPROVEMENT? It's a matter of probability. Performance measurement is as much art as science. * First, verify that there are no more obvious problems like bad config, bad cabling, etc. * We're much more likely to see improvement on a DX2/66 than a Pentium/166. With the 166, we may be overpowering the problem. We'll still have a problem, we just won't notice it as much because the Pentium has the extra speed to burn. A badly tuned Ferrari may still outperform a well-tuned VW :-) * System loading is very light. The problems that irqtune will fix are more likely to happen when more devices and more work are added. * What problem was occurring before? Which of the symptoms listed earlier is happening? If performance was 2500 bytes/second before using irqtune, we're less likely to notice the smaller jump to, say, 2800. * Verify that irqtune got the correct IRQ numbers for the specific configuration. For example, suppose our primary serial port is on IRQ 3, but we gave irqtune the value 4. We just made the serial device on IRQ 3 into priority 14, not priority 0. * We may have a rogue interrupt service that disables interrupts for something outrageous, say 2 ms. This is singularly longer than the 1194 us. in the earlier example. irqtune will still help, but the real solution here is to reduce interrupt lockout times in the other device below the 1194 us. threshold. _________________________________________________________________ WHAT ABOUT OTHER REMEDIES I'VE HEARD ABOUT? Q: What about disabling Van Jacobsen header compression? This reduces the amount of bottom-half processing the system has to do at the expense of larger packets being sent. Q: What about using ``hdparm -u'' to set the interrupt-unmask flag in hard disk driver? It suffers from the same problem as the bottom-half routine. The disk controller typically uses IRQ 14 and 15. While the slave interrupt controller would probably allow preemption, the master (on IRQ 2) would not because the priority of all slave devices is higher than the serial IRQ priority. Q: What about adjusting the MRU/MTU numbers? This will have less of an effect now. In fact, we normally reduced the MRU to a minimum (296) to reduce the bottom-half processing and latency at the expense of adding extra overhead bytes due to the reduced packet size. We may now actually be able to increase the MRU to regain the efficiency. Beware: Do this slowly as the optimal may not be 1500. The flip buffer in the serial/tty drivers is only 512 bytes. Q: What about going to newer kernel revisions? Although irqtune will work surprisingly well with just about any kernel revision, the low level IRQ handlers and device drivers have been vastly improved in the 2.0.X kernels. This will only improve irqtune's effect. In fact, 2.0.X and irqtune actually complement one another. _________________________________________________________________ DIDN'T YOU GIVE ANOTHER EXPLANATION BEFORE, INVOLVING BOTTOM-HALF ROUTINES? Yes. * My error was that I had looked at all the existing IRQ code in Linux, when I was first investigating the serial problem. I spaced out, misread the code, and concluded the EOI was done in the epilog. I also was looking at the slow interrupt prolog/epilog. * Anyway, I then did measurements and verified that the serial ISR never got control at an effective rate higher than 1000 bytes/sec. * It was clear that something was holding up the serial ISR. * This pointed to a priority problem. * I coded up irqtune and tried it. * irqtune worked. I went from 700 bytes/second to 2500 bytes/second. * I was so happy surfing at 3X my former speed, that I didn't bother to try to track down the culprit exactly as I had used this priority fix before. * After running it for about a month, I decided to package it up so others could use it. In my haste to get it out, I wrote the explanation too quickly. I also wanted to get some feedback from various configurations and change things accordingly. It was a case of round up the usual suspects. * In fact, now I'll be adding kernel tracing that will give some real world numbers to back things up. When I have completed this analysis, I'll add them to this FAQ. Many thanks to those that pointed this out. _________________________________________________________________ CHANGES FAQ 0.2 Changes: * Major rewrite and expansion of the problem explanation section * More thorough explanation of how and why irqtune works * Explanation of why serial devices must be highest priority * Impact on other devices * Cleaner and better installation instructions * Better benchmarking section * Problem resolution section * Explanation of my prior misread on the EOI thing