=title I<IRQTUNE> -- A Linux x86 IRQ Priority Optimizer
=head1 I<IRQTUNE> -- A Linux x86 IRQ Priority Optimizer
=for html <CENTER><EM>
Copyright 1996 by Craig Estey.
<BR>
This FAQ is revision 0.2
<BR>
Last updated: Mon Aug 26 04:53:01 PDT 1996
<BR>
See the Changes section at the bottom of this document.
</EM></CENTER><P>


I<irqtune> changes the IRQ priority of devices to allow devices that require
high priority and fast service (e.g. serial ports, modems) to have it.

B<With>
I<irqtune>,
B<a 3X speedup of serial/modem throughput is possible.>

=head1 Where do I get I<irqtune>?
I<irqtune> is free software under the terms and conditions of the GNU Public
License.  See the file COPYING, included in the distribution, for details.
=over 4
=item *
It's available via anonymous ftp from:
ftp://www.best.com/pub/cae/irqtune.tgz (gzipped tar archive)
=item *
The author is Craig Estey, (cae@best.com).
=item *
This FAQ file is available both in html and text versions included in the tar
file.  This FAQ is also available online via http://www.best.com/~cae/irqtune
=back

=head1 How do I know if I need I<irqtune>?
B<You are running Linux on an x86 PC, other architectures to be implemented>
B<later--Sorry.>

B<You probably need>
I<irqtune>,
B<if you are experiencing any of the following:>
=over 4
=item *
SLIP/PPP transfers seem slow.  For example, using a 28.8 (or better) modem,
the effective throughput is approximately 700 bytes/second instead of the
expected 2500 bytes/second.
=item *
A running SLIP/PPP dies at random times.
=item *
Serial connections are slow or drop data.
=item *
Netscape hangs mysteriously or stalls when trying to access a web page.
=item *
Equivalent serial/PPP programs under Windoze run much faster than under Linux.
=item *
Disk accesses seem to interfere with SLIP/PPP.
=item *
Interrupt handlers for specialized, time critical devices don't get control
when they need to.
=back
=head1 What is actually happening to cause these problems?

When the PC boots Linux, the timer is
given, by default, the highest IRQ priority in the system (it's IRQ 0 and thus,
priority 0).  On a standard configuration, the serial ports are priority
11 and 12!!!  This means that 10 other devices have higher priority.

B<Q: So what does IRQ priority do?>

When multiple devices are in contention to interrupt the CPU, their priority
decides which interrupts will occur in what order.

B<Q: When does this contention occur?>

After an arbitrary period of having interrupts disabled (e.g after a I<cli>),
at the point where they're reenabled (I<sti>).  This can happen in several
places:
=over 4
=item *
In an ISR that runs with interrupts locked, it happens in the I<epilog>, just
before attempting to execute the I<bottom-half>.
=item *
The I<bottom-half> itself may do a lock and unlock.
=item *
When a task that enters the kernel to do a system call, the system call
handler may lock and unlock interrupts briefly.
=item *
Almost anywhere in the kernel are brief periods where it does a I<cli> to
lock interrupts and then an I<sti> to unlock them again.
=back
B<Q: If there are multiple interrupts now pending,>
B<which one gets the service,>
B<the serial or some other?>

In the default configuration, the serial ISR will usually lose as it's
priority 11.

=head1 How does irqtune help this?

I<irqtune> gives priority 0 to the whatever device you specify.  If we
specify
a serial device, I<irqtune> guarantees that the serial ISR gets control
whenever a contention occurs.

=head1 Why does the serial interrupt service require the highest priority?

B<Q: Why does the serial device merit such special treatment?>

Serial devices are somewhat unique.  Even though they have one of
the slowest data rates (relative to a disk), they are the largest consumer
of interrupts and are extremely sensitive to I<interrupt latency> (the time
from when a device yanks the IRQ line until its ISR is executed).

B<Q: Could you give a concrete example of this?>

=over 4
=item *
For a modem running at 33.6, we can have a peak maximum (with
compression) of 6700 bytes/second.  The serial driver programs the silo to
interrupt after 8 bytes, leaving a latency window of 8 bytes.  This means
that when the serial port yanks its IRQ line, it can still absorb 8 more
characters before its buffer will overflow and data will be dropped
=item *
In terms of time, this means that the maximum that the serial ISR may be
delayed is
(8 / 6700) seconds or 1194 microseconds.  It also means that the serial ISR
will require 838 interrupts/second.
=item *
Currently there are 52 devices that install their ISR with the SA_INTERRUPT
option.
This means that they wish to run with interrupts disabled.
I have not looked at all the devices, but just
assume for the moment that all will run with interrupts locked for an
arbitrary period.
=item *
Assume we've got 4 of them pending (remember,
at I<higher> priority).  Assume that they will lock interrupts individually
for 320, 300, 190, and 500 microseconds, respectively.  Assume that the
serial ISR also wants an interrupt.
=item *
This means that the serial ISR will have to wait for (320 + 300 + 190 + 500)
or 1310us
for these other higher priority ISR's to run to completion.  This is greater
than the maximum of 1194 us.
=item *
With internal
modems, we probably just see start/stop behavior causing a slowdown.
With an external modem, B<data will be dropped>.  This will cause PPP to see
a CRC error and request transmission.  Remember, the I<entire> packet must be
retransmitted, which means we just wasted 296-1500 bytes (depending on the
selected MRU).
With an MRU of 1500 bytes, we just
wasted 23.4% of the bandwidth in a given second--all for the loss of a single
byte!
=item *
Worse yet, consider a 115 Kbps ISDN card that is
masquerading as a serial device I<(Yes, I know it should be a DMA device)>.
In this case, the maximum delay the serial ISR can wait is 695 us.  And that's
just one ISDN channel.  Try adding the other and, well, I<golly :-)>.
=back

B<Q: In this example, how would boosting serial IRQ priority help?>
=over 4
=item *
If the serial IRQ had priority 0, it would get in I<before> the others.  It
could also B<re-interrupt> after any individual lockout window.  Thus, the
maximum
it would be forced to wait would be largest individual time, not the
B<summation> of all of these times.  In this example, this means 500 us.
which is now within the 1194 us. maximum for the serial ISR.
=item *
It has been my experience that the serial device B<must always> win these
battles for contention.  When serial devices don't get what they I<need>,
when they need it, they slow horribly or drop data outright.
=back

=head1 Doesn't this hurt the performance of other devices?

B<Not really>.  In actual practice, most devices don't even notice the difference.
Most other
devices (e.g. disks, tape, ethernet) are I<DMA> devices.  The I<DMA> does
most of the work, thus greatly reducing their need for interrupts.  If the
device allows a request queue, it may function autonomously on several
requests, producing only one interrupt for the entire batch.

Furthermore, serial interrupt services are, themselves, I<very> fast.  They
slam their data as quickly as possible and get out ASAP.  No fancy
calculations, just the minimum, mindless data transfer.  Almost everything
else is handled later, in the I<bottom-half> with interrupts enabled.  In
fact, a serial ISR may have to re-interrupt it's own I<bottom-half> several
times.

Those devices that
I<do> experience some slight slowdown are more likely to have long interrupt
disable windows themselves.  Having several smaller I<cli/sti> windows is
much better than one large I<cli/sti> window--It's just harder to program.

B<Q: But suppose I really want both fast serial and fast disk?>

Ultimately, it's a bit of a compromise.  Which is better:
=over 4
=item *
Reliable serial and I<slightly> slower disk.
=item *
I<Slightly> faster disk and I<unreliable> serial/modem support.
=back

I<When paying an ISP for Internet access in $$$/hour, it's an easy decision :-)>.

=head1 Isn't this IRQ priority thing a bit of a new idea?

No.  It's actually an old idea.  I've been doing device drivers since 1977
and Unix kernel work since 1981.  I've personally written 8 serial drivers
have used this many times commercially.  Giving the serial device the
highest priority is actually standard practice in many systems.
I<With a 4Mhz CPU, these problems used to occur at 1200 baud :-)>

=head1 How do I install I<irqtune>?
=SS1
=over 4
=item *
Unpack the tar file:
=for html <PRE>
gzip -d < irqtune.tgz | tar xvf -
</PRE>


=item *
It will create a directory, B<irqtune> in
whatever is the current directory.
=item *
Go to that directory:
=for html <PRE>
cd ./irqtune
</PRE>


=item *
To install I<irqtune>, enter:
=for html <PRE>
make install
</PRE>


=item *
This will install the files:
=for html <PRE>
/sbin/irqtune
/sbin/irqtune_mod.o
</PRE>


=back
=SS0

=head1 How do I use I<irqtune>?  Don't I have to rebuild my kernel?
No, you do B<not> have to rebuild your kernel.  I<irqtune> uses I<insmod> and
I<rmmod> to dynamically load and unload a kernel module.  But you are correct
in sensing that irqtune is a kernel patch.

B<Q: Ok, if it's a kernel patch, why not just issue a kernel patch like>
B<everybody else does (e.g. diff -u output)?>

I<irqtune> will work even if you don't have the kernel source loaded.  It uses
I<insmod> to load the patch, invoke it, and then unload it.  The IRQ priority
changes will last so long as the kernel is booted.

B<Q: How do I invoke it?>

I<irqtune> takes two arguments optional arguments:
=over 4
=item *
irqtune I<master> I<slave>
=back
The only caveat is that you must specify the full pathname, even
if I<irqtune> is placed in a directory that is in $PATH.  This is required
because I<irqtune> uses argv[0] to locate its B<irqtune_mod.o> file.

The default is I<3 14> which will work for many standard configurations.
More on this later.

B<Q: Could I do this from my /etc/rc.d/rc.local file?>

Yes. Just add a B</sbin/irqtune> line to this file and you're
in business.
You may also issue another I<irqtune> command at any time.

B<Q: But aren't kernel patches dependent on the particular revision of the>
B<kernel?  What if my kernel revision is different from the kernel revision>
B<that you built it on?>

I<irqtune> is 99.44% kernel revision independent.  It is built
using ELF binaries, so your I<insmod> must understand them.  Also, make sure
that you have the correct I<insmod> for your kernel.  But that's
really about it.

B<Q: But what if I don't have ELF binary support, how can I still do things?>

Well, I'd recommend that you upgrade your kernel as ELF binaries are cool :-)
But if you insist, you'll just have to recompile I<irqtune>.  Just be sure
that B</usr/src/linux/include> is installed.  The exact procedure for building
I<a.out> binaries can vary with compiler revision, so be sure to check
your documentation on this (You may need to add a parameter or two).
=SS1
=over 4
=item *
Then, just type:
=for html <PRE>
make clean
make sbin
make install
</PRE>


=back
=SS0
That should do the trick.  But, if it doesn't, look at revision
numbers on I<insmod>, I<gcc>, I<ld>, and the I<kernel> for incompatibilities.
Upgrade as necessary.

=head1 What about my non-standard hardware configuration?
I<irqtune> defaults for a standard IRQ configuration.  It assumes that the
highest priority device should be on IRQ 3.  This is normally the first serial
port on standard configurations, which is what you want.

B<Q: How do I determine what my IRQ configuration is?>

Just type B<cat /proc/interrupts> and you'll get something like:
=for html <PRE>
     0:  8578913   timer
     1:   109547   keyboard
     2:        0 + cascade
     3:    86470 + serial
     4:   197648 + serial
    13:        1   math error
    14:    93123 + Ux4F
</PRE>


Note that
B</proc/interrupts>
only reports on active devices.  So to scope out
the serial IRQ's ideally you'd have X Windows up with your serial mouse and
be connected via PPP to the net.

B<Q: OK, I've got the output from /proc/interrupts, what do I do with it?>

The leftmost number is the IRQ number.  The rightmost column is the B<internal>
device name (not to be confused with /dev names).  In the above case, the two
serial ports are on IRQ 3 and IRQ 4.  Just use the lower number, in this
case 3:
=over 4
=item *
B</sbin/irqtune> I<3>
=back
This sets IRQ 3 to the highest priority.

In fact, before we invoked I<irqtune>, the IRQ number was also its priority:
=for html <PRE>
    IRQ  PRIOR
     0     0
     1     1
     2     2
     3     3
     4     4
     5     5
     6     6
     7     7
</PRE>



After this command, the IRQ priorities are now:
=for html <PRE>
    IRQ  PRIOR
     0     5
     1     6
     2     7
     3     0
     4     1
     5     2
     6     3
     7     4
</PRE>



B<Q: BTW, What's the cascade device I saw in the output of /proc/interrupts?>


Glad you asked.  There are actually two interrupt controllers, a I<master>
and a I<slave>.  The I<slave> is I<cascaded> to the I<master> via its IRQ 2.
The I<master> controls IRQ's 0-7 and the I<slave> controls IRQ's 8-15.

You actually may select two high IRQ priorities, one for the I<master> and one
for the I<slave>.  I<irqtune> defaults the I<slave> to IRQ 14, which is
normally the disk controller.

Although the normal notation is to refer to IRQ's as 0-15, it may be easier to
understand if we refer to the I<master> IRQ's as M0-M7 and the I<slave> IRQ's
as S0-S7.

B<Q: But I've also got an Ethernet controller on IRQ 12.  What about that?>

In other words, your configuration might look something like this:
=for html <PRE>
     0:  8578913   timer
     1:   109547   keyboard
     2:        0 + cascade
     3:    86470 + serial
     4:   197648 + serial
    12:    17968 + eth
    13:        1   math error
    14:    93123 + Ux4F<P>
</PRE>


In this case, you might want to use:
=over 4
=item *
B</sbin/irqtune> I<3 12>
=back
because you want your ethernet card to have a higher priority than the disk
controller.  Actually if you did have this configuration, setting 3 14
(the default) would make the ethernet card, the lowest priority device in the
system.

In our new notation IRQ 12 is S4, and the resulting priority would be:
=for html <PRE>
    IRQ M/S  PRIOR
    0   M0     5
    1   M1     6
    2   M2     7
    3   M3     0
    4   M4     1
    5   M5     2
    6   M6     3
    7   M7     4
    8   S0     12
    9   S1     13
    10  S2     14
    11  S3     15
    12  S4     8
    13  S5     9
    14  S6     10
    15  S7     11
</PRE>



B<Q: Suppose I also had a serial multiplexer card on IRQ 11?>

Once again, your configuration might look something like this:
=for html <PRE>
     0:  8578913   timer
     1:   109547   keyboard
     2:        0 + cascade
     3:    86470 + serial
     4:   197648 + serial
    11:   197648 + sermux
    12:    17968 + eth
    13:        1   math error
    14:    93123 + Ux4F<P>
</PRE>


This configuration is a bit tricky because now we've got a serial device on
the I<slave> controller.  It would be much better to put all serial cards on
the I<master> controller.  Things would stay much simpler.

In this case you would want to use:
=over 4
=item *
B</sbin/irqtune> I<2 11>
=back
The resulting priorities would be more complex and would result in something
like:
=for html <PRE>
    IRQ M/S  PRIOR
    0   M0     13
    1   M1     14
    2   M2     0
    3   M3     8
    4   M4     9
    5   M5     10
    6   M6     11
    7   M7     12
    8   S0     5
    9   S1     6
    10  S2     7
    11  S3     0
    12  S4     1
    13  S5     2
    14  S6     3
    15  S7     4
</PRE>


The reason things would be better if all serial devices were on the I<master>
is that now you have serial devices at priorities 0, 8, and 9.

B<Q: So what's wrong with that?>

Well, we boosted the priority of the serial multiplexer at the expense of the
regular serial ports.  The only way to allow all serial ports equally high
priority is to group them on consecutive IRQ's and set the high priority for
the lowest of those IRQ's.

=head1 How can I tell if I<irqtune> actually did anything for me?

Well, first off, if PPP/SLIP was dying mysteriously, it will probably be more
reliable.

Secondly, run without it and get a feel for the transfer rate:
=over 4
=item *
Hit I<many> favorite web sites and note the transfer rates in
bytes/second.  Make life easy.  Netscape is at least one browser
that reports transfer rates in bytes/second in the status line.
=item *
FTP reports the transfer time of a file in bytes/second.  Download (or upload)
a few files (300K or greater to smooth out the benchmark) and note the
transfer rates.
=item *
Try several things of varying duration, different times of day, different
sites to accomodate variations in network loading.  Don't stop until
there is an average set of numbers that are more or less repeatable.
=back

Repeat this using I<irqtune> and note the transfer times again.

I<NOTE: IRQTUNE just won't quit--if you want to test in the original mode>
I<again, reboot the system first.>

=head1 What if I still don't see any real improvement?

It's a matter of probability.  Performance measurement is as much art as
science.
=over 4
=item *
First, verify that there are no more obvious problems like bad config, bad
cabling, etc.
=item *
We're much more likely to see improvement on a DX2/66 than a Pentium/166.
With the 166, we may be I<overpowering> the problem.  We'll still have a
problem, we just won't notice it as much because the Pentium has the
extra speed to burn.
I<A badly tuned Ferrari may still outperform a well-tuned VW :-)>
=item *
System loading is very light.  The problems that I<irqtune> will fix are
more likely to happen when more devices and more work are added.
=item *
What problem was occurring before?  Which of the symptoms listed earlier is
happening?  If performance was 2500
bytes/second I<before> using I<irqtune>, we're less likely to notice the
smaller jump to, say, 2800.
=item *
Verify that I<irqtune> got the correct IRQ numbers for the specific
configuration.
For example, suppose our primary serial port is on IRQ 3, but we gave
I<irqtune> the value I<4>.  We just made the serial device on IRQ 3 into
priority 14, not priority 0.
=item *
We may have a B<rogue> interrupt service that disables interrupts for
something outrageous, say 2 ms.  This is singularly longer than the 1194 us. in
the earlier example. I<irqtune> will still help, but the real solution here
is to reduce interrupt lockout times in the other device below the 1194 us.
threshold.
=back

=head1 What about other remedies I've heard about?

B<Q: What about disabling Van Jacobsen header compression?>

This reduces the amount of I<bottom-half> processing the system has to do at
the expense of larger packets being sent.

B<Q: What about using "hdparm -u" to set the interrupt-unmask flag in hard>
B<disk driver?>

It suffers from the same problem as the I<bottom-half> routine.
The disk controller typically uses
IRQ 14 and 15.  While the I<slave> interrupt controller would
probably allow preemption,
the I<master> (on IRQ 2) would not because the priority of all I<slave>
devices is higher than the serial IRQ priority.

B<Q: What about adjusting the MRU/MTU numbers?>

This will have less of an effect now.  In fact, we normally reduced the MRU
to a minimum (296) to reduce the I<bottom-half> processing and <flip-buffer>
latency at the expense of
adding extra overhead bytes due to the reduced packet size.  We may now
actually be able to increase the MRU to regain the efficiency.

I<Beware: Do this slowly as the optimal may not be 1500.  The flip buffer>
I<in the serial/tty drivers is only 512 bytes.>

B<Q: What about going to newer kernel revisions?>

Although I<irqtune> will work surprisingly well with just about any kernel
revision, the low level IRQ handlers and device drivers have been vastly
improved in the 2.0.X
kernels.  This will only improve I<irqtune>'s effect.  In fact, 2.0.X and
I<irqtune> actually complement one another.

=head1 Didn't you give another explanation before, involving I<bottom-half> routines?
B<Yes.>
=over 4
=item *
My error was that I had looked at all the existing IRQ code in Linux,
when I was first
investigating the serial problem.  I spaced out, misread the code, and
concluded the EOI was done in the epilog.  I also was looking at the I<slow>
interrupt prolog/epilog.
=item *
Anyway, I then did measurements and verified that
the serial ISR never got control at an effective rate higher than 1000
bytes/sec.
=item *
It was clear that something was holding up the serial ISR.
=item *
This pointed to a priority problem.
=item *
I coded up I<irqtune> and tried it.
=item *
I<irqtune> worked.  I went from 700 bytes/second to 2500 bytes/second.
=item *
I was so happy surfing at 3X my former speed, that I didn't bother to try
to track down the culprit I<exactly> as I had used this priority fix before.
=item *
After running it for about a month, I decided to package it up so others
could use it.  In my haste to get it out, I wrote the explanation too quickly.
I also wanted to get some feedback from various configurations and change
things accordingly.
It was a case of I<round up the usual suspects>.
=item *
In fact, now I'll be adding kernel tracing that will give some real world
numbers to back things up.  When I have completed this analysis, I'll add them
to this FAQ.
=back
I<Many thanks to those that pointed this out.>

=head1 Changes

B<FAQ 0.2 Changes:>
=SS1
=over 4
=item *
Major rewrite and expansion of the problem explanation section
=item *
More thorough explanation of how and why I<irqtune> works
=item *
Explanation of why serial devices must be highest priority
=item *
Impact on other devices
=item *
Cleaner and better installation instructions
=item *
Better benchmarking section
=item *
Problem resolution section
=item *
Explanation of my prior misread on the EOI thing
=back
=SS0