[SDBUG] interrupts in top broken with SMP kernel

[Michael J McCafferty]

Mike,
         Thanks that is very interesting info about adding multiple PCI cards.
         My problem is that vmstat and top do *not* show the CPU % 
used by interrupts with the mp kernel, but *do* when the uniproc 
kernel is used. This is important to me because my testing is only 
able to generate so many packets per second and bandwidth (therefore 
interrupts per second), which ends up being under 100% cpu time on 
the system, so I can't just see what the max the system can handle 
is. My goal is to determine the % cpu used by the same test scenario 
and determine which of 3 systems runs a the lowest peak cpu 
utilization in interrupts. Interrupt handling is what the CPU spends 
the most time doing during the real work that these systems will do 
after the test.
         The problem is that the same software behaves differently 
with the mp kernel and the uniproc kernel. I was counting on the 
system telling me the % of maximum interrupt time. For me, the 
interrupts per second is worthless without knowing how many 
interrupts per second is the maximum.
         Part of my goal is to determine if a Core2Duo 6300 
(2x1.86GHz cores) system will have a higher capacity than a firewall 
than a P4 3.0GHz. These are also compared to a Celeron 3.3GHz. All of 
these are candidates to replace the existing Celeron 2.8GHz. The 
interesting thing is that the existing production system shows 
significantly higher interrupt time than the tests imply it should... 
when handling real life traffic. The most obvious difference is that 
it's handling hundreds of hosts instead of just the two test hosts.
         I performed this test on all of the systems and found that 
difference was not as great as one would expect (see below). Then, I 
realized that I had forgotten to use the mp kernel when testing the 
Core2 Duo system. The question in my head that I really need to 
actually *know* the answer to, not guess, surmise, or calculate based 
on conjecture, is: is the 2 cores better than 1 in this specific 
application and using the mp kernel.

         Your thoughts are appreciated.

Thanks,
Mike





At 08:39 AM 4/9/2007, you wrote:
>Hey Mike,
>
>When I am benching systems for hardware performance there are two main
>utilities I use.
>
>1. vmstat(8)
>The area in vmstat you are interested are the two columns concerning 
>interrupts:
>   System
>       in: The number of interrupts per second, including the clock.
>       cs: The number of context switches per second.
>
>You will also be interested in the 'id' column, if this drops into
>single digits you might have interrupt misses or other odd problems.
>Especially if you are working in a realtime environment, like a DNS
>server, Firewall, or VoIP application.
>
>2. top(1)
>
>Top has many bits of useful information. However there is the 'hi'
>indicator at the top. I always assumed that this was 'hardware
>interrupts', however with a quick look through the man page I couldn't
>find any description of that column. I have seen a direct corelation
>between the hi column and activity on the pci bus. IE: If I add more
>cards, the 'hi' column will increase.
>
>
>One important thing to note, is that when I am adding cards to the PCI
>bus I see a non linear increase in hardware interrupt load. Below are
>a few examples:
>
>1 pci card: 2% hi load
>2 pci cards: 15% hi load
>3 pci cards: 60% hi load
>
>I have attributed this behavior with both cards requesting interrupts
>at the same time, and having to retry, thus queuing up interrupts over
>time.
>
>Hope that helps.
>
>Cheers!
>Mike
>
>On 4/9/07, Michael J McCafferty <mike at m5computersecurity.com> wrote:
>>All,
>>     When I use the generic uniproc kernel and run top or systat vmstat
>>I can get info on the % of cpu for interrupts. However, when using the
>>generic bsd.mp kernel on a Core2Duo system I can not get anything to
>>tell me the % of cpu for interrupts.
>>     I can get the number of interrupts per second, but not the % of
>>the CPU that is handlign interrupts. This is a metric I need to know
>>as it will help me extrapolate the ultimate performance limit of the
>>systems I am testing.
>>     Does anyone have any thoughts on how to determine the % of CPU
>>that is handlign intgerrupts on a Core2Duo with teh mp kernel ?
>>
>>Thanks,
>>Mike
>>
>>--
>>************************************************************
>>Michael J. McCafferty
>>Principal, Security Engineer
>>M5 Hosting
>>858-576-7325 Voice
>>http://www.m5hosting.com
>>************************************************************
>>
>>_______________________________________________
>>SDBUG mailing list
>>SDBUG at sdbug.org
>>http://lists.sdbug.org/mailman/listinfo/sdbug
>
>
>--
>Mike Joyce
>mjoyce at obstinate.org
>_______________________________________________
>SDBUG mailing list
>SDBUG at sdbug.org
>http://lists.sdbug.org/mailman/listinfo/sdbug