[SDBUG] OpenBSD on Dell issues ?

[Michael J McCafferty]

Can,
    Yes, I know that the 32-bit PCI slot will be the bottleneck before  
the 1.4G extrapolated max. It wouldn't be much fun to have the  
firewalls at 100% at any time anyway. I'd be looking for an upgrade  
long before the daily peak CPU was even 70%. Daily we are at 40% now,  
and I am feverishly working to make the upgrade.
    ...and this is combined bandwidth on all interfaces, full-duplex.  
Currently the networks plugged in to these firewalls are maxed at  
600Mbps for now... and we aren't using near that yet. When we do get  
get closer to that, we will start upgrading the Internet connections  
to GigE.
    The problem is that even a moderate DoS causes too many  
interrupts. I mentioned polling because ultiamtely there has to be  
some way to mitigate the interrupts. Can't go 10G if the interrupts  
stop you at 1.4G
    Now, the other night I decided to try FreeBSD 6.2-RELEASE in place  
of the OpenBSD snapshot box. It reported that it was about HALF the  
CPU interrupt time as the OpenBSD boxes. Is this because the kernel is  
letting both cores handle interrupts ? Or is this an error in the way  
that the kernel is reporting interrupt time at the CPU ? I wasn't able  
to generate enough pps or mbps with my 4 traffic generating boxes  
(realtek NICs) to get it over 50% so I can't tell if it's reporting  
correctly or if it's actually doing it.
    Since FreeBSD has PF, CARP and pfsync, that might be an option. No ?

    Thanks, your comments are appreciated.

Mike

Quoting Can Erkin Acar <canacar at gmail.com>:

> On 8/13/07, Michael J McCafferty <mike at m5computersecurity.com> wrote:
>> Well... I got the hardware, and the results are in:
>
> Nice, and good results. There is a problem with your extrapolation
> calculations though. The peak theoretical bandwidth of PCI (32 bit) is
> about 128MB/sec which is equivalent to 1Gb/s, so you will not be able
> to reach 1.4Gb using 32 bit slots. It is probably ok though, since the
> interrupts are the real problem that limit the pps rate, usually at a
> much lower bandwidth with small packets. And this is what matters with
> DoS attacks.
>
> For a good overview of PCI technologies you can have a look at:
> http://arstechnica.com/articles/paedia/hardware/pcie.ars/1
>
>
>> I have a more complete spreadsheet, but here are the highlights:
>>
> [snip]
>>         I think it is relevant to note that interrupts were handled  
>>  by a single
>> core of the dual core CPUs in each case.
>
> Yes, only userland benefits from multiple processors at the moment.
>
>>         What I have learned from this is that when it comes to PCIe or PCI-X
>> and 133MHz 64-bit vs. 32-bit PCI... it's not really that relevant for
>> x86/x86_64/amd64 firewalls with OpenBSD. The bottleneck is hardware
>> interrupts to the CPU. I actually already knew that the interrupts were
>> the bottleneck, but what I didn't know was how 1333MHz FSB vs. 1066MHz
>> FSB, or 32-bit PCI vs. PCIe, or OpenBSD 3.8, OpenBSD 4.1 or the latest
>> snapshot would affect the final performance. Of course I also didn't
>> know the difference between Xeon 5130 and Core2 Duo E6600 for interrupts
>> either.
>>         There is one faster model of CPU available for the Dell system. The
>> 3GHz Woodcrest CPU is about $500 MORE than the CPU tested, each. *If* it
>> scales linearly, this would bring the performance to ~2Gbps and
>> 400,000pps. This is still 100,000pps short of my desire to handle over
>> 500,000pps (about 48Mbps of 12Byte UDP packets, a relatively small DDoS
>> attack). Really, I'd like to be able to handle 100Mbps+ of 12Byte
>> packets, but I realize a million pps is pretty steep.
>>
>>         Why oh why won't someone smarter than me work on device polling for
>> OpenBSD ? Or is that even the solution ?
>
> I dont claim to be an expert, but claudio@ says polling is not the
> solution (it may actually increase latency). A good network card and a
> good matching driver would probably perform much better.
>
> Some developers are working on improving the network performance.
> There are also several 10Gb card drivers being worked on, so we really
> aim to support higher speeds. The latest profiling and tuning effort
> at the last hackathon was a start.
>
>
> Enjoy your new setup :)
>
> Can
> _______________________________________________
> SDBUG mailing list
> SDBUG at sdbug.org
> http://lists.sdbug.org/mailman/listinfo/sdbug
>



-- 
************************************************************
Michael J. McCafferty
Principal, Security Engineer
M5 Hosting
858-576-7325 Voice
http://www.m5hosting.com
************************************************************