[SDBUG] OpenBSD on Dell issues ?

[Michael J McCafferty]

Good info. Thank you !

The cards I have been using all along are dual-port Intel GigE cards.
For the current solution they are "Intel PRO/1000MT Dual Port Server
Adapter", they are PCI/PCI-X cards in a 32-bit PCI card slot. The Dell
that were tested are PCIe using the PCIe version of the same card, which
I think are actually called Intel PRO/1000PT. Both use the em driver.



On Thu, 2007-08-16 at 08:17 -0700, Jeremiah Gowdy wrote:
> While OpenBSD has made progress on their SMP support, FreeBSD's work over
> the past few years in the development and stabilization of the 5.0 and 6.0
> branches was primarily about highly scalable SMP support.  If you're working
> with a dual core system, it wouldn't suprise me that FreeBSD is better able
> to schedule interrupts to service your network traffic.
> 
> I'm joining the conversation late and I don't see anything in the quotes for
> this email regarding which ethernet chipset you're talking about, but what
> you're looking for in this situation is interrupt moderation.  Polling
> interrupts is not the solution.  If you have an intel chipset based driver
> "em", that driver does support interrupt moderation.  This will allow you to
> reach much higher speeds by not overloading your CPU with interrupts.  The
> key here is to not have one interrupt per packet, which is what happens
> without interrupt moderation.  Since the driver will see *all* the packets
> the network card has received when an interrupt is serviced, it makes no
> sense to have one interrupt per packet.  So on cards that support interrupt
> moderation, depending on settings and the overall traffic level, 3,000-4,000
> packets can be queued for each interrupt that fires.  This will cause a huge
> drop in CPU load.
> 
> This is probably also related to the performance difference you see in
> FreeBSD, because I am guessing the em driver defaults interrupt moderation
> to "on".  Even if your ethernet is not an Intel chipset (Broadcoms are
> really common on my Dells right now), most of the mainstream drivers should
> support interrupt moderation.  For more information on interrupt moderation
> see this link:
> 
> http://download.intel.com/design/network/applnots/ap450.pdf
> 
> I would say that using FreeBSD for your firewall is a perfectly good
> solution.  If I was building a firewall personally and I was worried about
> performance, I would probably buy a dual or quad port Intel gigabit
> PCI-Express ethernet card, although it might not be necessary.  If your
> system is PCI-Express based (which it can be even if you have PCI / PCI-X
> ports, Dell sells mixed riser cards), then your built in gigabit ethernet
> (probably Broadcom) is already linked through PCI-Express.  If that's the
> case, don't add a PCI ethernet card and use that.  If your system has
> PCI-Express slots, you can add on a PCI-Express Intel card, but again
> probably not needed until you're pushing the performance levels or need more
> ports.
> 
> ----- Original Message ----- 
> From: "Michael J McCafferty" <mike at m5computersecurity.com>
> To: <sdbug at sdbug.org>
> Sent: Thursday, August 16, 2007 12:55 AM
> Subject: Re: [SDBUG] OpenBSD on Dell issues ?
> 
> 
> Can,
>     Yes, I know that the 32-bit PCI slot will be the bottleneck before
> the 1.4G extrapolated max. It wouldn't be much fun to have the
> firewalls at 100% at any time anyway. I'd be looking for an upgrade
> long before the daily peak CPU was even 70%. Daily we are at 40% now,
> and I am feverishly working to make the upgrade.
>     ...and this is combined bandwidth on all interfaces, full-duplex.
> Currently the networks plugged in to these firewalls are maxed at
> 600Mbps for now... and we aren't using near that yet. When we do get
> get closer to that, we will start upgrading the Internet connections
> to GigE.
>     The problem is that even a moderate DoS causes too many
> interrupts. I mentioned polling because ultiamtely there has to be
> some way to mitigate the interrupts. Can't go 10G if the interrupts
> stop you at 1.4G
>     Now, the other night I decided to try FreeBSD 6.2-RELEASE in place
> of the OpenBSD snapshot box. It reported that it was about HALF the
> CPU interrupt time as the OpenBSD boxes. Is this because the kernel is
> letting both cores handle interrupts ? Or is this an error in the way
> that the kernel is reporting interrupt time at the CPU ? I wasn't able
> to generate enough pps or mbps with my 4 traffic generating boxes
> (realtek NICs) to get it over 50% so I can't tell if it's reporting
> correctly or if it's actually doing it.
>     Since FreeBSD has PF, CARP and pfsync, that might be an option. No ?
> 
>     Thanks, your comments are appreciated.
> 
> Mike
> 
> Quoting Can Erkin Acar <canacar at gmail.com>:
> 
> > On 8/13/07, Michael J McCafferty <mike at m5computersecurity.com> wrote:
> >> Well... I got the hardware, and the results are in:
> >
> > Nice, and good results. There is a problem with your extrapolation
> > calculations though. The peak theoretical bandwidth of PCI (32 bit) is
> > about 128MB/sec which is equivalent to 1Gb/s, so you will not be able
> > to reach 1.4Gb using 32 bit slots. It is probably ok though, since the
> > interrupts are the real problem that limit the pps rate, usually at a
> > much lower bandwidth with small packets. And this is what matters with
> > DoS attacks.
> >
> > For a good overview of PCI technologies you can have a look at:
> > http://arstechnica.com/articles/paedia/hardware/pcie.ars/1
> >
> >
> >> I have a more complete spreadsheet, but here are the highlights:
> >>
> > [snip]
> >>         I think it is relevant to note that interrupts were handled  by a 
> >> single
> >> core of the dual core CPUs in each case.
> >
> > Yes, only userland benefits from multiple processors at the moment.
> >
> >>         What I have learned from this is that when it comes to PCIe or 
> >> PCI-X
> >> and 133MHz 64-bit vs. 32-bit PCI... it's not really that relevant for
> >> x86/x86_64/amd64 firewalls with OpenBSD. The bottleneck is hardware
> >> interrupts to the CPU. I actually already knew that the interrupts were
> >> the bottleneck, but what I didn't know was how 1333MHz FSB vs. 1066MHz
> >> FSB, or 32-bit PCI vs. PCIe, or OpenBSD 3.8, OpenBSD 4.1 or the latest
> >> snapshot would affect the final performance. Of course I also didn't
> >> know the difference between Xeon 5130 and Core2 Duo E6600 for interrupts
> >> either.
> >>         There is one faster model of CPU available for the Dell system. 
> >> The
> >> 3GHz Woodcrest CPU is about $500 MORE than the CPU tested, each. *If* it
> >> scales linearly, this would bring the performance to ~2Gbps and
> >> 400,000pps. This is still 100,000pps short of my desire to handle over
> >> 500,000pps (about 48Mbps of 12Byte UDP packets, a relatively small DDoS
> >> attack). Really, I'd like to be able to handle 100Mbps+ of 12Byte
> >> packets, but I realize a million pps is pretty steep.
> >>
> >>         Why oh why won't someone smarter than me work on device polling 
> >> for
> >> OpenBSD ? Or is that even the solution ?
> >
> > I dont claim to be an expert, but claudio@ says polling is not the
> > solution (it may actually increase latency). A good network card and a
> > good matching driver would probably perform much better.
> >
> > Some developers are working on improving the network performance.
> > There are also several 10Gb card drivers being worked on, so we really
> > aim to support higher speeds. The latest profiling and tuning effort
> > at the last hackathon was a start.
> >
> >
> > Enjoy your new setup :)
> >
> > Can
> > _______________________________________________
> > SDBUG mailing list
> > SDBUG at sdbug.org
> > http://lists.sdbug.org/mailman/listinfo/sdbug
> >
> 
> 
> 
-- 
************************************************************
Michael J. McCafferty
Principal, Security Engineer
M5 Hosting
http://www.m5hosting.com

You can have your own custom Dedicated Server up and running today !
RedHat Enterprise, CentOS, Fedora, Debian, OpenBSD, FreeBSD, and more
************************************************************