[SDBUG] OpenBSD on Dell issues ?
Michael J McCafferty
mike at m5computersecurity.com
Thu Aug 16 08:57:11 PDT 2007
FreeBSD on M5Hosting 6600 w/ "em" driver
TCP @18.0% CPU
91.2Mbps
91.1Mbps
88.7Mbps
88.7Mbps
Total 359.7Mbps /18% = 1.998Gbps
UDP @40% CPU
236/3274819 (0.0072% loss) = 109,161pps
172/3060692 (0.0056% loss) = 102,023pps
Total = 211,184pps / 40% = 527,960pps
For ease of comparison, here are the OpenBSD numbers on the same
hardware:
TCP @25.0% CPU
91.7Mbps
91.4Mbps
91.1Mbps
91.2Mbps
Total 365.4Mbps /25% = 1.462Gbps
UDP @78% CPU
1824/2962283 (0.062% loss) = 98,742pps
221/4012309 (0.0055% loss) = 133,744pps
Total = 232,486pps / 78% = 298,059pps
On Thu, 2007-08-16 at 08:17 -0700, Jeremiah Gowdy wrote:
> While OpenBSD has made progress on their SMP support, FreeBSD's work over
> the past few years in the development and stabilization of the 5.0 and 6.0
> branches was primarily about highly scalable SMP support. If you're working
> with a dual core system, it wouldn't suprise me that FreeBSD is better able
> to schedule interrupts to service your network traffic.
>
> I'm joining the conversation late and I don't see anything in the quotes for
> this email regarding which ethernet chipset you're talking about, but what
> you're looking for in this situation is interrupt moderation. Polling
> interrupts is not the solution. If you have an intel chipset based driver
> "em", that driver does support interrupt moderation. This will allow you to
> reach much higher speeds by not overloading your CPU with interrupts. The
> key here is to not have one interrupt per packet, which is what happens
> without interrupt moderation. Since the driver will see *all* the packets
> the network card has received when an interrupt is serviced, it makes no
> sense to have one interrupt per packet. So on cards that support interrupt
> moderation, depending on settings and the overall traffic level, 3,000-4,000
> packets can be queued for each interrupt that fires. This will cause a huge
> drop in CPU load.
>
> This is probably also related to the performance difference you see in
> FreeBSD, because I am guessing the em driver defaults interrupt moderation
> to "on". Even if your ethernet is not an Intel chipset (Broadcoms are
> really common on my Dells right now), most of the mainstream drivers should
> support interrupt moderation. For more information on interrupt moderation
> see this link:
>
> http://download.intel.com/design/network/applnots/ap450.pdf
>
> I would say that using FreeBSD for your firewall is a perfectly good
> solution. If I was building a firewall personally and I was worried about
> performance, I would probably buy a dual or quad port Intel gigabit
> PCI-Express ethernet card, although it might not be necessary. If your
> system is PCI-Express based (which it can be even if you have PCI / PCI-X
> ports, Dell sells mixed riser cards), then your built in gigabit ethernet
> (probably Broadcom) is already linked through PCI-Express. If that's the
> case, don't add a PCI ethernet card and use that. If your system has
> PCI-Express slots, you can add on a PCI-Express Intel card, but again
> probably not needed until you're pushing the performance levels or need more
> ports.
>
> ----- Original Message -----
> From: "Michael J McCafferty" <mike at m5computersecurity.com>
> To: <sdbug at sdbug.org>
> Sent: Thursday, August 16, 2007 12:55 AM
> Subject: Re: [SDBUG] OpenBSD on Dell issues ?
>
>
> Can,
> Yes, I know that the 32-bit PCI slot will be the bottleneck before
> the 1.4G extrapolated max. It wouldn't be much fun to have the
> firewalls at 100% at any time anyway. I'd be looking for an upgrade
> long before the daily peak CPU was even 70%. Daily we are at 40% now,
> and I am feverishly working to make the upgrade.
> ...and this is combined bandwidth on all interfaces, full-duplex.
> Currently the networks plugged in to these firewalls are maxed at
> 600Mbps for now... and we aren't using near that yet. When we do get
> get closer to that, we will start upgrading the Internet connections
> to GigE.
> The problem is that even a moderate DoS causes too many
> interrupts. I mentioned polling because ultiamtely there has to be
> some way to mitigate the interrupts. Can't go 10G if the interrupts
> stop you at 1.4G
> Now, the other night I decided to try FreeBSD 6.2-RELEASE in place
> of the OpenBSD snapshot box. It reported that it was about HALF the
> CPU interrupt time as the OpenBSD boxes. Is this because the kernel is
> letting both cores handle interrupts ? Or is this an error in the way
> that the kernel is reporting interrupt time at the CPU ? I wasn't able
> to generate enough pps or mbps with my 4 traffic generating boxes
> (realtek NICs) to get it over 50% so I can't tell if it's reporting
> correctly or if it's actually doing it.
> Since FreeBSD has PF, CARP and pfsync, that might be an option. No ?
>
> Thanks, your comments are appreciated.
>
> Mike
>
> Quoting Can Erkin Acar <canacar at gmail.com>:
>
> > On 8/13/07, Michael J McCafferty <mike at m5computersecurity.com> wrote:
> >> Well... I got the hardware, and the results are in:
> >
> > Nice, and good results. There is a problem with your extrapolation
> > calculations though. The peak theoretical bandwidth of PCI (32 bit) is
> > about 128MB/sec which is equivalent to 1Gb/s, so you will not be able
> > to reach 1.4Gb using 32 bit slots. It is probably ok though, since the
> > interrupts are the real problem that limit the pps rate, usually at a
> > much lower bandwidth with small packets. And this is what matters with
> > DoS attacks.
> >
> > For a good overview of PCI technologies you can have a look at:
> > http://arstechnica.com/articles/paedia/hardware/pcie.ars/1
> >
> >
> >> I have a more complete spreadsheet, but here are the highlights:
> >>
> > [snip]
> >> I think it is relevant to note that interrupts were handled by a
> >> single
> >> core of the dual core CPUs in each case.
> >
> > Yes, only userland benefits from multiple processors at the moment.
> >
> >> What I have learned from this is that when it comes to PCIe or
> >> PCI-X
> >> and 133MHz 64-bit vs. 32-bit PCI... it's not really that relevant for
> >> x86/x86_64/amd64 firewalls with OpenBSD. The bottleneck is hardware
> >> interrupts to the CPU. I actually already knew that the interrupts were
> >> the bottleneck, but what I didn't know was how 1333MHz FSB vs. 1066MHz
> >> FSB, or 32-bit PCI vs. PCIe, or OpenBSD 3.8, OpenBSD 4.1 or the latest
> >> snapshot would affect the final performance. Of course I also didn't
> >> know the difference between Xeon 5130 and Core2 Duo E6600 for interrupts
> >> either.
> >> There is one faster model of CPU available for the Dell system.
> >> The
> >> 3GHz Woodcrest CPU is about $500 MORE than the CPU tested, each. *If* it
> >> scales linearly, this would bring the performance to ~2Gbps and
> >> 400,000pps. This is still 100,000pps short of my desire to handle over
> >> 500,000pps (about 48Mbps of 12Byte UDP packets, a relatively small DDoS
> >> attack). Really, I'd like to be able to handle 100Mbps+ of 12Byte
> >> packets, but I realize a million pps is pretty steep.
> >>
> >> Why oh why won't someone smarter than me work on device polling
> >> for
> >> OpenBSD ? Or is that even the solution ?
> >
> > I dont claim to be an expert, but claudio@ says polling is not the
> > solution (it may actually increase latency). A good network card and a
> > good matching driver would probably perform much better.
> >
> > Some developers are working on improving the network performance.
> > There are also several 10Gb card drivers being worked on, so we really
> > aim to support higher speeds. The latest profiling and tuning effort
> > at the last hackathon was a start.
> >
> >
> > Enjoy your new setup :)
> >
> > Can
> > _______________________________________________
> > SDBUG mailing list
> > SDBUG at sdbug.org
> > http://lists.sdbug.org/mailman/listinfo/sdbug
> >
>
>
>
--
************************************************************
Michael J. McCafferty
Principal, Security Engineer
M5 Hosting
http://www.m5hosting.com
You can have your own custom Dedicated Server up and running today !
RedHat Enterprise, CentOS, Fedora, Debian, OpenBSD, FreeBSD, and more
************************************************************
More information about the SDBUG
mailing list