[SDBUG] OpenBSD on Dell issues ?
Can Erkin Acar
canacar at gmail.com
Thu Aug 16 13:39:51 PDT 2007
Well, I am not really sure how each OS computes/reports the Interrupt
time. FreeBSD definitely handles the kernel locking in a much more
granular way, and multiple processors are utilized more efficiently in
the kernel. On the other hand, I dont think the pps rate will scale
linearly with CPU utilization.
The em(4) driver is written by Intel, and both FreeBSD and OpenBSD use
the same driver. Comparing default OpenBSD and FreeBSD driver
parameters show there are some differences, although the
MAXIMUM_INTS_PER_SEC is 8000 for both, which how the interrupt
limiting is done.
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/dev/em/if_em.h?rev=1.61;content-type=text%2Fx-cvsweb-markup
http://www.openbsd.org/cgi-bin/cvsweb/src/sys/dev/pci/if_em.h?rev=1.35&content-type=text/x-cvsweb-markup
A friend of mine, trying to deal with DoS attacks had reported some
success with increasing EM_RDTR, which is accompanied by a cautionary
note in the driver source:
* CAUTION: When setting EM_RDTR to a value other than 0, adapters
* may hang (stop transmitting) under certain network conditions.
* If this occurs a WATCHDOG message is logged in the system
* event log. In addition, the controller is automatically reset,
* restoring the network connection. To eliminate the potential
* for the hang ensure that EM_RDTR is set to 0.
Note that once you start playing with these knobs, you are on your
own. Both systems set safe limits, suitable for a wide range of
supported cards and platforms. You have to know what you are doing,
and I believe you dont need to go there yet.
In the end, it all comes down to which system works for you the best,
and where you feel most comfortable.
Can
On 8/16/07, Michael J McCafferty <mike at m5computersecurity.com> wrote:
>
>
> FreeBSD on M5Hosting 6600 w/ "em" driver
>
> TCP @18.0% CPU
> 91.2Mbps
> 91.1Mbps
> 88.7Mbps
> 88.7Mbps
> Total 359.7Mbps /18% = 1.998Gbps
>
> UDP @40% CPU
> 236/3274819 (0.0072% loss) = 109,161pps
> 172/3060692 (0.0056% loss) = 102,023pps
> Total = 211,184pps / 40% = 527,960pps
>
>
> For ease of comparison, here are the OpenBSD numbers on the same
> hardware:
>
> TCP @25.0% CPU
> 91.7Mbps
> 91.4Mbps
> 91.1Mbps
> 91.2Mbps
> Total 365.4Mbps /25% = 1.462Gbps
>
> UDP @78% CPU
> 1824/2962283 (0.062% loss) = 98,742pps
> 221/4012309 (0.0055% loss) = 133,744pps
> Total = 232,486pps / 78% = 298,059pps
>
>
>
>
> On Thu, 2007-08-16 at 08:17 -0700, Jeremiah Gowdy wrote:
> > While OpenBSD has made progress on their SMP support, FreeBSD's work over
> > the past few years in the development and stabilization of the 5.0 and 6.0
> > branches was primarily about highly scalable SMP support. If you're working
> > with a dual core system, it wouldn't suprise me that FreeBSD is better able
> > to schedule interrupts to service your network traffic.
> >
> > I'm joining the conversation late and I don't see anything in the quotes for
> > this email regarding which ethernet chipset you're talking about, but what
> > you're looking for in this situation is interrupt moderation. Polling
> > interrupts is not the solution. If you have an intel chipset based driver
> > "em", that driver does support interrupt moderation. This will allow you to
> > reach much higher speeds by not overloading your CPU with interrupts. The
> > key here is to not have one interrupt per packet, which is what happens
> > without interrupt moderation. Since the driver will see *all* the packets
> > the network card has received when an interrupt is serviced, it makes no
> > sense to have one interrupt per packet. So on cards that support interrupt
> > moderation, depending on settings and the overall traffic level, 3,000-4,000
> > packets can be queued for each interrupt that fires. This will cause a huge
> > drop in CPU load.
> >
> > This is probably also related to the performance difference you see in
> > FreeBSD, because I am guessing the em driver defaults interrupt moderation
> > to "on". Even if your ethernet is not an Intel chipset (Broadcoms are
> > really common on my Dells right now), most of the mainstream drivers should
> > support interrupt moderation. For more information on interrupt moderation
> > see this link:
> >
> > http://download.intel.com/design/network/applnots/ap450.pdf
> >
> > I would say that using FreeBSD for your firewall is a perfectly good
> > solution. If I was building a firewall personally and I was worried about
> > performance, I would probably buy a dual or quad port Intel gigabit
> > PCI-Express ethernet card, although it might not be necessary. If your
> > system is PCI-Express based (which it can be even if you have PCI / PCI-X
> > ports, Dell sells mixed riser cards), then your built in gigabit ethernet
> > (probably Broadcom) is already linked through PCI-Express. If that's the
> > case, don't add a PCI ethernet card and use that. If your system has
> > PCI-Express slots, you can add on a PCI-Express Intel card, but again
> > probably not needed until you're pushing the performance levels or need more
> > ports.
> >
> > ----- Original Message -----
> > From: "Michael J McCafferty" <mike at m5computersecurity.com>
> > To: <sdbug at sdbug.org>
> > Sent: Thursday, August 16, 2007 12:55 AM
> > Subject: Re: [SDBUG] OpenBSD on Dell issues ?
> >
> >
> > Can,
> > Yes, I know that the 32-bit PCI slot will be the bottleneck before
> > the 1.4G extrapolated max. It wouldn't be much fun to have the
> > firewalls at 100% at any time anyway. I'd be looking for an upgrade
> > long before the daily peak CPU was even 70%. Daily we are at 40% now,
> > and I am feverishly working to make the upgrade.
> > ...and this is combined bandwidth on all interfaces, full-duplex.
> > Currently the networks plugged in to these firewalls are maxed at
> > 600Mbps for now... and we aren't using near that yet. When we do get
> > get closer to that, we will start upgrading the Internet connections
> > to GigE.
> > The problem is that even a moderate DoS causes too many
> > interrupts. I mentioned polling because ultiamtely there has to be
> > some way to mitigate the interrupts. Can't go 10G if the interrupts
> > stop you at 1.4G
> > Now, the other night I decided to try FreeBSD 6.2-RELEASE in place
> > of the OpenBSD snapshot box. It reported that it was about HALF the
> > CPU interrupt time as the OpenBSD boxes. Is this because the kernel is
> > letting both cores handle interrupts ? Or is this an error in the way
> > that the kernel is reporting interrupt time at the CPU ? I wasn't able
> > to generate enough pps or mbps with my 4 traffic generating boxes
> > (realtek NICs) to get it over 50% so I can't tell if it's reporting
> > correctly or if it's actually doing it.
> > Since FreeBSD has PF, CARP and pfsync, that might be an option. No ?
> >
> > Thanks, your comments are appreciated.
> >
> > Mike
> >
> > Quoting Can Erkin Acar <canacar at gmail.com>:
> >
> > > On 8/13/07, Michael J McCafferty <mike at m5computersecurity.com> wrote:
> > >> Well... I got the hardware, and the results are in:
> > >
> > > Nice, and good results. There is a problem with your extrapolation
> > > calculations though. The peak theoretical bandwidth of PCI (32 bit) is
> > > about 128MB/sec which is equivalent to 1Gb/s, so you will not be able
> > > to reach 1.4Gb using 32 bit slots. It is probably ok though, since the
> > > interrupts are the real problem that limit the pps rate, usually at a
> > > much lower bandwidth with small packets. And this is what matters with
> > > DoS attacks.
> > >
> > > For a good overview of PCI technologies you can have a look at:
> > > http://arstechnica.com/articles/paedia/hardware/pcie.ars/1
> > >
> > >
> > >> I have a more complete spreadsheet, but here are the highlights:
> > >>
> > > [snip]
> > >> I think it is relevant to note that interrupts were handled by a
> > >> single
> > >> core of the dual core CPUs in each case.
> > >
> > > Yes, only userland benefits from multiple processors at the moment.
> > >
> > >> What I have learned from this is that when it comes to PCIe or
> > >> PCI-X
> > >> and 133MHz 64-bit vs. 32-bit PCI... it's not really that relevant for
> > >> x86/x86_64/amd64 firewalls with OpenBSD. The bottleneck is hardware
> > >> interrupts to the CPU. I actually already knew that the interrupts were
> > >> the bottleneck, but what I didn't know was how 1333MHz FSB vs. 1066MHz
> > >> FSB, or 32-bit PCI vs. PCIe, or OpenBSD 3.8, OpenBSD 4.1 or the latest
> > >> snapshot would affect the final performance. Of course I also didn't
> > >> know the difference between Xeon 5130 and Core2 Duo E6600 for interrupts
> > >> either.
> > >> There is one faster model of CPU available for the Dell system.
> > >> The
> > >> 3GHz Woodcrest CPU is about $500 MORE than the CPU tested, each. *If* it
> > >> scales linearly, this would bring the performance to ~2Gbps and
> > >> 400,000pps. This is still 100,000pps short of my desire to handle over
> > >> 500,000pps (about 48Mbps of 12Byte UDP packets, a relatively small DDoS
> > >> attack). Really, I'd like to be able to handle 100Mbps+ of 12Byte
> > >> packets, but I realize a million pps is pretty steep.
> > >>
> > >> Why oh why won't someone smarter than me work on device polling
> > >> for
> > >> OpenBSD ? Or is that even the solution ?
> > >
> > > I dont claim to be an expert, but claudio@ says polling is not the
> > > solution (it may actually increase latency). A good network card and a
> > > good matching driver would probably perform much better.
> > >
> > > Some developers are working on improving the network performance.
> > > There are also several 10Gb card drivers being worked on, so we really
> > > aim to support higher speeds. The latest profiling and tuning effort
> > > at the last hackathon was a start.
> > >
> > >
> > > Enjoy your new setup :)
> > >
> > > Can
> > > _______________________________________________
> > > SDBUG mailing list
> > > SDBUG at sdbug.org
> > > http://lists.sdbug.org/mailman/listinfo/sdbug
> > >
> >
> >
> >
> --
> ************************************************************
> Michael J. McCafferty
> Principal, Security Engineer
> M5 Hosting
> http://www.m5hosting.com
>
> You can have your own custom Dedicated Server up and running today !
> RedHat Enterprise, CentOS, Fedora, Debian, OpenBSD, FreeBSD, and more
> ************************************************************
>
> _______________________________________________
> SDBUG mailing list
> SDBUG at sdbug.org
> http://lists.sdbug.org/mailman/listinfo/sdbug
>
More information about the SDBUG
mailing list