[SDBUG] OpenBSD on Dell issues ?

Michael J McCafferty mike at m5computersecurity.com
Mon Aug 13 17:07:28 PDT 2007 

Well... I got the hardware, and the results are in:


I have a more complete spreadsheet, but here are the highlights:

Hardware: 
	Single Dell 1950 w/ 1x Xeon 5130, 2G 667MHz RAM, 2x Intel dual-port
GigE PCIe NICs
Cost:
	About $2250 ea.
Results:
	TCP 4 streams totaling 362Mbps @ 23% peak CPU (interrupts) with
83.5KByte TCP Window Size
	UDP 239,935pps @ 0.0032% loss @ 80% peak CPU (interrupts)
Extrapolation
	1.577Gbps Max TCP
	299,919pps Max UDP


Hardware: 
	Single M5Hosting Core2 Duo E6600 Server (1GB DDR2-533, 1x Intel
dual-port GigE PCI-X NIC in 32-bit PCI slot
Cost:
	About $834 ea.
Results:
	TCP 4 streams totaling 365Mbps @ 25% peak CPU (interrupts) with
83.5KByte TCP Window Size
	UDP 232,486pps @ 0.0338% loss @ 78% peak CPU (interrupts)
Extrapolation:
	1.462Gbps Max TCP
	298,059pps Max UDP


	To perform the tests, I used 4 M5Hosting E6600 servers to generate and
receive packets using "iPerf" open source software. 2 were set "inside"
the firewall, and two were "outside" the firewall. The base OS used for
these systems was Ubuntu 7.04 64-bit. Both firewalls were installed with
the latest OpenBSD snapshot at the time of the test (between 4.1 and
4.2, and after significant PF performance improvements). The firewalls
had PF enables but no rules. Essentially the performance measured was
that of a OpenBSD router. In actual use, the PF firewall has never
presented much load on the systems I have used, in comparison to the
interrupts from the network interface hardware.
	The traffic generator systems have 100mbps interfaces, and were
connected to 100mbps interfaces on a Dell PowerConnect 3448 switch. The
firewalls each had 1Gbps interfaces, and were connected to the Gig
interfaces of the same Dell PowerConnect 3448 switch. The "outside"
interfaces were configured to be vlan 1, and the inside interfaces were
configured to be vlan 2.
	I think it is relevant to note that interrupts were handled by a single
core of the dual core CPUs in each case.

	What I have learned from this is that when it comes to PCIe or PCI-X
and 133MHz 64-bit vs. 32-bit PCI... it's not really that relevant for
x86/x86_64/amd64 firewalls with OpenBSD. The bottleneck is hardware
interrupts to the CPU. I actually already knew that the interrupts were
the bottleneck, but what I didn't know was how 1333MHz FSB vs. 1066MHz
FSB, or 32-bit PCI vs. PCIe, or OpenBSD 3.8, OpenBSD 4.1 or the latest
snapshot would affect the final performance. Of course I also didn't
know the difference between Xeon 5130 and Core2 Duo E6600 for interrupts
either.
	There is one faster model of CPU available for the Dell system. The
3GHz Woodcrest CPU is about $500 MORE than the CPU tested, each. *If* it
scales linearly, this would bring the performance to ~2Gbps and
400,000pps. This is still 100,000pps short of my desire to handle over
500,000pps (about 48Mbps of 12Byte UDP packets, a relatively small DDoS
attack). Really, I'd like to be able to handle 100Mbps+ of 12Byte
packets, but I realize a million pps is pretty steep. 

	Why oh why won't someone smarter than me work on device polling for
OpenBSD ? Or is that even the solution ?

	The Dell's are being returned this week and the M5Hosting servers will
be placed in to service within a week.

	Thanks all for the lively conversation about this in the past... and
perhaps the future. Your thoughts, comments and suggestions are
welcome. 

Thanks,
Mike

PS: OpenBSD 4.1 and the 4.2 snapshots worked flawlessly with all
hardware on the Dells, including the RAID card and the Broadcom GigE
on-board NICs.

On Fri, 2007-08-03 at 15:31 -0700, Michael J McCafferty wrote:
> Can,
> 	Based on the feedback you gave at/after the meeting last night and
> Ben's feedback, I went ahead and bought the Dells.
> 
> PowerEdge 1950
> Xeon 5130 (Woodcrest dual core @2GHz 4M cache 1333MHz FSB)
> 2 x 1GB 667MHz RAM
> 2 x integrated Broadcom NetXtreme GigE
> 2 x Dual port Intel PRO 1000PT PCIe
> 2 x 80G SATA drives w/ integrated RAID
> 
> 	So, each unit will have 6 GigE interfaces, way more RAM than needed.
> These will run CARP and pfsync.
> 	Originally I was thinking that I would get another pair for the other
> network in the same location, but if these things are as fast as I think
> they will be, I bought more interfaces than I needed so that maybe I'll
> use these as the firewalls for the other network too. This will mean
> that I will expect these to handle 400Mbps (combined in + out) with
> mixed packet sizes to/from 500+ servers, and hope these can handle
> 600Mbps to 800Mbps peaks. I wonder if it will be too much to ask for
> these things to stay alive during a 1 Gigabit DoS attack of 1byte UDP
> packets ? :o)
> 	If I need to get faster cores, there are 3.0GHz Woodcrest CPUs
> available for these servers.
> 
> I'll have the hardware next week. I'll update the group as I learn
> anything worth reporting.
> 
> Cheers,
> Mike
> 
> 
> On Thu, 2007-08-02 at 18:28 -0700, Can Erkin Acar wrote:
> > On 8/2/07, Michael J McCafferty <mike at m5computersecurity.com> wrote:
> > > All,
> > > 	I am am on the verge of pulling the trigger on a hardware buy for some
> > > new firewalls to run OpenBSD 4.1, PF, CARP, pfsync. As we have discussed
> > > at the SDBUG meetings in the past, I have issues with interrupts on the
> > > CPU when the packets per second get high.
> > > 	I am replacing the current hardware within 2 weeks. The current system
> > > was installed when it was expected to handle 10 to 20 Mbps peak and
> > > about 2 to 5Mbps average. I now need a setup that can handle 20 to
> > > 50Mbps average now, with peaks to 200Mbps and future growth to several
> > > hundred Mbps peak at which time I assume that unless there is some major
> > > advance in servers/PCs/x86_64 architecture, I will have to go to ASIC
> > > based devices (ie: Netscreens, etc) and not be able to use my beloved
> > > PF. But I digress... what I really need to know is:
> > 
> > I have not used one, but here are some comments,
> > First of all, for network performance, you should try -current,
> > (it is in 4.2-beta now) there are many network performance
> > improvements done after 4.1.
> > 
> > > 	Is anyone currently running OpenBSD on Dell 1950s with the SAS 5/i
> > > SATA/SAS controller ? Any problems ? The Dell dude I spoke to said there
> > > may be issues with FreeBSD and the disk controller.. but that he only
> > > heard that some place and has no details on that. Or shall I use the
> > > PERC 5/i controller instead ?
> > 
> > There are threads on misc@ that suggest that it should work fine.
> > 
> > see for instance the following thread:
> > http://marc.info/?l=openbsd-misc&m=117551048515741&w=2
> > 
> > PERC 5/i uses the mfi(4) driver and SAS 5/iR uses the mpi(4) driver
> > with the mfi driver you can get raid status information through the bio(4)
> > framework.
> > 
> > > 	Is anyone using Intel PCIe Gig NICs ? I have been using the dual-port
> > > Intel GigE NICs for PCI/PCI-X, but not PCIe. Any known issues ?
> > > 	The system comes with Dual Embedded Broadcom NetXtreme II 5708 Gigabit
> > > interfaces. Will these be usable under OpenBSD 4.1 ?
> > 
> > These are bnx(4) they are usable as far as I know.
> > 
> > Can
> > 
> > > 	A beer for useful input ! Say, isn't there a meeting tonight ?
> > >
> > > Thank you !
> > > Mike
> > >
> > >
> > > --
> > > ************************************************************
> > > Michael J. McCafferty
> > > Principal, Security Engineer
> > > M5 Hosting
> > > http://www.m5hosting.com
> > >
> > > You can have your own custom Dedicated Server up and running today !
> > > RedHat Enterprise, CentOS, Fedora, Debian, OpenBSD, FreeBSD, and more
> > > ************************************************************
> > >
> > > _______________________________________________
> > > SDBUG mailing list
> > > SDBUG at sdbug.org
> > > http://lists.sdbug.org/mailman/listinfo/sdbug
> > >
> > _______________________________________________
> > SDBUG mailing list
> > SDBUG at sdbug.org
> > http://lists.sdbug.org/mailman/listinfo/sdbug
-- 
************************************************************
Michael J. McCafferty
Principal, Security Engineer
M5 Hosting
http://www.m5hosting.com

You can have your own custom Dedicated Server up and running today !
RedHat Enterprise, CentOS, Fedora, Debian, OpenBSD, FreeBSD, and more
************************************************************

More information about the SDBUG mailing list