[SDBUG] Anyone have a rackmount server laying around?
Michael J McCafferty
mike at m5computersecurity.com
Mon Mar 12 09:29:32 PDT 2007
OK, so right this moment:
~20Mbit/sec, about 8,000 pps according to pf (but 16,000 pps
according to my IDS/sniffer box)
Interrupts:
12629 total
6658 em0
5734 em1
9 fxp0
pciide0
100 clock
128 rtc
CPU states: 0.2% user, 0.0% nice, 0.2% system, 32.1% interrupt, 67.6% idle
Memory: Real: 9036K/71M act/tot Free: 426M Swap: 0K/1024M used/tot
Note: The "em" interfaces are the inside and outside GigE interfaces.
The fxp interface is the 100Mbit interface for management.
At 08:32 AM 3/12/2007, you wrote:
>Hey Mike,
>
>The Intel is indeed fxp. But it is not to be concluded directly with the
>card, but both the card and motherboard combination. Having a 10% Hardware
>Interrupt load is quite intensive high. It is hard to give a specific
>recommendation for a couple of reasons:
>1. A lot of the motherboards have lifecycles that are < 12 months, and as
>such are no longer available. Often times, even if the model number stays
>the same, the reference board changes slightly and causes a (sometimes
>severe) impact to performance.
>2. Something that worked for me may not work great for you! =(
>
>However, the combination on the HP motherboards (I have been using the Intel
>Xeon Proliants) has proved an excellent choice with the dual, built on
>(intel based) gige ethernet ports.
>
>Also, although I have not tested it directly I suspect that bridging
>performance would increase on a dual-port card as opposed to two single port
>cards. In my other experience I have seen that two cards (even of the same
>type) can cause a kernel driver to go a bit wacky and see the performance
>crash due to (sometimes 10x) increase in interrupts, even if there is no
>load increase. However this may be due to the poor driver that I am using
>and have no performance impact on what you are doing.
>
>Your best bet is to test this extensively in a testing environment if
>resources allow it. But then again, with the resources you end up spending
>during testing, it might just be easier to buy a PIX? =)
>
>Cheers!
>Mike
>
>On 3/12/07, Michael J McCafferty <mike at m5computersecurity.com> wrote:
>>
>>Mike,
>> Thanks for the excellent feedback !
>> Isn't the Ethernet Express driver the fxp driver ? I am
>>using the "em" driver, for the Gigabit cards (specifically "Intel
>>PRO/1000MT Dual Port PWLA8492MT")
>> I chose the Intel cards because I understood they were
>>excellent cards for what I am doing. I was led to believe the cards
>>had excellent caching and interrupt mitigation. What is a better card ?
>>
>>Mike
>>
>>At 10:35 PM 3/11/2007, you wrote:
>> >Mike,
>> >
>> >Using, Testing, and Building Linux systems for VoIP systems, I can tell
>>you
>> >with a high level of confidence that your hardware interrupt rates can be
>> >significantly reduced by a combination of the two following things:
>> >
>> >1) New ethernet controller (specifically, switching the driver used)
>> >2) New Northbridge / PCI controller (Motherboard)
>> >
>> >In my experience, Intel based ethernet controllers perform poorly with
>>very
>> >high packet rates (10,000+ pps) due to cache optimization for very large
>> >packets, at a low rate. The 20ms interval UDP packets just don't jive
>>with
>> >the Ethernet Express line, and I wouldn't reccomend them for something
>>like
>> >a DNS or VoIP box, but might work great under FTP / HTTP / SMTP load.
>> >
>> >It takes about 20-50 motherboard combinations to find one that works well
>>at
>> >high load, and it seems to be a somewhat random mix. This is at the low
>>end,
>> >your typical sub $150 motherboards.
>> >
>> >At the high end I have found HP to have an extraordinarily high success
>> >rate, with any of the Proliant series being a great option.
>> >
>> >As far as switching to an SMP kernel, all of the load for this kind of
>> >application is due to the interrupt processing, changing the kernel from
>> >XT-PIC to APIC helps tremendously in Linux, I assume there are similar
>> >compile options for *BSD for choice of interrupt controllers.
>> >
>> >Lastly, I found that using Celerons has a dramatic performance impact on
>> >VoIP (very small, and frequent packet) performance. When upgrading to a
>>P4 I
>> >can see nearly a 4x performance increase (sometimes more), and this is
>>with
>> >the same clockspeed and chipset! This is likely attributable to the cache
>> >size in the P4 allowing for faster flow of the packets through the L2
>>cache.
>> >I would consider the very inexpensive upgrade to a processor with a
>>larger
>> >cache.
>> >
>> >Hope all goes well with your firewall. Please let me (and the list if you
>> >are willing) know how it goes.
>> >
>> >Cheers!
>> >Mike
>> >
>> >
>> >On 3/11/07, Michael J McCafferty <mike at m5computersecurity.com> wrote:
>> >>
>> >>Ron,
>> >> Yes. I am very interested. See, I have OpenBSD with PF in
>> >>production now as a transparent firewall, and looking to go to
>> >>OpenBSD 4.0 with CARP and pfsync as a redundant pair of routing
>> >>firewalls. I am concerned because
>> >>http://www.tancsa.com/blast.html shows that OpenBSD can't route as
>> >>many pps as FreeBSD. I'd like to stay with OpenBSD if I can, but with
>> >>our rate of growth, and with trying to guesstimate according the
>> >>performance numbers on that page... I don't know.
>> >> Day in and day out, interrupts are currently using between
>> >>10% and 20% CPU on Celeron 2.8GHz on Intel 865 chipset, using a dual
>> >>port Intel GigE server card. User and system time are nothing. My
>> >>concern is that anomalous traffic will cause problems, as well as our
>> >>growth rate indicates we will have more than double our current
>> >>bandwidth with six months. I can throw more hardware at the problem,
>> >>but it doesn't look like SMP is of any use. A P4 3.2GHz on Intel 945
>> >>Chipset, same dual-port Intel GigE server network card is the planned
>> >>hardware.
>> >>
>> >>Mike
>> >>
>> >>
>> >>At 05:42 PM 3/11/2007, you wrote:
>> >> >Well it s not on a soekris if that is what you are asking. The one
>> >> >piece I left out was the CPU. It is a Hyper-threaded 3.0G P4. The
>> >> >bandwidth it is handling at the moment is a DSL link but I would
>> >> >definitely a production based installation. This is not my server. I
>> >> >am running a practically a barebones version of pfsense on my
>> >> >Soekris4801-60 with a hard drive.
>> >> >
>> >> >Have not ran an iperf test on it. If you are interested I will see
>> >> >what I can do between 2 interfaces.
>> >> >
>> >> >-Ron
>> >> >
>> >> >P.S. The IDS is only watching the WAN link.
>> >> >
>> >> >On Mar 11, 2007, at 4:48 PM, Michael J McCafferty wrote:
>> >> >
>> >> >>Ron,
>> >> >> How many packets/sec or megabits/sec are you sending
>> >> >>through this thing ? Is this just a home firewall or is it handling
>> >> >>some production traffic ?
>> >> >>Thanks,
>> >> >>Mike
>> >> >>
>> >> >>
>> >> >>At 02:21 PM 3/11/2007, you wrote:
>> >> >>>Have you taken a look at pfsense (http://www.pfsense.com). I have it
>> >> >>>running on a server with two drives running raid 1, 2 Gigs of Ram, 1
>> >> >>>100Megbit interface, and three gigabit interfaces. It is a pretty
>> >> >>>sweet setup Below is a few things I have it doing:
>> >> >>>
>> >> >>>- Captive portal on one of the interfaces (Airport network plugged
>> >> >>>into here)
>> >> >>>- Snort for detecting bad guys
>> >> >>>- OpenVPN for raod warrior connections
>> >> >>>- IPSEC for connecting networks via tunnels
>> >> >>>- Pfflowd sending data to an internal server running nfsen for
>> >> >>>anomaly detection
>> >> >>>- Spamd to assist the internal mail server with fighting SPAM
>> >> >>>- Traffic shaping to keep traffic under control and allow other app
>> >> >>>the bandwidth when needed.
>> >> >>>
>> >> >>>The above are to just name a few. System has been rock solid since I
>> >> >>>installed it.
>> >> >>>
>> >> >>>-Ron
>> >> >>>
>> >> >>>On Mar 11, 2007, at 2:22 PM, Kevin Stevens wrote:
>> >> >>>
>> >> >>>>Group member hooked me up with a nice DL36. Now debating Free-
>> >> >>>>(which I know) vs Open- (which I don't). Probably go with Open-,
>> >> >>>>since it's targeted at stuff like this and a good learning
>> >> >>>>opportunity.
>> >> >>>>
>> >> >>>>Thanks all!
>> >> >>>>
>> >> >>>>KeS
>> >> >>>>
>> >> >>>>On Mar 9, 2007, at 21:28, Kevin Stevens wrote:
>> >> >>>>
>> >> >>>>>I'm tired of waiting for Juniper to provide IPv6 code for my
>> >> >>>>>Netscreen GT, so I'm going to build a firewall/router appliance
>> >> >>>>>(with a separate interface for my wireless). I can get boxes off
>> >> >>>>>of eBay for $100-$150 w/shipping, but if someone has one locally I
>> >> >>>>>can grab this weekend, that would be great.
>> >> >>>>>
>> >> >>>>>Looking for a DL360 type of thing - single or dual 500-1000MHz
>> >> >>>>>CPU, 256-512MB, 9-40GB drives (SCSI or IDE), CDROM. USB and/or
>> >> >>>>>gigabit would be pluses. At least one PCI slot for additional
>> >> >>>>>NIC. Intent is to run FreeBSD.
>> >> >>>>>
>> >> >>>>>Thanks, let me know if you have a good candidate you want to get
>> >> >>>>>rid of!
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>_______________________________________________
>> >> >>>SDBUG mailing list
>> >> >>>SDBUG at sdbug.org
>> >> >>>http://lists.sdbug.org/mailman/listinfo/sdbug
>> >> >>
>> >> >>_______________________________________________
>> >> >>SDBUG mailing list
>> >> >>SDBUG at sdbug.org
>> >> >>http://lists.sdbug.org/mailman/listinfo/sdbug
>> >> >
>> >> >
>> >> >
>> >> >_______________________________________________
>> >> >SDBUG mailing list
>> >> >SDBUG at sdbug.org
>> >> >http://lists.sdbug.org/mailman/listinfo/sdbug
>> >>
>> >>_______________________________________________
>> >>SDBUG mailing list
>> >>SDBUG at sdbug.org
>> >>http://lists.sdbug.org/mailman/listinfo/sdbug
>> >
>> >
>> >
>> >--
>> >Mike Joyce
>> >mjoyce at obstinate.org
>> >_______________________________________________
>> >SDBUG mailing list
>> >SDBUG at sdbug.org
>> >http://lists.sdbug.org/mailman/listinfo/sdbug
>>
>>_______________________________________________
>>SDBUG mailing list
>>SDBUG at sdbug.org
>>http://lists.sdbug.org/mailman/listinfo/sdbug
>
>
>
>--
>Mike Joyce
>mjoyce at obstinate.org
>_______________________________________________
>SDBUG mailing list
>SDBUG at sdbug.org
>http://lists.sdbug.org/mailman/listinfo/sdbug
More information about the SDBUG
mailing list