DIY pfSense UTM

Discussion in 'Networking, Telephony & Internet' started by samus, Mar 14, 2013.

  1. samus

    samus Member

    Joined:
    Jun 3, 2002
    Messages:
    1,234
    Location:
    Baulkham Hills, Sydney.
    DIY pfSense!

    Hello again OCAU!

    Sorry for the delay, life got in the way, and my wife decided to break her foot. On we go!

    As from my other threads, and my post’s around the place, and more than a few PM’s, I’m a bit of a pfSense fan. So I thought I would finally post a review/overview/rambling thoughts on some new hardware I built, and also a more thorough review of the current state of pfSense (as of 08/03/2012 the stable release is 2.0.2). This also will be (unavoidably) a bit of a How-To, only because I don’t know any other way to explain an installation without detailing how it’s done.

    A bit of background first, I work in a small government site (<100 people) with a very tight IT budget. I’m not a networking or security person (except out of necessity), however as most of you know (and many of you do!), IT on small sites get to do a bit of everything. That said I have several open source systems in place now, from my iSCSI server (OpenFiler), PBX (Asterisk+FreePBX), monitoring (Opsview/Nagios), and document distribution (ownCloud). Anything that I can build or do for free (or very little dollars) is appreciated.

    Originally we replaced a Fortigate 100A with a ClearBox 300 and ClearOS (which was an absolute disaster, and not in the scope of this post), and then I replaced the OS on the Clearbox hardware with pfSense, and couldn’t be happier.

    So a few weeks ago we had a major hardware failure on our ClearOS box, so we thought we would replace it, with something a bit more “standard”.


    Click to view full size!


    Meet my new firewall. I call it “Standard Overkill”

    Onto some pictures and explanations of what I’m trying to achieve. (Ignore the silver POS PSU I had, was just to test while i waited for my real PSU’s to arrive)


    Click to view full size!



    Click to view full size!


    Hardware first! As I said before I’m going for a more standard build, with ATX hardware and off the shelf components. This reduces cost, as it’s basically a PC, with very easy repairability. And yes, why build one..


    Click to view full size!


    .. when you can build two for twice the price?

    So the idea of this is to have a HA firewall/UTM solution.

    Cases are PCI Case IPC-RK-238B models, so they take a Micro-ATX motherboard, standard ATX power supplies, and low profile cards, as well as up to 2 3.5 inch drives and 2 5.25 inch optical drives.


    Click to view full size!


    They are solid cases, I love the ATX PSU space, and they are nicely powder-coated. Not tool-less of course, and the way you mount drives is very fiddly. They do have rubber grommets on the base to reduce noise. The 5.25 cage is very painful to remove, you need to undo 4 screws, 2 on the bottom of the case, and 2 inside the case, then you slide the cage out. Other than that, there are a few sharp edges, but I didn’t cut myself during the build. Fans are dead quiet, great for them being in my office. For $200, (no PSU) they are OK value, but not great.

    Now the guts of the machine:


    Click to view full size!



    Click to view full size!


    I needed a M/B that was MicroATX, as well as having an Intel NIC and 3 PCI Express slots. The only one I could find with this G1 Killer M3 from Gigabyte. It has all solid state caps, a Z77 chipset, as well as being bright green. You also get a few sheets of stickers, as well as a poster as part of the package. They went up in the server room, just for a laugh. This is also my first experience with UEFI at a desktop level, (I have many IBM servers with UEFI) and I like Gigabytes implementation. It’s very easy to navigate, though I completely ignored the overclocking section of the BIOS. Only things I set were Always on, and boot from SATA only.


    Click to view full size!


    Processor is an Intel i5-3470S, the (slightly) lower-power version, with a 65W TDP. Superb performance and not that expensive.

    RAM is 8GB of Kingston’s Hyper-X Blu series, only because it had tight timings and was on the approved RAM list. Not much else to say about it.


    Click to view full size!


    Now I’m using SSD for storage, and I’m well aware that Squid and SSD’s don’t mix, however I don’t mind throwing the SSD out after year for the performance gain in cache speed. From what I have read, the Samsung 840 pro’s have the best garbage collection in that price range, and since pfSense's FreeBSD version (8.1 RELEASE) doesn’t support TRIM yet, (it will!) that what I needed. 120GB is more than enough, the pf install is only a few MB, and the cache will be only a few GB. Squid will put everything in RAM if I tell it to anyway. I’m also NOT using mirroring, because software on FreeBSD (gmirror) is useless, the on-board is fake RAID (and just a useless as software raid) and it’s just as easy to have a “cold” spare in the box, and replace as needed. In a HA environment, I can take one offline with no drama.


    Click to view full size!


    Network cards are Intel I350-T2 dual port low-profile PCIe, giving 7 ports in total. I don’t need TCP offload or anything fancy, just interfaces. These also have native FreeBSD support.

    The PSU is a an FSP Silencer 400W unit, 80+ Gold certified as well as being fanless. It scored very well in the JohnnyGuru review, and I won’t be pushing it very hard at all, maybe drawing 100w at most.

    So put it all together:


    Click to view full size!



    Click to view full size!



    Click to view full size!


    The LCD is from LCDmodkit, it’s just a standard HD44880 4x20 characters, with a USB controller board on the back (in Red on Black). They work very well under pfSense, I’ve bought many from him over the years. You can output almost anything to it, we will talk more about that below.

    Lastly, I hooked up an internal USB header, so I can have a small USB stick in there. Not to boot off, although you can with a version of pfSense on NanoBSD, just for config backups. I have a cron job that runs once a week to back up the configs to the USB.

    So that’s it for the hardware, pretty standard stuff, on purpose. This only took a few hours to build, and I took my time, and tested them overnight.


    Click to view full size!



    Click to view full size!



    Click to view full size!


    Software time!

    Installation is a snap, the only gotcha here is that FreeBSD didn’t recognise the SATA3 ports, but plug the drive into a SATA2 and you are good to go. I just chose defaults on the install, using all the drive and default swap space. After that just choose a WAN and LAN port, then you are ready to configure from the front-end. As an aside here, this is FAST. Takes seconds to build and 30 seconds to reboot, from the time you press the reboot button.

    There is actually a nice wizard to take you through the initial setup if you like, otherwise we should jump right into configuring.

    There are some initial setting you should look at, particularly in the general and advanced settings, for ssh, and password protecting the console (definitely!) and a few others.


    Click to view full size!



    Click to view full size!



    Click to view full size!


    Then we can assign interfaces as such.


    Click to view full size!


    Finally the rules!


    Click to view full size!



    Click to view full size!



    Click to view full size!


    Now we can look at some of the really nice aspects of pf, making it a proper UTM, with content filtering, antivirus, and the such.

    Firstly, you “build” your copy of pfSense, with packages that are published from pf and contributors:


    Click to view full size!



    Click to view full size!


    Most of the obvious ones are there, like snort and squid.

    There is HVAP, which you can parent with squid.

    Then as an extension from squid, you can use dansguardian, or squidguard, however a subscription to rules will cost you money, but it’s really a nominal amount.

    With 2.0.2 PPTP is disabled by default, and there is a warning message if you try and enable it, due to the compromise of the MS-CHAP protocol. OpenVPN is a great alternative, and there is now an iPad app to support it. The windows client isn’t very pretty, but it’s very easy to deploy and use, and you can sync it to your Active Directory or any other LDAP.

    The LCD module is configured here, though you might have to drop to a shell to run dmesg, and then edit the config file to suit. in the case of these boxes, the LCD found itself on ugen1.3. Then you can output whatever you want to the display and rotate it every few seconds.


    Click to view full size!



    Click to view full size!


    I wanted this to be more about the software then the hardware, since the hardware is all of-the-shelf stuff. I wanted to demonstrate and impress upon people that for commodity stuff like a firewall, VPN and Internet filtering, there really isn’t a need to pay for it. Even for larger sites.

    To give you a breakdown of costs: (Rough guide, includes GST)

    Case: $200
    CPU: $220
    Motherboard: $212
    RAM: $71
    PSU: $150
    SSDs: $ 328
    NICs: $450


    So total hardware (each) is : $1631.

    Software = $0
    Maintenance = $0
    Support = $0

    Just for reference last time I got quoted to a XTM505 was $2400 for the hardware, plus NBD replacement $600, and the “Pro” upgrade (more clients basically) $220. so call it $3600, and I can’t fix, upgrade, or manage it outside a windows client. Or use a secure VPN solution without paying more for their client.

    And now we talk about the elephant in the room, support. For me, and for most SME’s that I’ve dealt with, this is a bit of a non-issue, as I’m the first support call anyway. If I can’t fix it, well I’d have to wait for support anyway. In an N+1 environment, the chances are small that both would fail. I understand that this sort of arrangement isn’t for everyone, but it’s an acceptable risk for me. There is no support outside of me and my team, and you can get corporate support from pfSense for $500 USD a year, with 5 hours remote support. In the several years I’ve used pfSense, I’ve never had a failure via a software fault, besides my own stupidity with gmirror. I suppose this sort of thing isn’t for everyone, however it really works for me.

    SO my final thoughts. This has been a very worthwhile exercise, and you don’t need to do what I’ve done, However I want this hardware to last a few years at least. I’ve seen sites similar to mine run on much less hardware, even an embedded Atom system would be OK. This flies along, and it really, really easy to backup and restore, even to replace with any sort of hardware you may have lying around. The real seller of this sort of setup for me is the close-to-zero (power and depreciation notwithstanding) ongoing costs. The facts that it performs as well as it does is a bonus.

    AS always, I welcome questions or comments, and if you want to know anything more, just let me know below or in a PM. Though I would prefer to keep any discussion public.

    Cheers, Mario.

    PS. I bought some other toys, there will be another thread in a bit.
     
    Last edited: Mar 14, 2013
  2. FiShy

    FiShy Member

    Joined:
    Aug 15, 2001
    Messages:
    9,682
    Abit of over kill on the spec lol :) but i like it.
     
  3. aza2001

    aza2001 Member

    Joined:
    Sep 14, 2002
    Messages:
    2,016
    Location:
    Northmead
    This is an idea I have been toying with over the last few days except I am considering running it under esxi, just for ease of movement replication, backups and restores.... (plus I would be running it with a ton of other vm's as well)
     
  4. Renza

    Renza Member

    Joined:
    Dec 1, 2004
    Messages:
    4,661
    Location:
    Melbourne
    I personally would have gone for an Asus P8B-M for around $50 more, and a Xeon E3-1220 v2 for the same price, And then add on some ECC ram (~$10 more).

    also lol at the stickers all on the front of the case...
     
    Last edited: Mar 14, 2013
  5. GooSE

    GooSE New Member

    Joined:
    Jun 26, 2001
    Messages:
    6,679
    Location:
    Sydney
    Very nice write up. I must correct you on something though..

    I don't know what version of FreeBSD pfSense uses, but you're wrong. TRIM has been supported on UFS since 9.0-RELEASE, at least.
     
  6. OP
    OP
    samus

    samus Member

    Joined:
    Jun 3, 2002
    Messages:
    1,234
    Location:
    Baulkham Hills, Sydney.
    pfSense uses 8.1-RELEASE-p13. I know that "stock" FreeBSD supports TRIM. Ill edit my post to note that.

    Thanks GooSE for the correction. OP Updated.
     
    Last edited: Mar 14, 2013
  7. LostBenji

    LostBenji Member

    Joined:
    Oct 5, 2007
    Messages:
    6,077
    Location:
    Up a tower somewhere....
    http://ark.intel.com/products/67302/Intel-Server-Board-S1200BTSR

    20 bucks cheaper and then coupled with a small Xeon and ECC, you would have enterprise grade for same coins.

    Sorry dude, but choice of PSU is asking for trouble. The only ATX PSU's that should ever be placed in a 2U case are those packing a 80mm fan in the rear or front of the PSU. That unit will cook. The SFF PSU's do well in this cases as well.

    Not sure what you call a "limited budget"


    Otherwise, great to see PF getting a write-up.
     
    Last edited: Mar 14, 2013
  8. closed_gate

    closed_gate Member

    Joined:
    Oct 21, 2004
    Messages:
    736
    Location:
    Brisbane
    I would assume that these are going into a fully air-con server room. They would be fine in that situation.



    Otherwise, excellent write up. I'm looking at using PFsense for my small business builds.
     
  9. resr

    resr Member

    Joined:
    May 18, 2004
    Messages:
    247
    Thanks for the info :)

    I've been messing about with sophos UTM recently (astaro). There's a free non business, and it's very point and click, but I've recently pulled the hardware out to throw pf on it.

    Sophos licencing costs for small numbers of users is quite nice too.
     
  10. KDog

    KDog Member

    Joined:
    Jan 9, 2002
    Messages:
    246
    Location:
    ACT
    Thanks for the write up.

    I am going to give this a shot on a spare desktop I have around here. Getting fed up with all the license costs for the cisco ASA I have, plus it's not as user friendly as all the guides, training etc make out once you need to do more than a few interfaces, multiple VLANs with access rules everywhere (or maybe it's just me, I am not a sysadmin by trade).

    Will this support two factor authentication? I will need to add on some RSA 2F, token, for external VPN connections at some point. Is anyone doing this?
     
  11. Xcaliba000

    Xcaliba000 Member

    Joined:
    Feb 19, 2007
    Messages:
    1
    I use PF-Sense as a virtual firewalll appliance on nearly all of my small business that i manage - Excellent piece of software
     
  12. Xon

    Xon Member

    Joined:
    Jan 11, 2006
    Messages:
    50
    I'ld recommend using the nanobsd+vga version of pfSense if you are installing it in a VM or on a SSD.

    It is functionally the same as a full pfsense install, except it mounts the OS disk as readonly and basicly stops a crazy amount of random diskchatter from random parts of the OS. Something like Squid can go live on it's on partition which gets mounted as read/write anyway.

    That you can make pfsense authenticate against AD is also quite cool.
     
  13. Glide

    Glide Member

    Joined:
    Aug 22, 2002
    Messages:
    1,151
    Location:
    Was: Sydney Now: USA
    I gave pfsense a go the other day as I needed to test a specific networking scenario in the lab and endian FW didnt seem to play ball.

    Found the same issue with pfsense, could not make it do a regular NAT/port forward between two non-wan interfaces.

    Nearly gave up and then tried m0n0wall, and with the same config it worked exactly as expected. Strange.

    That being said, I do like endian for its ease, but pfsense seems quite good as well, with a lot more options.
     
  14. OP
    OP
    samus

    samus Member

    Joined:
    Jun 3, 2002
    Messages:
    1,234
    Location:
    Baulkham Hills, Sydney.
    Thanks everyone for the replies!

    Lost_benji, i didn't even know Intel had that sort of board out there! and closed_gate is right, in my ~20c server room exhaust air is only about 21-22c. I'm drawing literally about 80w at load.

    KDog, this thread may be of intrest to you if you haven't already read it.

    Glide, I've tried endian, and i can confidently say that pf beats it in just about every way, except the pretty interface endian has. What exactly are you trying to do over with your NAT scenario?
     
  15. aza2001

    aza2001 Member

    Joined:
    Sep 14, 2002
    Messages:
    2,016
    Location:
    Northmead
    I don't think you "nat" a port exactly (I guess you could) but you would probably need to update the metric to show how the traffic should flow. (Similar situation I faced last night with our "enterprise" dlink at work - long story) but I am nearly 100
     
  16. maddhatter

    maddhatter Member

    Joined:
    Jun 27, 2001
    Messages:
    4,798
    Location:
    Mackay, QLD.
    A bog standard copy of pfSense != UTM.

    With the exception of Squid, all i'm seeing is a fancy NAT Router?

    Unless you're not telling us something :)
     
  17. OP
    OP
    samus

    samus Member

    Joined:
    Jun 3, 2002
    Messages:
    1,234
    Location:
    Baulkham Hills, Sydney.
    And you are absolutely right!

    The packages is what gives pf the power to do so much more,

    In my environment, i use squid in conjunction with squidguard to enable content filtering, HVAP to scan incoming content for viruses and malware, snort for intrusion detection, and openvpn to enable remote access. As well as some prettying up with lcdmod and widescreen to make the webui look nicer.

    Is this something you would like me to cover as well? I got the basics down, but if people want me to cover the package(s) configuration ill do that too.

    I also use it as an ipsec tunnel to a remote site, and created a walled garden for some wifi users.
     
    Last edited: Mar 16, 2013
  18. Fitzi

    Fitzi Member

    Joined:
    Jun 27, 2001
    Messages:
    519
    Location:
    Central Coast, NSW
    I would be interested in seeing how you have put the config together UTM style.
     
  19. LostBenji

    LostBenji Member

    Joined:
    Oct 5, 2007
    Messages:
    6,077
    Location:
    Up a tower somewhere....
    The shear amount of Packages makes it so flexible to suit the needs of the requirements. I have used it plenty of times but not full-time like a want to. I just don't have the coins to replace my other combos just yet.

    The best bit I like is the auto mirroring of drives (set a pair of drives to AHCI before installing) to give that seamless Pro feel. The add the backup config that allows you to run up another machine, import the config and you do nothing more than assign adapters if its new or different hardware. It will pull down all packages you had on other machine.
     
  20. OP
    OP
    samus

    samus Member

    Joined:
    Jun 3, 2002
    Messages:
    1,234
    Location:
    Baulkham Hills, Sydney.
    OK then, Im putting together some screenshots and stuff now.
     

Share This Page

Advertisement: