Setup A FreeBSD Web Server–Part 2–Jails

on August 18, 2011 in Freebsd

This post is continuing from Part 1 of our series on how to setup a FreeBSD Web Server.

If you ordered a number of IP address then you might want to set them up as individual jails. One for your DNS server, one for you backend server and the rest as individual webserver IP addresses (I call them frontend servers). I know you can do this in apache with virtual servers but using jails makes it more secure and gives you more flexibility.

I prefer to use what is referred to as a “complete” jail in FreeBSD that is similar in concept to a Solaris 10 zone. Maybe some of the FreeBSD developers work for Sun (now Oracle). Who knows! Some people create what is termed “service” jails that are just for one service or application e.g. apache. This would give you added security benefits but is a lot of work.

First we need to allow for ping and other raw socket services to work in jails /etc/sysctl.conf setting. You can skip this step if you want more security and don’t need ping.

hk# echo ‘security.jail.allow_raw_sockets=1′ >> /etc/sysctl.conf

Now enable ezjails to start on server boot up.

hk# echo ‘ezjail_enable=”YES”‘ >> /etc/rc.conf

Now set up some default configuration for the jails

hk# cd /usr/local/etc/
hk# cp ezjail.conf.sample ezjail.conf

Edit /usr/local/etc/ezjail.conf and uncomment the following.

ezjail_jaildir=/usr/jails
ezjail_jailtemplate=${ezjail_jaildir}/newjail
ezjail_jailbase=${ezjail_jaildir}/basejail
ezjail_sourcetree=/usr/src
ezjail_uglyperlhack=”YES”
ezjail_mount_enable=”YES”
ezjail_devfs_enable=”YES”
ezjail_procfs_enable=”YES”
ezjail_fdescfs_enable=”YES”
ezjail_ftphost=ftp6.freebsd.org

Jails are like a mini version of the operating system. So a very time consuming step we need to do is create a full root file system.

hk# cp /usr/share/examples/cvsup/standard-supfile /root/supfile
hk# vi /root/supfile

Go to the FreeBSD handbook and select the closest CVS server to your server to download the source tree from, and then add this CVS server to /root/supfile.

*default host=cvsup6.FreeBSD.org

Now tell csup to download the latest source tree for the current release.

hk# cd /usr/src
hk# csup /root/supfile

Go and get a cup of coffee as this will take a long time.

Next we build a complete operating system for for the jail. This will create the base jail which has the binaries shared by all jails.

/usr/jails/basejail/

hk# ezjail-admin update –b

Now go and get another cup of coffee.

The next step is to create a standard configuration for a jail such as /etc/resolv.conf, /etc/nsswitch.conf and /etc/make.conf

hk# cd /usr/jails/flavours/
hk# cp -R example standard
hk# cd standard
hk# mkdir PORTS

Edit etc/nsswitch.conf and remove all references to NIS. You don’t need it.

group: files
hosts: files dns
networks: files
passwd: files
shells: files
services: files
protocols: files
rpc: files

Edit etc/make.conf and append the following lines

NO_X=true
NO_X11=true
WITHOUT_X11=true
NO_NIS=true
WITHOUT_NIS=true
# Recompile apache to get this working
SUEXEC_DOCROOT=”/home”

OPTIONAL:

You can setup to use google’s DNS servers or the opendns.org DNS servers here also

vi etc/resolv.conf

nameserver 8.8.8.8
nameserver 8.8.8.4

Now setup your jail ip addresses in your main /etc/hosts file and also in /etc/rc.conf

/etc/hosts

192.168.1.51 ns5.ftmon.org
192.168.1.52 backend1.ftmon.org
192.168.1.53 frontend1.ftmon.org
192.168.1.54 frontend2.ftmon.org

/etc/rc.conf

ifconfig_em0_alias0=”inet 192.168.1.51 netmask 255.255.255.0″
ifconfig_em0_alias1=”inet 192.168.1.52 netmask 255.255.255.0″
ifconfig_em0_alias2=”inet 192.168.1.53 netmask 255.255.255.0″
ifconfig_em0_alias2=”inet 192.168.1.54 netmask 255.255.255.0″

At this point you can reboot or plumb up the interfaces manually

hk# ifconfig em0 192.168.1.51 netmask 255.255.255.0 alias
hk# ifconfig em0 192.168.1.52 netmask 255.255.255.0 alias
hk# ifconfig em0 192.168.1.53 netmask 255.255.255.0 alias
hk# ifconfig em0 192.168.1.54 netmask 255.255.255.0 alias

And verify

hk# ifconfig -a
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 08:00:27:ab:01:fd
        inet 192.168.1.50 netmask 0xffffff00 broadcast 192.168.1.255
        inet 192.168.1.51 netmask 0xffffff00 broadcast 192.168.1.255
        inet 192.168.1.52 netmask 0xffffff00 broadcast 192.168.1.255
        inet 192.168.1.53 netmask 0xffffff00 broadcast 192.168.1.255
        inet 192.168.1.54 netmask 0xffffff00 broadcast 192.168.1.255

Now let’s create the jails

hk# ezjail-admin create -f standard ns5.ftmon.org 192.168.1.51
hk# ezjail-admin create -f standard backend1.ftmon.org 192.168.1.52
hk# ezjail-admin create -f standard frontend1.ftmon.org 192.168.1.53
hk# ezjail-admin create -f standard frontend2.ftmon.org 192.168.1.53

NOTE:

Warning: Some services already seem to be listening on all IP, (including 192.168.1.53)
This may cause some confusion, here they are:
root sshd 772 4 tcp4 *:22 *:*
root syslogd 490 7 udp4 *:514 *:*

If you get this error when creating jails then make sure that syslog and sshd only bind to the main IP not the jail ips.

/etc/rc.conf

syslogd_flags=”-ss”
inetd_flags=”-wW -C 60 -a 192.168.1.50″

/etc/ssh/sshd_config

ListenAddress 192.168.1.50

To allow for packages to be installed within each jail, I like to reference to the main operating tree packages list for space efficiency.

hk# cd /usr/jails/basejail/usr
hk# ln -s /PORTS ports

Then for each jail fstab add a link to the main operating systems /usr/ports

hk# vi /etc/fstab.ns5_ftmon_org
hk# vi /etc/fstab.frontend1_ftmon_org
hk# vi /etc/fstab.frontend2_ftmon_org
hk# vi /etc/fstab.backend1_ftmon_org

/usr/ports /usr/jails/ns5.ftmon.org/PORTS nullfs ro 0 0

You can start up your jails at this point or do a sanity reboot to see if they come up.

hk# /usr/local/etc/rc.d/ezjail.sh start
ezjailConfiguring jails:.
Starting jails: ns5.ftmon.org frontend2.ftmon.org frontend1.ftmon.org backend1.ftmon.org.

To check the status of your jails, use the jls command.

hk# jls
   JID IP Address      Hostname                      Path
     1 192.168.1.51    ns5.ftmon.org                 /usr/jails/ns5.ftmon.org
     2 192.168.1.53    frontend2.ftmon.org           /usr/jails/frontend2.ftmon.org
     3 192.168.1.53    frontend1.ftmon.org           /usr/jails/frontend1.ftmon.org
     4 192.168.1.52    backend1.ftmon.org            /usr/jails/backend1.ftmon.org

To connect to a jails use the jexec command.