3.\"
4.NH
5Managing Jails and the Jail File System Environment
6.NH 2
7Creating a Jail Environment
8.PP
9While the jail(2) call could be used in a number of ways, the expected
10configuration creates a complete FreeBSD installation for each jail.
11This includes copies of all relevant system binaries, data files, and its
12own \fC/etc\fP directory.
13Such a configuration maximises the independence of various jails,
14and reduces the chances of interference between jails being possible,
15especially when it is desirable to provide root access within a jail to
16a less trusted user.
17.PP
18On a box making use of the jail facility, we refer to two types of
19environment: the host environment, and the jail environment.
20The host environment is the real operating system environment, which is
21used to configure interfaces, and start up the jails.
22There are then one or more jail environments, effectively virtual
23FreeBSD machines.
24When configuring Jail for use, it is necessary to configure both the
25host and jail environments to prevent overlap.
26.PP
27As jailed virtual machines are generally bound to an IP address configured
28using the normal IP alias mechanism, those jail IP addresses are also
29accessible to host environment applications to use.
30If the accessibility of some host applications in the jail environment is
31not desirable, it is necessary to configure those applications to only
32listen on appropriate addresses.
33.PP
34In most of the production environments where jail is currently in use,
35one IP address is allocated to the host environment, and then a number
36are allocated to jail boxes, with each jail box receiving a unique IP.
37In this situation, it is sufficient to configure the networking applications
38on the host to listen only on the host IP.
39Generally, this consists of specifying the appropriate IP address to be
40used by inetd and SSH, and disabling applications that are not capable
41of limiting their address scope, such as sendmail, the port mapper, and
42syslogd.
43Other third party applications that have been installed on the host must also be
44configured in this manner, or users connecting to the jailbox will
45discover the host environment service, unless the jailbox has
46specifically bound a service to that port.
47In some situations, this can actually be the desirable behaviour.
48.PP
49The jail environments must also be custom-configured.
50This consists of building and installing a miniature version of the
51FreeBSD file system tree off of a subdirectory in the host environment,
52usually \fC/usr/jail\fP, or \fC/data/jail\fP, with a subdirectory per jail.
53Appropriate instructions for generating this tree are included in the
54jail(8) man page, but generally this process may be automated using the
55FreeBSD build environment.
56.PP
57One notable difference from the default FreeBSD install is that only
58a limited set of device nodes should be created.
| 3.\"
4.NH
5Managing Jails and the Jail File System Environment
6.NH 2
7Creating a Jail Environment
8.PP
9While the jail(2) call could be used in a number of ways, the expected
10configuration creates a complete FreeBSD installation for each jail.
11This includes copies of all relevant system binaries, data files, and its
12own \fC/etc\fP directory.
13Such a configuration maximises the independence of various jails,
14and reduces the chances of interference between jails being possible,
15especially when it is desirable to provide root access within a jail to
16a less trusted user.
17.PP
18On a box making use of the jail facility, we refer to two types of
19environment: the host environment, and the jail environment.
20The host environment is the real operating system environment, which is
21used to configure interfaces, and start up the jails.
22There are then one or more jail environments, effectively virtual
23FreeBSD machines.
24When configuring Jail for use, it is necessary to configure both the
25host and jail environments to prevent overlap.
26.PP
27As jailed virtual machines are generally bound to an IP address configured
28using the normal IP alias mechanism, those jail IP addresses are also
29accessible to host environment applications to use.
30If the accessibility of some host applications in the jail environment is
31not desirable, it is necessary to configure those applications to only
32listen on appropriate addresses.
33.PP
34In most of the production environments where jail is currently in use,
35one IP address is allocated to the host environment, and then a number
36are allocated to jail boxes, with each jail box receiving a unique IP.
37In this situation, it is sufficient to configure the networking applications
38on the host to listen only on the host IP.
39Generally, this consists of specifying the appropriate IP address to be
40used by inetd and SSH, and disabling applications that are not capable
41of limiting their address scope, such as sendmail, the port mapper, and
42syslogd.
43Other third party applications that have been installed on the host must also be
44configured in this manner, or users connecting to the jailbox will
45discover the host environment service, unless the jailbox has
46specifically bound a service to that port.
47In some situations, this can actually be the desirable behaviour.
48.PP
49The jail environments must also be custom-configured.
50This consists of building and installing a miniature version of the
51FreeBSD file system tree off of a subdirectory in the host environment,
52usually \fC/usr/jail\fP, or \fC/data/jail\fP, with a subdirectory per jail.
53Appropriate instructions for generating this tree are included in the
54jail(8) man page, but generally this process may be automated using the
55FreeBSD build environment.
56.PP
57One notable difference from the default FreeBSD install is that only
58a limited set of device nodes should be created.
|
61.PP
62To improve storage efficiency, a fair number of the binaries in the system tree
63may be deleted, as they are not relevant in a jail environment.
64This includes the kernel, boot loader, and related files, as well as
65hardware and network configuration tools.
66.PP
67After the creation of the jail tree, the easiest way to configure it is
68to start up the jail in single-user mode.
69The sysinstall admin tool may be used to help with the task, although
70it is not installed by default as part of the system tree.
71These tools should be run in the jail environment, or they will affect
72the host environment's configuration.
73.DS
74.ft C
75.ps -2
76# mkdir /data/jail/192.168.11.100/stand
77# cp /stand/sysinstall /data/jail/192.168.11.100/stand
78# jail /data/jail/192.168.11.100 testhostname 192.168.11.100 \e
79 /bin/sh
80.ps +2
81.R
82.DE
83.PP
84After running the jail command, the shell is now within the jail environment,
85and all further commands
86will be limited to the scope of the jail until the shell exits.
87If the network alias has not yet been configured, then the jail will be
88unable to access the network.
89.PP
90The startup configuration of the jail environment may be configured so
91as to quell warnings from services that cannot run in the jail.
92Also, any per-system configuration required for a normal FreeBSD system
93is also required for each jailbox.
94Typically, this includes:
95.IP "" 5n
96\(bu Create empty /etc/fstab
97.IP
98\(bu Disable portmapper
99.IP
100\(bu Run newaliases
101.IP
102\(bu Disabling interface configuration
103.IP
104\(bu Configure the resolver
105.IP
106\(bu Set root password
107.IP
108\(bu Set timezone
109.IP
110\(bu Add any local accounts
111.IP
112\(bu Install any packets
113.NH 2
114Starting Jails
115.PP
116Jails are typically started by executing their /etc/rc script in much
117the same manner a shell was started in the previous section.
118Before starting the jail, any relevant networking configuration
119should also be performed.
120Typically, this involves adding an additional IP address to the
121appropriate network interface, setting network properties for the
122IP address using IP filtering, forwarding, and bandwidth shaping,
123and mounting a process file system for the jail, if the ability to
124debug processes from within the jail is desired.
125.DS
126.ft C
127.ps -2
128# ifconfig ed0 inet add 192.168.11.100 netmask 255.255.255.255
129# mount -t procfs proc /data/jail/192.168.11.100/proc
130# jail /data/jail/192.168.11.100 testhostname 192.168.11.100 \e
131 /bin/sh /etc/rc
132.ps +2
133.ft P
134.DE
135.PP
136A few warnings are generated for sysctl's that are not permitted
137to be set within the jail, but the end result is a set of processes
138in an isolated process environment, bound to a single IP address.
139Normal procedures for accessing a FreeBSD machine apply: telneting in
140through the network reveals a telnet prompt, login, and shell.
141.DS
142.ft C
143.ps -2
144% ps ax
145 PID TT STAT TIME COMMAND
146 228 ?? SsJ 0:18.73 syslogd
147 247 ?? IsJ 0:00.05 inetd -wW
148 249 ?? IsJ 0:28.43 cron
149 252 ?? SsJ 0:30.46 sendmail: accepting connections on port 25
150 291 ?? IsJ 0:38.53 /usr/local/sbin/sshd
15193694 ?? SJ 0:01.01 sshd: rwatson@ttyp0 (sshd)
15293695 p0 SsJ 0:00.06 -csh (csh)
15393700 p0 R+J 0:00.00 ps ax
154.ps +2
155.ft P
156.DE
157.PP
158It is immediately obvious that the environment is within a jailbox: there
159is no init process, no kernel daemons, and a J flag is present beside all
160processes indicating the presence of a jail.
161.PP
162As with any FreeBSD system, accounts may be created and deleted,
163mail is delivered, logs are generated, packages may be added, and the
164system may be hacked into if configured incorrectly, or running a buggy
165version of a piece of software.
166However, all of this happens strictly within the scope of the jail.
167.NH 2
168Jail Management
169.PP
170Jail management is an interesting prospect, as there are two perspectives
171from which a jail environment may be administered: from within the jail,
172and from the host environment.
173From within the jail, as described above, the process is remarkably similar
174to any regular FreeBSD install, although certain actions are prohibited,
175such as mounting file systems, modifying system kernel properties, etc.
176The only area that really differs are that of shutting
177the system down: the processes within the jail may deliver signals
178between them, allowing all processes to be killed, but bringing the
179system back up requires intervention from outside of the jailbox.
180.PP
181From outside of the jail, there are a range of capabilities, as well
182as limitations.
183The jail environment is, in effect, a subset of the host environment:
184the jail file system appears as part of the host file system, and may
185be directly modified by processes in the host environment.
186Processes within the jail appear in the process listing of the host,
187and may likewise be signalled or debugged.
188The host process file system makes the hostname of the jail environment
189accessible in /proc/procnum/status, allowing utilities in the host
190environment to manage processes based on jailname.
191However, the default configuration allows privileged processes within
192jails to set the hostname of the jail, which makes the status file less
193useful from a management perspective if the contents of the jail are
194malicious.
195To prevent a jail from changing its hostname, the
196"security.jail.set_hostname_allowed" sysctl may be set to 0 prior to
197starting any jails.
198.PP
199One aspect immediately observable in an environment with multiple jails
200is that uids and gids are local to each jail environment: the uid associated
201with a process in one jail may be for a different user than in another
202jail.
203This collision of identifiers is only visible in the host environment,
204as normally processes from one jail are never visible in an environment
205with another scope for user/uid and group/gid mapping.
206Managers in the host environment should understand these scoping issues,
207or confusion and unintended consequences may result.
208.PP
209Jailed processes are subject to the normal restrictions present for
210any processes, including resource limits, and limits placed by the network
211code, including firewall rules.
212By specifying firewall rules for the IP address bound to a jail, it is
213possible to place connectivity and bandwidth limitations on individual
214jails, restricting services that may be consumed or offered.
215.PP
216Management of jails is an area that will see further improvement in
217future versions of FreeBSD. Some of these potential improvements are
218discussed later in this paper.
| 59.PP
60To improve storage efficiency, a fair number of the binaries in the system tree
61may be deleted, as they are not relevant in a jail environment.
62This includes the kernel, boot loader, and related files, as well as
63hardware and network configuration tools.
64.PP
65After the creation of the jail tree, the easiest way to configure it is
66to start up the jail in single-user mode.
67The sysinstall admin tool may be used to help with the task, although
68it is not installed by default as part of the system tree.
69These tools should be run in the jail environment, or they will affect
70the host environment's configuration.
71.DS
72.ft C
73.ps -2
74# mkdir /data/jail/192.168.11.100/stand
75# cp /stand/sysinstall /data/jail/192.168.11.100/stand
76# jail /data/jail/192.168.11.100 testhostname 192.168.11.100 \e
77 /bin/sh
78.ps +2
79.R
80.DE
81.PP
82After running the jail command, the shell is now within the jail environment,
83and all further commands
84will be limited to the scope of the jail until the shell exits.
85If the network alias has not yet been configured, then the jail will be
86unable to access the network.
87.PP
88The startup configuration of the jail environment may be configured so
89as to quell warnings from services that cannot run in the jail.
90Also, any per-system configuration required for a normal FreeBSD system
91is also required for each jailbox.
92Typically, this includes:
93.IP "" 5n
94\(bu Create empty /etc/fstab
95.IP
96\(bu Disable portmapper
97.IP
98\(bu Run newaliases
99.IP
100\(bu Disabling interface configuration
101.IP
102\(bu Configure the resolver
103.IP
104\(bu Set root password
105.IP
106\(bu Set timezone
107.IP
108\(bu Add any local accounts
109.IP
110\(bu Install any packets
111.NH 2
112Starting Jails
113.PP
114Jails are typically started by executing their /etc/rc script in much
115the same manner a shell was started in the previous section.
116Before starting the jail, any relevant networking configuration
117should also be performed.
118Typically, this involves adding an additional IP address to the
119appropriate network interface, setting network properties for the
120IP address using IP filtering, forwarding, and bandwidth shaping,
121and mounting a process file system for the jail, if the ability to
122debug processes from within the jail is desired.
123.DS
124.ft C
125.ps -2
126# ifconfig ed0 inet add 192.168.11.100 netmask 255.255.255.255
127# mount -t procfs proc /data/jail/192.168.11.100/proc
128# jail /data/jail/192.168.11.100 testhostname 192.168.11.100 \e
129 /bin/sh /etc/rc
130.ps +2
131.ft P
132.DE
133.PP
134A few warnings are generated for sysctl's that are not permitted
135to be set within the jail, but the end result is a set of processes
136in an isolated process environment, bound to a single IP address.
137Normal procedures for accessing a FreeBSD machine apply: telneting in
138through the network reveals a telnet prompt, login, and shell.
139.DS
140.ft C
141.ps -2
142% ps ax
143 PID TT STAT TIME COMMAND
144 228 ?? SsJ 0:18.73 syslogd
145 247 ?? IsJ 0:00.05 inetd -wW
146 249 ?? IsJ 0:28.43 cron
147 252 ?? SsJ 0:30.46 sendmail: accepting connections on port 25
148 291 ?? IsJ 0:38.53 /usr/local/sbin/sshd
14993694 ?? SJ 0:01.01 sshd: rwatson@ttyp0 (sshd)
15093695 p0 SsJ 0:00.06 -csh (csh)
15193700 p0 R+J 0:00.00 ps ax
152.ps +2
153.ft P
154.DE
155.PP
156It is immediately obvious that the environment is within a jailbox: there
157is no init process, no kernel daemons, and a J flag is present beside all
158processes indicating the presence of a jail.
159.PP
160As with any FreeBSD system, accounts may be created and deleted,
161mail is delivered, logs are generated, packages may be added, and the
162system may be hacked into if configured incorrectly, or running a buggy
163version of a piece of software.
164However, all of this happens strictly within the scope of the jail.
165.NH 2
166Jail Management
167.PP
168Jail management is an interesting prospect, as there are two perspectives
169from which a jail environment may be administered: from within the jail,
170and from the host environment.
171From within the jail, as described above, the process is remarkably similar
172to any regular FreeBSD install, although certain actions are prohibited,
173such as mounting file systems, modifying system kernel properties, etc.
174The only area that really differs are that of shutting
175the system down: the processes within the jail may deliver signals
176between them, allowing all processes to be killed, but bringing the
177system back up requires intervention from outside of the jailbox.
178.PP
179From outside of the jail, there are a range of capabilities, as well
180as limitations.
181The jail environment is, in effect, a subset of the host environment:
182the jail file system appears as part of the host file system, and may
183be directly modified by processes in the host environment.
184Processes within the jail appear in the process listing of the host,
185and may likewise be signalled or debugged.
186The host process file system makes the hostname of the jail environment
187accessible in /proc/procnum/status, allowing utilities in the host
188environment to manage processes based on jailname.
189However, the default configuration allows privileged processes within
190jails to set the hostname of the jail, which makes the status file less
191useful from a management perspective if the contents of the jail are
192malicious.
193To prevent a jail from changing its hostname, the
194"security.jail.set_hostname_allowed" sysctl may be set to 0 prior to
195starting any jails.
196.PP
197One aspect immediately observable in an environment with multiple jails
198is that uids and gids are local to each jail environment: the uid associated
199with a process in one jail may be for a different user than in another
200jail.
201This collision of identifiers is only visible in the host environment,
202as normally processes from one jail are never visible in an environment
203with another scope for user/uid and group/gid mapping.
204Managers in the host environment should understand these scoping issues,
205or confusion and unintended consequences may result.
206.PP
207Jailed processes are subject to the normal restrictions present for
208any processes, including resource limits, and limits placed by the network
209code, including firewall rules.
210By specifying firewall rules for the IP address bound to a jail, it is
211possible to place connectivity and bandwidth limitations on individual
212jails, restricting services that may be consumed or offered.
213.PP
214Management of jails is an area that will see further improvement in
215future versions of FreeBSD. Some of these potential improvements are
216discussed later in this paper.
|