1.\"
2.\" $FreeBSD: head/share/doc/papers/jail/mgt.ms 89414 2002-01-16 06:55:30Z arr $
3.\"
4.NH
5Managing Jails and the Jail File System Environment
6.NH 2
7Creating a Jail Environment
8.PP
9While the jail(2) call could be used in a number of ways, the expected
10configuration creates a complete FreeBSD installation for each jail.
11This includes copies of all relevant system binaries, data files, and its
12own \fC/etc\fP directory.
13Such a configuration maximises the independence of various jails,
14and reduces the chances of interference between jails being possible,
15especially when it is desirable to provide root access within a jail to
16a less trusted user.
17.PP
18On a box making use of the jail facility, we refer to two types of
19environment: the host environment, and the jail environment.
20The host environment is the real operating system environment, which is
21used to configure interfaces, and start up the jails.
22There are then one or more jail environments, effectively virtual
23FreeBSD machines.
24When configuring Jail for use, it is necessary to configure both the
25host and jail environments to prevent overlap.
26.PP
27As jailed virtual machines are generally bound to an IP address configured
28using the normal IP alias mechanism, those jail IP addresses are also
29accessible to host environment applications to use.
30If the accessibility of some host applications in the jail environment is
31not desirable, it is necessary to configure those applications to only
32listen on appropriate addresses.
33.PP
34In most of the production environments where jail is currently in use,
35one IP address is allocated to the host environment, and then a number
36are allocated to jail boxes, with each jail box receiving a unique IP.
37In this situation, it is sufficient to configure the networking applications
38on the host to listen only on the host IP.
39Generally, this consists of specifying the appropriate IP address to be
40used by inetd and SSH, and disabling applications that are not capable
41of limiting their address scope, such as sendmail, the port mapper, and
42syslogd.
43Other third party applications that have been installed on the host must also be
44configured in this manner, or users connecting to the jailbox will
45discover the host environment service, unless the jailbox has
46specifically bound a service to that port.
47In some situations, this can actually be the desirable behaviour.
48.PP
49The jail environments must also be custom-configured.
50This consists of building and installing a miniature version of the
51FreeBSD file system tree off of a subdirectory in the host environment,
52usually \fC/usr/jail\fP, or \fC/data/jail\fP, with a subdirectory per jail.
53Appropriate instructions for generating this tree are included in the
54jail(8) man page, but generally this process may be automated using the
55FreeBSD build environment.
56.PP
57One notable difference from the default FreeBSD install is that only
58a limited set of device nodes should be created.
59MAKEDEV(8) has been modified to accept a ``jail'' argument that creates
60the correct set of nodes.
61.PP
62To improve storage efficiency, a fair number of the binaries in the system tree
63may be deleted, as they are not relevant in a jail environment.
64This includes the kernel, boot loader, and related files, as well as
65hardware and network configuration tools.
66.PP
67After the creation of the jail tree, the easiest way to configure it is
68to start up the jail in single-user mode.
69The sysinstall admin tool may be used to help with the task, although
70it is not installed by default as part of the system tree.
71These tools should be run in the jail environment, or they will affect
72the host environment's configuration.
73.DS
74.ft C
75.ps -2
76# mkdir /data/jail/192.168.11.100/stand
77# cp /stand/sysinstall /data/jail/192.168.11.100/stand
78# jail /data/jail/192.168.11.100 testhostname 192.168.11.100 \e
79 /bin/sh
80.ps +2
81.R
82.DE
83.PP
84After running the jail command, the shell is now within the jail environment,
85and all further commands
86will be limited to the scope of the jail until the shell exits.
87If the network alias has not yet been configured, then the jail will be
88unable to access the network.
89.PP
90The startup configuration of the jail environment may be configured so
91as to quell warnings from services that cannot run in the jail.
92Also, any per-system configuration required for a normal FreeBSD system
93is also required for each jailbox.
94Typically, this includes:
95.IP "" 5n
96\(bu Create empty /etc/fstab
97.IP
98\(bu Disable portmapper
99.IP
100\(bu Run newaliases
101.IP
102\(bu Disabling interface configuration
103.IP
104\(bu Configure the resolver
105.IP
106\(bu Set root password
107.IP
108\(bu Set timezone
109.IP
110\(bu Add any local accounts
111.IP
112\(bu Install any packets
113.NH 2
114Starting Jails
115.PP
116Jails are typically started by executing their /etc/rc script in much
117the same manner a shell was started in the previous section.
118Before starting the jail, any relevant networking configuration
119should also be performed.
120Typically, this involves adding an additional IP address to the
121appropriate network interface, setting network properties for the
122IP address using IP filtering, forwarding, and bandwidth shaping,
123and mounting a process file system for the jail, if the ability to
124debug processes from within the jail is desired.
125.DS
126.ft C
127.ps -2
128# ifconfig ed0 inet add 192.168.11.100 netmask 255.255.255.255
129# mount -t procfs proc /data/jail/192.168.11.100/proc
130# jail /data/jail/192.168.11.100 testhostname 192.168.11.100 \e
131 /bin/sh /etc/rc
132.ps +2
133.ft P
134.DE
135.PP
136A few warnings are generated for sysctl's that are not permitted
137to be set within the jail, but the end result is a set of processes
138in an isolated process environment, bound to a single IP address.
139Normal procedures for accessing a FreeBSD machine apply: telneting in
140through the network reveals a telnet prompt, login, and shell.
141.DS
142.ft C
143.ps -2
144% ps ax
145 PID TT STAT TIME COMMAND
146 228 ?? SsJ 0:18.73 syslogd
147 247 ?? IsJ 0:00.05 inetd -wW
148 249 ?? IsJ 0:28.43 cron
149 252 ?? SsJ 0:30.46 sendmail: accepting connections on port 25
150 291 ?? IsJ 0:38.53 /usr/local/sbin/sshd
15193694 ?? SJ 0:01.01 sshd: rwatson@ttyp0 (sshd)
15293695 p0 SsJ 0:00.06 -csh (csh)
15393700 p0 R+J 0:00.00 ps ax
154.ps +2
155.ft P
156.DE
157.PP
158It is immediately obvious that the environment is within a jailbox: there
159is no init process, no kernel daemons, and a J flag is present beside all
160processes indicating the presence of a jail.
161.PP
162As with any FreeBSD system, accounts may be created and deleted,
163mail is delivered, logs are generated, packages may be added, and the
164system may be hacked into if configured incorrectly, or running a buggy
165version of a piece of software.
166However, all of this happens strictly within the scope of the jail.
167.NH 2
168Jail Management
169.PP
170Jail management is an interesting prospect, as there are two perspectives
171from which a jail environment may be administered: from within the jail,
172and from the host environment.
173From within the jail, as described above, the process is remarkably similar
174to any regular FreeBSD install, although certain actions are prohibited,
175such as mounting file systems, modifying system kernel properties, etc.
176The only area that really differs are that of shutting
177the system down: the processes within the jail may deliver signals
178between them, allowing all processes to be killed, but bringing the
179system back up requires intervention from outside of the jailbox.
180.PP
181From outside of the jail, there are a range of capabilities, as well
182as limitations.
183The jail environment is, in effect, a subset of the host environment:
184the jail file system appears as part of the host file system, and may
185be directly modified by processes in the host environment.
186Processes within the jail appear in the process listing of the host,
187and may likewise be signalled or debugged.
188The host process file system makes the hostname of the jail environment
189accessible in /proc/procnum/status, allowing utilities in the host
190environment to manage processes based on jailname.
191However, the default configuration allows privileged processes within
192jails to set the hostname of the jail, which makes the status file less
193useful from a management perspective if the contents of the jail are
194malicious.
195To prevent a jail from changing its hostname, the
196"security.jail.set_hostname_allowed" sysctl may be set to 0 prior to
197starting any jails.
198.PP
199One aspect immediately observable in an environment with multiple jails
200is that uids and gids are local to each jail environment: the uid associated
201with a process in one jail may be for a different user than in another
202jail.
203This collision of identifiers is only visible in the host environment,
204as normally processes from one jail are never visible in an environment
205with another scope for user/uid and group/gid mapping.
206Managers in the host environment should understand these scoping issues,
207or confusion and unintended consequences may result.
208.PP
209Jailed processes are subject to the normal restrictions present for
210any processes, including resource limits, and limits placed by the network
211code, including firewall rules.
212By specifying firewall rules for the IP address bound to a jail, it is
213possible to place connectivity and bandwidth limitations on individual
214jails, restricting services that may be consumed or offered.
215.PP
216Management of jails is an area that will see further improvement in
217future versions of FreeBSD. Some of these potential improvements are
218discussed later in this paper.
2.\" $FreeBSD: head/share/doc/papers/jail/mgt.ms 89414 2002-01-16 06:55:30Z arr $
3.\"
4.NH
5Managing Jails and the Jail File System Environment
6.NH 2
7Creating a Jail Environment
8.PP
9While the jail(2) call could be used in a number of ways, the expected
10configuration creates a complete FreeBSD installation for each jail.
11This includes copies of all relevant system binaries, data files, and its
12own \fC/etc\fP directory.
13Such a configuration maximises the independence of various jails,
14and reduces the chances of interference between jails being possible,
15especially when it is desirable to provide root access within a jail to
16a less trusted user.
17.PP
18On a box making use of the jail facility, we refer to two types of
19environment: the host environment, and the jail environment.
20The host environment is the real operating system environment, which is
21used to configure interfaces, and start up the jails.
22There are then one or more jail environments, effectively virtual
23FreeBSD machines.
24When configuring Jail for use, it is necessary to configure both the
25host and jail environments to prevent overlap.
26.PP
27As jailed virtual machines are generally bound to an IP address configured
28using the normal IP alias mechanism, those jail IP addresses are also
29accessible to host environment applications to use.
30If the accessibility of some host applications in the jail environment is
31not desirable, it is necessary to configure those applications to only
32listen on appropriate addresses.
33.PP
34In most of the production environments where jail is currently in use,
35one IP address is allocated to the host environment, and then a number
36are allocated to jail boxes, with each jail box receiving a unique IP.
37In this situation, it is sufficient to configure the networking applications
38on the host to listen only on the host IP.
39Generally, this consists of specifying the appropriate IP address to be
40used by inetd and SSH, and disabling applications that are not capable
41of limiting their address scope, such as sendmail, the port mapper, and
42syslogd.
43Other third party applications that have been installed on the host must also be
44configured in this manner, or users connecting to the jailbox will
45discover the host environment service, unless the jailbox has
46specifically bound a service to that port.
47In some situations, this can actually be the desirable behaviour.
48.PP
49The jail environments must also be custom-configured.
50This consists of building and installing a miniature version of the
51FreeBSD file system tree off of a subdirectory in the host environment,
52usually \fC/usr/jail\fP, or \fC/data/jail\fP, with a subdirectory per jail.
53Appropriate instructions for generating this tree are included in the
54jail(8) man page, but generally this process may be automated using the
55FreeBSD build environment.
56.PP
57One notable difference from the default FreeBSD install is that only
58a limited set of device nodes should be created.
59MAKEDEV(8) has been modified to accept a ``jail'' argument that creates
60the correct set of nodes.
61.PP
62To improve storage efficiency, a fair number of the binaries in the system tree
63may be deleted, as they are not relevant in a jail environment.
64This includes the kernel, boot loader, and related files, as well as
65hardware and network configuration tools.
66.PP
67After the creation of the jail tree, the easiest way to configure it is
68to start up the jail in single-user mode.
69The sysinstall admin tool may be used to help with the task, although
70it is not installed by default as part of the system tree.
71These tools should be run in the jail environment, or they will affect
72the host environment's configuration.
73.DS
74.ft C
75.ps -2
76# mkdir /data/jail/192.168.11.100/stand
77# cp /stand/sysinstall /data/jail/192.168.11.100/stand
78# jail /data/jail/192.168.11.100 testhostname 192.168.11.100 \e
79 /bin/sh
80.ps +2
81.R
82.DE
83.PP
84After running the jail command, the shell is now within the jail environment,
85and all further commands
86will be limited to the scope of the jail until the shell exits.
87If the network alias has not yet been configured, then the jail will be
88unable to access the network.
89.PP
90The startup configuration of the jail environment may be configured so
91as to quell warnings from services that cannot run in the jail.
92Also, any per-system configuration required for a normal FreeBSD system
93is also required for each jailbox.
94Typically, this includes:
95.IP "" 5n
96\(bu Create empty /etc/fstab
97.IP
98\(bu Disable portmapper
99.IP
100\(bu Run newaliases
101.IP
102\(bu Disabling interface configuration
103.IP
104\(bu Configure the resolver
105.IP
106\(bu Set root password
107.IP
108\(bu Set timezone
109.IP
110\(bu Add any local accounts
111.IP
112\(bu Install any packets
113.NH 2
114Starting Jails
115.PP
116Jails are typically started by executing their /etc/rc script in much
117the same manner a shell was started in the previous section.
118Before starting the jail, any relevant networking configuration
119should also be performed.
120Typically, this involves adding an additional IP address to the
121appropriate network interface, setting network properties for the
122IP address using IP filtering, forwarding, and bandwidth shaping,
123and mounting a process file system for the jail, if the ability to
124debug processes from within the jail is desired.
125.DS
126.ft C
127.ps -2
128# ifconfig ed0 inet add 192.168.11.100 netmask 255.255.255.255
129# mount -t procfs proc /data/jail/192.168.11.100/proc
130# jail /data/jail/192.168.11.100 testhostname 192.168.11.100 \e
131 /bin/sh /etc/rc
132.ps +2
133.ft P
134.DE
135.PP
136A few warnings are generated for sysctl's that are not permitted
137to be set within the jail, but the end result is a set of processes
138in an isolated process environment, bound to a single IP address.
139Normal procedures for accessing a FreeBSD machine apply: telneting in
140through the network reveals a telnet prompt, login, and shell.
141.DS
142.ft C
143.ps -2
144% ps ax
145 PID TT STAT TIME COMMAND
146 228 ?? SsJ 0:18.73 syslogd
147 247 ?? IsJ 0:00.05 inetd -wW
148 249 ?? IsJ 0:28.43 cron
149 252 ?? SsJ 0:30.46 sendmail: accepting connections on port 25
150 291 ?? IsJ 0:38.53 /usr/local/sbin/sshd
15193694 ?? SJ 0:01.01 sshd: rwatson@ttyp0 (sshd)
15293695 p0 SsJ 0:00.06 -csh (csh)
15393700 p0 R+J 0:00.00 ps ax
154.ps +2
155.ft P
156.DE
157.PP
158It is immediately obvious that the environment is within a jailbox: there
159is no init process, no kernel daemons, and a J flag is present beside all
160processes indicating the presence of a jail.
161.PP
162As with any FreeBSD system, accounts may be created and deleted,
163mail is delivered, logs are generated, packages may be added, and the
164system may be hacked into if configured incorrectly, or running a buggy
165version of a piece of software.
166However, all of this happens strictly within the scope of the jail.
167.NH 2
168Jail Management
169.PP
170Jail management is an interesting prospect, as there are two perspectives
171from which a jail environment may be administered: from within the jail,
172and from the host environment.
173From within the jail, as described above, the process is remarkably similar
174to any regular FreeBSD install, although certain actions are prohibited,
175such as mounting file systems, modifying system kernel properties, etc.
176The only area that really differs are that of shutting
177the system down: the processes within the jail may deliver signals
178between them, allowing all processes to be killed, but bringing the
179system back up requires intervention from outside of the jailbox.
180.PP
181From outside of the jail, there are a range of capabilities, as well
182as limitations.
183The jail environment is, in effect, a subset of the host environment:
184the jail file system appears as part of the host file system, and may
185be directly modified by processes in the host environment.
186Processes within the jail appear in the process listing of the host,
187and may likewise be signalled or debugged.
188The host process file system makes the hostname of the jail environment
189accessible in /proc/procnum/status, allowing utilities in the host
190environment to manage processes based on jailname.
191However, the default configuration allows privileged processes within
192jails to set the hostname of the jail, which makes the status file less
193useful from a management perspective if the contents of the jail are
194malicious.
195To prevent a jail from changing its hostname, the
196"security.jail.set_hostname_allowed" sysctl may be set to 0 prior to
197starting any jails.
198.PP
199One aspect immediately observable in an environment with multiple jails
200is that uids and gids are local to each jail environment: the uid associated
201with a process in one jail may be for a different user than in another
202jail.
203This collision of identifiers is only visible in the host environment,
204as normally processes from one jail are never visible in an environment
205with another scope for user/uid and group/gid mapping.
206Managers in the host environment should understand these scoping issues,
207or confusion and unintended consequences may result.
208.PP
209Jailed processes are subject to the normal restrictions present for
210any processes, including resource limits, and limits placed by the network
211code, including firewall rules.
212By specifying firewall rules for the IP address bound to a jail, it is
213possible to place connectivity and bandwidth limitations on individual
214jails, restricting services that may be consumed or offered.
215.PP
216Management of jails is an area that will see further improvement in
217future versions of FreeBSD. Some of these potential improvements are
218discussed later in this paper.