1.\" $FreeBSD: head/usr.bin/fetch/fetch.1 57670 2000-03-01 12:20:22Z sheldonh $
2.Dd February 22, 1999
| 1.\"-
2.\" Copyright (c) 2000 Dag-Erling Co�dan Sm�rgrav
3.\" All rights reserved.
4.\"
5.\" Redistribution and use in source and binary forms, with or without
6.\" modification, are permitted provided that the following conditions
7.\" are met:
8.\" 1. Redistributions of source code must retain the above copyright
9.\" notice, this list of conditions and the following disclaimer
10.\" in this position and unchanged.
11.\" 2. Redistributions in binary form must reproduce the above copyright
12.\" notice, this list of conditions and the following disclaimer in the
13.\" documentation and/or other materials provided with the distribution.
14.\" 3. The name of the author may not be used to endorse or promote products
15.\" derived from this software without specific prior written permission.
16.\"
17.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
18.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
19.\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
20.\" IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
21.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
22.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
23.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
24.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
25.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
26.\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
27.\"
28.\" $FreeBSD: head/usr.bin/fetch/fetch.1 62216 2000-06-28 16:55:15Z des $
29.\"
30.Dd June 28, 2000
|
3.Dt FETCH 1
| 31.Dt FETCH 1
|
4.Os FreeBSD 4.0
| 32.Os
|
5.Sh NAME
6.Nm fetch
7.Nd retrieve a file by Uniform Resource Locator
8.Sh SYNOPSIS
| 33.Sh NAME
34.Nm fetch
35.Nd retrieve a file by Uniform Resource Locator
36.Sh SYNOPSIS
|
9.Nm fetch
10.Op Fl AFMPablmnpqrtv
11.Op Fl S Ar size
12.Op Fl T Ar timeout
| 37.Nm
38.Op Fl 146AFHMPRalmnpqrsv
39.Op Fl B Ar bytes
40.Op Fl S Ar bytes
41.Op Fl T Ar seconds
|
13.Op Fl o Ar file
| 42.Op Fl o Ar file
|
| 43.Op Fl w Ar seconds
|
14.Ar URL
15.Op Ar ...
| 44.Ar URL
45.Op Ar ...
|
16.Nm fetch
17.Op Fl FMPRlmnpqrv
18.Op Fl S Ar size
19.Op Fl o Ar file
20.Op Fl c Ar dir
21.Fl f Ar file
22.Fl h Ar host
| |
23.Sh DESCRIPTION
| 46.Sh DESCRIPTION
|
24.Nm fetch
25allows a user to transfer files from a remote network site using
26either the
27.Tn FTP
28or the
29.Tn HTTP
30protocol.
31In the first form of the command, the
32.Ar URL
33may be of the form
34.Li http://site.domain/path/to/the/file
35or
36.Li ftp://site.domain/path/to/the/file .
37To denote a local filename to be copied or linked to (see the
38.Fl l
39flag below), the
40.Em file:/path/to/the/file
41URL form is used. See URL SYNTAX, below.
| 47.Nm Fetch
48provides a command-line interface to the
49.Xr fetch 3
50library.
51Its purpose is to retrieve the file(s) pointed to by the URL(s) on the
52command line.
|
42.Pp
| 53.Pp
|
43The second form of the command can be used to get a file using the
44.Tn FTP
45protocol, specifying the file name and the remote host with the
46.Fl h
47and the
48.Fl f
49flags.
50.Pp
| |
51The following options are available:
52.Bl -tag -width Fl
| 54The following options are available:
55.Bl -tag -width Fl
|
| 56.It Fl \&1
57Stop and return exit code 0 at the first successfully retrieved file.
58.It Fl 4
59Forces
60.Nm
61to use IPv4 addresses only.
62.It Fl 6
63Forces
64.Nm
65to use IPv6 addresses only.
|
53.It Fl A
| 66.It Fl A
|
54Do not automatically follow ``temporary'' (302) redirects. Some
55broken Web sites will return a redirect instead of a not-found error
56when the requested object does not exist.
| 67Do not automatically follow ``temporary'' (302) redirects.
68Some broken Web sites will return a redirect instead of a not-found
69error when the requested object does not exist.
|
57.It Fl a
58Automatically retry the transfer upon soft failures.
| 70.It Fl a
71Automatically retry the transfer upon soft failures.
|
59.It Fl b
60Work around a bug in some
61.Tn HTTP
62servers which fail to correctly implement the
63.Tn TCP
64protocol.
| 72.It Fl B Ar bytes
73Specify the read buffer size in bytes.
74The default is 4096 bytes.
75Attempts to set a buffer size lower than this will be silently
76ignored.
77The number of reads actually performed is reported at verbosity level
78two or higher (see the
79.Fl v
80flag).
|
65.It Fl c Ar dir
66The file to retrieve is in directory
67.Ar dir
68on the remote host.
69.It Fl F
| 81.It Fl c Ar dir
82The file to retrieve is in directory
83.Ar dir
84on the remote host.
85.It Fl F
|
70Force restart without checking for the local file's date matching
71that of the remote file. Use this with
| 86In combination with the
|
72.Fl r
| 87.Fl r
|
73to restart a transfer of a partial file where the modification
74time on the local file has been changed and you are sure that the
75remote file is still the same, as this will prevent retrieval from
76starting anew.
| 88flag, forces a restart even if the local and remote files have
89different modification times.
|
77.It Fl f Ar file
78The file to retrieve is named
79.Ar file
80on the remote host.
| 90.It Fl f Ar file
91The file to retrieve is named
92.Ar file
93on the remote host.
|
| 94.It Fl H
95When using passive FTP, allocate a high port for the data connection.
96See
97.Xr ip 4
98for details on how to specify which port range this corresponds to.
|
81.It Fl h Ar host
82The file to retrieve is located on the host
83.Ar host .
84.It Fl l
| 99.It Fl h Ar host
100The file to retrieve is located on the host
101.Ar host .
102.It Fl l
|
85If target is a
86.Ar file:/
87style of URL, make a link to the target rather than trying
88to copy it.
| 103If the target is a file-scheme URL, make a symbolic link to the target
104rather than trying to copy it.
|
89.It Fl M
90.It Fl m
| 105.It Fl M
106.It Fl m
|
91Mirror mode: Set the modification time of the file so that it is
92identical to the modification time of the file at the remote host.
93If the file already exists on the local host and is identical (as
94gauged by size and modification time), no transfer is done.
| 107Mirror mode: set the modification time of the local file to that of
108the remote file.
109If the file already exists locally and has the same size and
110modification time as the remote file, it will not be fetched.
|
95.It Fl n
| 111.It Fl n
|
96Don't preserve the modtime of the transferred file, use the current time.
| 112Don't preserve the modtime of the transfered file, use the current
113time.
|
97.It Fl o Ar file
98Set the output file name to
99.Ar file .
100By default, a ``pathname'' is extracted from the specified URI, and
| 114.It Fl o Ar file
115Set the output file name to
116.Ar file .
117By default, a ``pathname'' is extracted from the specified URI, and
|
101its basename is used as the name of the output file. A
| 118its basename is used as the name of the output file.
119A
|
102.Ar file
103argument of
104.Sq Li \&-
105indicates that results are to be directed to the standard output.
106.It Fl P
107.It Fl p
| 120.Ar file
121argument of
122.Sq Li \&-
123indicates that results are to be directed to the standard output.
124.It Fl P
125.It Fl p
|
108Use the passive mode of the
109.Tn FTP
110protocol. This is useful for crossing certain sorts of firewalls.
| 126Use passive FTP.
127This is useful if you are behind a firewall which blocks incoming
128connections.
129Try this flag if
130.Nm
131seems to hang when retrieving FTP URLs.
|
111.It Fl q
112Quiet mode.
| 132.It Fl q
133Quiet mode.
|
113Do not report transfer progress on the terminal.
| 134This works by setting the verbosity level to zero; see the
135.Fl v
136option.
|
114.It Fl R
| 137.It Fl R
|
115The filenames specified are ``precious'', and should not be deleted
116under any circumstances, even if the transfer failed or was incomplete.
| 138The output files are precious, and should not be deleted under any
139circumstances, even if the transfer failed or was incomplete.
|
117.It Fl r
118Restart a previously interrupted transfer.
119.It Fl S Ar bytes
| 140.It Fl r
141Restart a previously interrupted transfer.
142.It Fl S Ar bytes
|
120Require the file size reported by
121.Tn FTP
122or
123.Tn HTTP
124server to match the value specified with this option.
125On mismatch, a message is printed and the file will not be fetched.
126If the server does not support reporting of file sizes, the option
127will be ignored and the file will be retrieved anyway.
128This option is useful to prevent
129.Nm fetch
130from downloading a file that is either incomplete or the wrong version,
131given the correct size of the file in advance.
| 143Require the file size reported by the server to match the specified
144value.
145If it does not, a message is printed and the file is not fetched.
146If the server does not support reporting file sizes, this option is
147ignored and the file is fetched unconditionally.
|
132.It Fl s
| 148.It Fl s
|
133Ask server for size of file in bytes and print it to stdout.
134Do not
135actually fetch the file.
| 149Print the size in bytes of each requested file, without fetching it.
|
136.It Fl T Ar seconds
137Set timeout value to
138.Ar seconds.
139Overrides the environment variables
140.Ev FTP_TIMEOUT
| 150.It Fl T Ar seconds
151Set timeout value to
152.Ar seconds.
153Overrides the environment variables
154.Ev FTP_TIMEOUT
|
141for ftp transfers or
| 155for FTP transfers or
|
142.Ev HTTP_TIMEOUT
| 156.Ev HTTP_TIMEOUT
|
143for http transfers if set.
| 157for HTTP transfers if set.
|
144.It Fl t
145Work around a different set of buggy
146.Tn TCP
147implementations.
148.It Fl v
| 158.It Fl t
159Work around a different set of buggy
160.Tn TCP
161implementations.
162.It Fl v
|
149Increase verbosity. More
150.Fl v Ns \&'s
151result in more information.
| 163Each instance of this flag increases the verbosity level by one.
164Level one (the default) only gives a summary after each file; level
165two show a running count during the transfer, provided that the
166standard output goes to a terminal; level three enables messages from
167the
168.Xr fetch 3
169library.
170.It Fl w Ar seconds
171When the
172.Fl a
173flag is specified, wait this many seconds between successive retries.
|
152.El
| 174.El
|
153.Pp
154Many options are also controlled solely by the environment (this is a
155bug).
156.Sh URL SYNTAX
157.Nm
158accepts
159.Tn http
160and
161.Tn ftp
162URL's, as described in RFC1738. For
163.Tn ftp
164URL's, a username and password may be specified, using the syntax
165.Li ftp://user:password@host/.
166If the path is to be absolute, as opposed to relative to the user's
167home directory, it must start with %2F, as in
168.Li ftp://root:mypass@localhost/%2Fetc/passwd .
169.Nm Fetch
170condenses multiple slashes in an
171.Tn ftp
172URL into a single slash; literal multiple slashes translate to an
173.Tn ftp
174protocol error.
175.Sh PROXY SERVERS
176Many sites use application gateways (``proxy servers'') in their
177firewalls in order to allow communication across the firewall using a
178trusted protocol. The
179.Nm fetch
180program can use both the
181.Tn FTP
182and the
183.Tn HTTP
184protocol with a proxy server.
185.Tn FTP
186proxy servers can only relay
187.Tn FTP
188requests;
189.Tn HTTP
190proxy servers can relay both
191.Tn FTP
192and
193.Tn HTTP
194requests.
195A proxy server can be configured by defining an environment variable
196named
197.Dq Va PROTO Ns Ev _PROXY ,
198where
199.Va PROTO
200is the name of the protocol in upper case. The value of the
201environment variable specifies a hostname, optionally followed by a
202colon and a port number.
203.Pp
| 175.Sh DIAGNOSTICS
|
204The
| 176The
|
205.Tn FTP
206proxy client passes the remote username, host and port as the
207.Tn FTP
208session's username, in the form
209.Do Va remoteuser Ns Li \&@ Ns Va remotehost
210.Op Li \&@ Ns Va port
211.Dc .
212The
213.Tn HTTP
214proxy client simply passes the originally-requested URI to the remote
215server in an
216.Tn HTTP
217.Dq Li GET
218request. HTTP proxy authentication is not yet implemented.
219.Sh HTTP AUTHENTICATION
220The
221.Tn HTTP
222protocol includes support for various methods of authentication.
223Currently, the
224.Dq basic
225method, which provides no security from packet-sniffing or
226man-in-the-middle attacks, is the only method supported in
227.Nm fetch .
228Authentication is enabled by the
229.Ev HTTP_AUTH
230and
231.Ev HTTP_PROXY_AUTH
232environment variables. Both variables have the same format, which
233consists of space-separated list of parameter settings, where each
234setting consists of a colon-separated list of parameters. The first
235two parameters are always the (case-insensitive) authentication scheme
236name and the realm in which authentication is to be performed. If the
237realm is specified as
238.Sq Li \&* ,
239then it will match all realms not specified otherwise.
240.Pp
241The
242.Li basic
243authentication scheme uses two additional optional parameters; the
244first is a user name, and the second is the password associated with
245it. If either the password or both parameters are not specified in
246the environment, and the standard input of
| |
247.Nm
| 177.Nm
|
248is connected to a terminal, then
| 178command returns zero on success, or one on failure.
179If multiple URLs are listed on the command line,
|
249.Nm
| 180.Nm
|
250will prompt the user to enter the missing parameters. Thus, if the
251user is known as
252.Dq Li jane
253in the
254.Dq Li WallyWorld
255realm, and has a password of
256.Dq Li QghiLx79
257there, then she might set her
258.Ev HTTP_AUTH
259variable to:
260.Bl -enum -offset indent
261.It
262.Dq Li basic:WallyWorld:jane:QghiLx79
263.It
264.Dq Li basic:WallyWorld:jane ,
265or
266.It
267.Dq Li basic:WallyWorld
268.El
269.Pp
270and
271.Nm
272will prompt for any missing information when it is required. She might
273also specify a realm of
274.Dq Li \&*
275instead of
276.Dq Li WallyWorld
277to indicate that the parameters can be applied to any realm. (This is
278most commonly used in a construction such as
279.Dq Li basic:* ,
280which indicates to
281.Nm
282that it may offer to do
283.Li basic
284authentication for any realm.
285.Sh ERRORS
286The
287.Nm
288command returns zero on success, or a non-zero value from
289.Aq Pa sysexits.h
290on failure. If multiple URIs are given for retrieval,
291.Nm
292will attempt all of them and return zero only if all succeeded
293(otherwise it will return the error from the last failure).
| 181will attempt to retrieve them each of them in turn, and return zero
182only if they were all successfully retrieved.
|
294.Sh ENVIRONMENT
295.Bl -tag -width FTP_PASSIVE_MODE -offset indent
296.It Ev FTP_TIMEOUT
297maximum time, in seconds, to wait before aborting an
298.Tn FTP
299connection.
| 183.Sh ENVIRONMENT
184.Bl -tag -width FTP_PASSIVE_MODE -offset indent
185.It Ev FTP_TIMEOUT
186maximum time, in seconds, to wait before aborting an
187.Tn FTP
188connection.
|
300.It Ev FTP_LOGIN
301the login name used for
302.Tn FTP
303transfers (default
304.Dq Li anonymous )
305.It Ev FTP_PASSIVE_MODE
306force the use of passive mode FTP
307.It Ev FTP_PASSWORD
308the password used for
309.Tn FTP
310transfers (default
311.Dq Va yourname Ns Li \&@ Ns Va yourhost )
312.It Ev FTP_PROXY
313the address (in the form
314.Do Va hostname Ns
315.Op Li : Ns Va port
316.Dc )
317of a proxy server which understands
318.Tn FTP
319.It Ev HTTP_AUTH
320defines authentication parameters for
321.Tn HTTP
322.It Ev HTTP_PROXY
323the address (in the form
324.Do Va hostname Ns
325.Op Li : Ns Va port
326.Dc )
327of a proxy server which understands
328.Tn HTTP
329.It Ev HTTP_PROXY_AUTH
330defines authentication parameters for
331.Tn HTTP
332proxy servers
| |
333.It Ev HTTP_TIMEOUT
334maximum time, in seconds, to wait before aborting an
335.Tn HTTP
336connection.
| 189.It Ev HTTP_TIMEOUT
190maximum time, in seconds, to wait before aborting an
191.Tn HTTP
192connection.
|
| 193.El
194.Pp
195All environment variables mentioned in the documentation for the
196.Xr fetch 3
197library are supported.
|
337.Sh SEE ALSO
| 198.Sh SEE ALSO
|
338.Xr ftp 1 ,
339.Xr tftp 1
340.Rs
341.%A R. Fielding
342.%A J. Gettys
343.%A J. Mogul
344.%A H. Frystyk
345.%A T. Berners-Lee
346.%T "Hypertext Transfer Protocol \-\- HTTP/1.1"
347.%O RFC 2068
348.%D January 1997
349.Re
350.Rs
351.%A T. Berners-Lee
352.%A L. Masinter
353.%A M. McCahill
354.%T "Uniform Resource Locators (URL)"
355.%O RFC 1738
356.%D December 1994
357.Re
358.Rs
359.%A J. Postel
360.%A J.K. Reynolds
361.%T "File Transfer Protocol"
362.%O RFC 959 / STD 9
363.%D October 1985
364.Re
365.Rs
366.%A M.R. Horton
367.%T "Standard for interchange of USENET messages."
368.%O RFC 850
369.%D June 1983
370.Re
| 199.Xr fetch 3
|
371.Sh HISTORY
372The
| 200.Sh HISTORY
201The
|
373.Nm fetch
| 202.Nm
|
374command appeared in
375.Fx 2.1.5 .
| 203command appeared in
204.Fx 2.1.5 .
|
| 205This implementation first appeared in
206.Fx 4.1 .
|
376.Sh AUTHORS
377The original implementation of
378.Nm
379was done by
380.An Jean-Marc Zucconi .
381It was extensively re-worked for
382.Fx 2.2
383by
| 207.Sh AUTHORS
208The original implementation of
209.Nm
210was done by
211.An Jean-Marc Zucconi .
212It was extensively re-worked for
213.Fx 2.2
214by
|
384.An Garrett Wollman .
385.Sh BUGS
386There are too many environment variables and command-line options.
387.Pp
| 215.An Garrett Wollman ,
216and later completely rewritten to use the
217.Xr fetch 3
218library by
219.An Dag-Erling Sm�rgrav .
220.Sh NOTES
|
388The
| 221The
|
389.Fl a
390option is only implemented for certain kinds of
391.Tn HTTP
392failures, and no
393.Tn FTP
394failures.
395.Pp
396Only the
397.Dq basic
398authentication mode is implemented for
399.Tn HTTP .
400This should be replaced by digest authentication.
401.Pp
402Some
403.Tn TCP
404implementations (other than
405.Tn FreeBSD )
406fail to correctly implement cases where the
407.Dv SYN
408and/or
409.Dv FIN
410control flags are specified in packets which also contain data.
411The
412.Sq Fl t
413flag works around the latter deficiency and the
414.Sq Fl b
415flag works around the former. Since these are errors of the server's
416.Tn TCP
417stack, the best we can do is provide these workarounds. Given a correct
418server, an optimal
419.Tn HTTP
420transfer without
421.Fl t
422and
| |
423.Fl b
| 222.Fl b
|
424involves a minimum of two round trips (for small replies), one less than
425other implementations.
| 223and
224.Fl t
225options are no longer supported and will generate warnings.
226They were workarounds for bugs in other OSes which this implementation
227does not trigger.
|
426.Pp
427The
| 228.Pp
229The
|
428.Tn HTTP
429standard requires interpretation of the
430.Tn RFC 850
431date format, which does not provide a century indication. Versions of
432.Nm fetch
433prior to
434.Fx 3.1
435would interpret all such dates as being in the 1900s. This version of
436.Nm fetch
437interprets such dates according to the rule given in
438.Tn RFC 2068 :
439.Bd -literal -offset indent
440 o HTTP/1.1 clients and caches should assume that an RFC-850 date
441 which appears to be more than 50 years in the future is in fact
442 in the past (this helps solve the "year 2000" problem).
443.Ed
| 230.Fl f
231and
232.Fl h
233options (used for specifying an file to fetch and a host to fetch
234from) are no longer supported and will generate errors.
235Use URLs.
236RFC1738 is your friend.
237.Xr fetch 3
238library.
|
| |