.Dd February 22, 1999 .Dt FETCH 1 .Os FreeBSD 4.0 .Sh NAME .Nm fetch .Nd retrieve a file by Uniform Resource Locator .Sh SYNOPSIS .Nm fetch .Op Fl AMPablmnpqrtv .Op Fl S Ar size .Op Fl T Ar timeout .Op Fl o Ar file .Ar URL .Op Ar ... .Nm fetch .Op Fl MPRlmnpqrv .Op Fl S Ar size .Op Fl o Ar file .Op Fl c Ar dir .Fl f Ar file .Fl h Ar host .Sh DESCRIPTION .Nm fetch allows a user to transfer files from a remote network site using either the .Tn FTP or the .Tn HTTP protocol. In the first form of the command, the .Ar URL may be of the form .Li http://site.domain/path/to/the/file or .Li ftp://site.domain/path/to/the/file . To denote a local filename to be copied or linked to (see the .Fl l flag below), the .Em file:/path/to/the/file URL form is used. See URL SYNTAX, below.
p The second form of the command can be used to get a file using the .Tn FTP protocol, specifying the file name and the remote host with the .Fl h and the .Fl f flags.
p The following options are available: l -tag -width Fl t Fl A Do not automatically follow ``temporary'' (302) redirects. Some broken Web sites will return a redirect instead of a not-found error when the requested object does not exist. t Fl a Automatically retry the transfer upon soft failures. t Fl b Work around a bug in some .Tn HTTP servers which fail to correctly implement the .Tn TCP protocol. t Fl c Ar dir The file to retrieve is in directory .Ar dir on the remote host. t Fl f Ar file The file to retrieve is named .Ar file on the remote host. t Fl h Ar host The file to retrieve is located on the host .Ar host . t Fl l If target is a .Ar file:/ style of URL, make a link to the target rather than trying to copy it. t Fl M t Fl m Mirror mode: Set the modification time of the file so that it is identical to the modification time of the file at the remote host. If the file already exists on the local host and is identical (as gauged by size and modification time), no transfer is done. t Fl n Don't preserve the modtime of the transferred file, use the current time. t Fl o Ar file Set the output file name to .Ar file . By default, a ``pathname'' is extracted from the specified URI, and its basename is used as the name of the output file. A .Ar file argument of .Sq Li - indicates that results are to be directed to the standard output. t Fl P t Fl p Use the passive mode of the .Tn FTP protocol. This is useful for crossing certain sorts of firewalls. t Fl q Quiet mode. Do not report transfer progress on the terminal. t Fl R The filenames specified are ``precious'', and should not be deleted under any circumstances, even if the transfer failed or was incomplete. t Fl r Restart a previously interrupted transfer. t Fl S Ar bytes Require the file size reported by .Tn FTP or .Tn HTTP server to match the value specified with this option. On mismatch, a message is printed and the file will not be fetched. If the server does not support reporting of file sizes, the option will be ignored and the file will be retrieved anyway. This option is useful to prevent .Nm fetch from downloading a file that is either incomplete or the wrong version, given the correct size of the file in advance. t Fl s Ask server for size of file in bytes and print it to stdout. Do not actually fetch the file. t Fl T Ar seconds Set timeout value to .Ar seconds. Overrides the environment variables .Ev FTP_TIMEOUT for ftp transfers or .Ev HTTP_TIMEOUT for http transfers if set. t Fl t Work around a different set of buggy .Tn TCP implementations. t Fl v Increase verbosity. More .Fl v Ns 's result in more information. .El
p Many options are also controlled solely by the environment (this is a bug). .Sh URL SYNTAX .Nm accepts .Tn http and .Tn ftp URL's, as described in RFC1738. For .Tn ftp URL's, a username and password may be specified, using the syntax .Li ftp://user:password@host/. If the path is to be absolute, as opposed to relative to the user's home directory, it must start with %2F, as in .Li ftp://root:mypass@localhost/%2Fetc/passwd . .Nm Fetch condenses multiple slashes in an .Tn ftp URL into a single slash; literal multiple slashes translate to an .Tn ftp protocol error. .Sh PROXY SERVERS Many sites use application gateways (``proxy servers'') in their firewalls in order to allow communication across the firewall using a trusted protocol. The .Nm fetch program can use both the .Tn FTP and the .Tn HTTP protocol with a proxy server. .Tn FTP proxy servers can only relay .Tn FTP requests; .Tn HTTP proxy servers can relay both .Tn FTP and .Tn HTTP requests. A proxy server can be configured by defining an environment variable named .Dq Va PROTO Ns Ev _PROXY , where .Va PROTO is the name of the protocol in upper case. The value of the environment variable specifies a hostname, optionally followed by a colon and a port number.
p The .Tn FTP proxy client passes the remote username, host and port as the .Tn FTP session's username, in the form .Do Va remoteuser Ns Li @ Ns Va remotehost .Op Li @ Ns Va port .Dc . The .Tn HTTP proxy client simply passes the originally-requested URI to the remote server in an .Tn HTTP .Dq Li GET request. HTTP proxy authentication is not yet implemented. .Sh HTTP AUTHENTICATION The .Tn HTTP protocol includes support for various methods of authentication. Currently, the .Dq basic method, which provides no security from packet-sniffing or man-in-the-middle attacks, is the only method supported in .Nm fetch . Authentication is enabled by the .Ev HTTP_AUTH and .Ev HTTP_PROXY_AUTH environment variables. Both variables have the same format, which consists of space-separated list of parameter settings, where each setting consists of a colon-separated list of parameters. The first two parameters are always the (case-insensitive) authentication scheme name and the realm in which authentication is to be performed. If the realm is specified as .Sq Li * , then it will match all realms not specified otherwise.
p The .Li basic authentication scheme uses two additional optional parameters; the first is a user name, and the second is the password associated with it. If either the password or both parameters are not specified in the environment, and the standard input of .Nm is connected to a terminal, then .Nm will prompt the user to enter the missing parameters. Thus, if the user is known as .Dq Li jane in the .Dq Li WallyWorld realm, and has a password of .Dq Li QghiLx79 there, then she might set her .Ev HTTP_AUTH variable to: l -enum -offset indent t .Dq Li basic:WallyWorld:jane:QghiLx79 t .Dq Li basic:WallyWorld:jane , or t .Dq Li basic:WallyWorld .El
p and .Nm will prompt for any missing information when it is required. She might also specify a realm of .Dq Li * instead of .Dq Li WallyWorld to indicate that the parameters can be applied to any realm. (This is most commonly used in a construction such as .Dq Li basic:* , which indicates to .Nm that it may offer to do .Li basic authentication for any realm. .Sh ERRORS The .Nm command returns zero on success, or a non-zero value from .Aq Pa sysexits.h on failure. If multiple URIs are given for retrieval, .Nm will attempt all of them and return zero only if all succeeded (otherwise it will return the error from the last failure). .Sh ENVIRONMENT l -tag -width FTP_PASSIVE_MODE -offset indent t Ev FTP_TIMEOUT maximum time, in seconds, to wait before aborting an .Tn FTP connection. t Ev FTP_LOGIN the login name used for .Tn FTP transfers (default .Dq Li anonymous ) t Ev FTP_PASSIVE_MODE force the use of passive mode FTP t Ev FTP_PASSWORD the password used for .Tn FTP transfers (default .Dq Va yourname Ns Li @ Ns Va yourhost ) t Ev FTP_PROXY the address (in the form .Do Va hostname Ns .Op Li : Ns Va port .Dc ) of a proxy server which understands .Tn FTP t Ev HTTP_AUTH defines authentication parameters for .Tn HTTP t Ev HTTP_PROXY the address (in the form .Do Va hostname Ns .Op Li : Ns Va port .Dc ) of a proxy server which understands .Tn HTTP t Ev HTTP_PROXY_AUTH defines authentication parameters for .Tn HTTP proxy servers t Ev HTTP_TIMEOUT maximum time, in seconds, to wait before aborting an .Tn HTTP connection. .Sh SEE ALSO .Xr ftp 1 , .Xr tftp 1 .Rs .%A R. Fielding .%A J. Gettys .%A J. Mogul .%A H. Frystyk .%A T. Berners-Lee .%T "Hypertext Transfer Protocol -- HTTP/1.1" .%O RFC 2068 .%D January 1997 .Re .Rs .%A T. Berners-Lee .%A L. Masinter .%A M. McCahill .%T "Uniform Resource Locators (URL)" .%O RFC 1738 .%D December 1994 .Re .Rs .%A J. Postel .%A J.K. Reynolds .%T "File Transfer Protocol" .%O RFC 959 / STD 9 .%D October 1985 .Re .Rs .%A M.R. Horton .%T "Standard for interchange of USENET messages." .%O RFC 850 .%D June 1983 .Re .Sh HISTORY The .Nm fetch command appeared in .Fx 2.1.5 . .Sh AUTHORS The original implementation of .Nm was done by .An Jean-Marc Zucconi . It was extensively re-worked for .Fx 2.2 by .An Garrett Wollman . .Sh BUGS There are too many environment variables and command-line options.
p The .Fl a option is only implemented for certain kinds of .Tn HTTP failures, and no .Tn FTP failures.
p Only the .Dq basic authentication mode is implemented for .Tn HTTP . This should be replaced by digest authentication.
p Some .Tn TCP implementations (other than .Tn FreeBSD ) fail to correctly implement cases where the .Dv SYN and/or .Dv FIN control flags are specified in packets which also contain data. The .Sq Fl t flag works around the latter deficiency and the .Sq Fl b flag works around the former. Since these are errors of the server's .Tn TCP stack, the best we can do is provide these workarounds. Given a correct server, an optimal .Tn HTTP transfer without .Fl t and .Fl b involves a minimum of two round trips (for small replies), one less than other implementations.
p The .Tn HTTP standard requires interpretation of the .Tn RFC 850 date format, which does not provide a century indication. Versions of .Nm fetch prior to .Fx 3.1 would interpret all such dates as being in the 1900s. This version of .Nm fetch interprets such dates according to the rule given in .Tn RFC 2068 : d -literal -offset indent o HTTP/1.1 clients and caches should assume that an RFC-850 date which appears to be more than 50 years in the future is in fact in the past (this helps solve the "year 2000" problem). .Ed