Copy stable/10@r272459 to releng/10.1 as part of
the 10.1-RELEASE process.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFH (r261233): fix whitespace and bump copyright
MFH (r261234): increase buffer size

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Include an Accept header in requests.

PR: kern/180917
MFC after: 1 week

Add Michael Gmelin's name to the copyright statement.

Remove EOL whitespace.

Implement certificate verification, and many other SSL-related
imrovements; complete details in the PR.

PR: kern/175514
Submitted by: Michael Gmelin <freebsd@grem.de>
MFC after: 1 week

mdoc nit

Revert r232274 - unauthorized, unnecessary and incorrect.

General mdoc(7) and typo fixes.

PR: 167696
Submitted by: Nobuyuki Koganemaru (kogane!jp.freebsd.org)
MFC after: 3 days

Document the 'URL' argument.

latin1 -> utf8

Document the fact that passive mode is now the default.
Update copyright dates and strip my middle name.

Add support for HTTP 1.1 If-Modified-Since behavior.

fetch(1) accepts a new argument -i <file> that if specified will cause
the file to be downloaded only if it is more recent than the mtime of
<file>.

libfetch(3) accepts the mtime in the url structure and a flag to
indicate when this behavior is desired.

PR: bin/87841
Submitted by: Jukka A. Ukkonen <jau@iki.fi> (partially)
Reviewed by: des, ru
MFC after: 3 weeks

Improve usage and sync SYNOPSIS with usage.

List all environment variables supported by libfetch, along with a reference
to the fetch(3) man page.

MFC after: 1 week

Fix a typo.

Submitted by: Pawel Worach <pawel.worach at gmail dot com>
MFC after: 1 week

Expand *n't contractions.

Sort sections.

Added the EXIT STATUS section where appropriate.

Update copyright years.

Document the behaviour of the "-o file" option when "file" is a directory
(downloaded file(s) are created inside the directory).

MFC after: 3 days

Restore part of the word that got (accidentally) removed.
Removed whitespace at EOL.

Remove EXAMPLES section which duplicates text in fetch(3).

Remove the inadvertant HTML additions to this man page.
These appear to have been inserted in the PR text by gnatsweb.

Noticed by: des (maintainer)

Revert changes of rev 1.55; replace with strong encouragement to
read fetch(3) as well.

Requested by: des (maintainer)

Previous commit did the reverse of what the log message said (and of
what I intended to do). Really remove my middle name.

I don't normally use my middle name, so remove it from attributions in
man pages (though not from copyright notices). While I'm here, add email
addresses where appropriate.

Document fetch(1) environment variables in fetch(1) man page, instead of
taking lazy users on the arduous journey through SEE ALSO to fetch(3).

PR: docs/61759
Submitted by: Christian S.J. Peron <maneo@bsdpro.com>

Finish the reversion of rev. 1.52.

Revert 1.52. This should have been added to fetch.3.

Discussed with: des, ru

Add an EXAMPLES section.
Xref sh(1) as we use environment variables.

PR: 43899
Reviewed by: des

Clarify that -r implies -R.

mdoc(7) police: Scheduled sweep.

Document netrc support.

Use `The .Nm utility'

Document fetch(1) reaction to SIGINFO.

PR: docs/33640
Submitted by: Maxim Konovalov <maxim@macomnet.ru>
Approved by: des

Remove whitespace at EOL.

Now that libfetch uses the high port range by default, add a -U option to
make it use the low (default) port range instead.

Fix a formatting bug (MFC candidate)

mdoc(7) police: split punctuation characters + misc fixes.

mdoc(7) police: do not split author names in the AUTHORS section.

Spell the des's name correctly.

* Remove description of obsolete -t option.
* Insert newline before "and" in ".Fl c and".

Approved by: des

The description of the semantics of -v and -q has long since been OBE.

Document the fact that -r and -m are mutually exclusive.

Clarify the meaning of -m and -n (thanks to bde).

Add MIT copyright

Note that the -h, -c and -f options are deprecated, and remove note that
incorrectly stated that they were not implemented.
Document the -d option.

New libfetch-based fetch.

Remove single-space hard sentence breaks. These degrade the quality
of the typeset output, tend to make diffs harder to read and provide
bad examples for new-comers to mdoc.

Add the -F option. This is for forcing restarts with -r by inhibiting
transmission of the If-Range HTTP header field.

$Id$ -> $FreeBSD$

"transfered" -> "transferred"

Reviewed by: dict(1)

Deal with broken Web sites which return 302 responses rather than 404
and an error document when the requested resource does not exist. Grrr.

Requested by: asami

Warn about collapsing multiple slashes into 1 in ftp URL's.
Look at the FTP_PASSIVE_MODE environment variable like the man page says.

PR: bin/9464
Submitted by: John A. Shue <John.Shue@symmetron.com>

Add references to RFC's 1790, 959, 850.

PR: doc/6564

For RFC 850 dates received in HTTP responses, implement the century pivot
described in RFC 2068. Include a reference to same in the manual page.

Add -s option, just report size of file that would be fetched.
Reviewed by: -current list

main.c:
Fix usage string. Fix getopt() string. Fix ordering of compatibility options.
fetch.1:
Fix synopsis. Fix ordering of T and t options. Fix minor grammar nit.

Add -S flag; this is benign since fetch's behavior is unchanged if you
don't use it. That's why I'm bringing it in during our "code slush"
Submitted by: Stefan Esser <se@mi.uni-koeln.de>

Fix my last change: the .Fx macro doesn't do anything useful without an
argument, so use `.Tn FreeBSD' instead.

Documentation fix: delete references to T/TCP as they are not relevant
to the case in question. (Said case being quite clearly defined by
the basic TCP specification, RFC 792.)

Describe absolute paths in FTP URL's. This patch has been sitting
around uncommitted for 2 months =(

.Sh AUTHOR -> .Sh AUTHORS. Use .An/.Aq.

Add -t option which turns T/TCP off as workaround for some broken servers
Submitted by: Marc Slemko <marcs@znep.com>

Provide a new `-b' flag to work around some broken HTTP/TCP implementations
that can't deal with a half-closed connection.

Fix FTP_PROXY to use user@host[@port] for FTP proxy and eliminate
undocumented FTP_PROXY_USER
Make FTP file errors contian hostname and path.
Pass the FTP port to libftp.
Partially un-HTMLify error messages returned from server
Handle "HTTP NNN" instead of "HTTP/V.vv NNN" response sent by
pre-HTTP/1.0 servers

Reviewed by: wollman

These man page will appear in 2.2, not 3.0.

Revert $FreeBSD$ to $Id$

Some bug-fixes, clean-ups, and one new feature:

- Fix the bug with URIs of the form ftp://host/filename.
- Fix some more string-termination bugs in util.c.
- Use safe_malloc() rather than testing the return value of
regular malloc() in 15 places.
- Implement HTTP authentication, for both servers and proxies.
Currently only ``basic'' authentication is supported; This Is A Bug
(but less of one tjhan nmot supporting any authentication).

I think there is only one more feature which is required for full
HTTP/1.1 support, which is Transfer-Encoding: chunked; this should
not be toohard, but it isn't very important, either.

Some fixes for HTTP:

1) Implement redirects (or try to, at least).
2) Implement automatic retry after 503 errors when Retry-After is given.
3) Implement a -a flag to enable both of these behaviors.
4) Recognize Transfer-Encoding headers and emit a warning that the file
is likely to be damaged.
5) Bug fix: only write the amount of data we read.
6) Actually document some of these.
7) Fix the usage message to display flags in semi-alphabetical order.

Here is my long-threatened revamping of fetch. Jean-Marc probably won't
recognize it any more. This makes the following significant changes:

- The main body of the program doesn't know a thing about URIs,
HTTP, or FTP. This makes it possible to easily plug in other
protocols. (The next revision will probably be able to dynamically
add new recognizers.)

- There are no longer arbitrary timeouts for the protocols. If you want
to set one for yourself, use the environment variables.

- FTP proxies are now supported (if I implemented it right).

- The HTTP implementation is much more complete, and can now do restarts,
preserve modtimes, and mrun in mirror mode. It's not yet up to 1.1,
but it's getting there.

- Transaction TCP is now used for sending HTTP requests. The HTTP/1.1 syntax
for requesting that the connection be closed after one request is
implemented.

In all of this, I have doubtless broken somebody. Please test it and tell me
about the bugs.

Sort cross references.

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

add missing comma(s) in .Xr macros

Attempt to untangle the timeout code a bit, also make the default ftp
and http timeouts the same, since when using a http proxy to do ftp
transfers, the http timeout was being used for what is coming in via
ftp.

Allow proper ftp verbosity with a new -v flag.

Use the .Fx macro where appropriate.

1. Understand file:/path style URLs.
2. Add a -l flag for symlinking to rather than copying file:/path style URLs.
3. Add a -T flag for setting the timeout interval (overrides FTP_TIMEOUT if set)

Add support for HTTP proxies, fix some bugs with http transfers.
Submitted-By: Mikael Hybsch <micke@dynas.se>

Add -n flag to use the current modtime rather than preserving it.

New man page for fetch(1).
Submitted by: jmz (with some small tweaks)

This commit was generated by cvs2svn to compensate for changes in r16504,
which included commits to RCS files with non-trunk default branches.

Jean-Marc's url fetch program, with Josh MacDonald's patches and
Jordan's ftpio library.

Submitted by: jmz, jkh, jmacd (three-j!)