Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Merge from head r253514,r253680,r253805,r254650:
---------------------------------------------------------------------
r253514 | des | 2013-07-21 10:59:56 +0400 (вс, 21 июл 2013) | 5 lines

Use the correct request syntax for proxied (tunneled) HTTPS requests.

PR: bin/180666
MFC after: 3 days

---------------------------------------------------------------------
r253680 | des | 2013-07-26 19:53:43 +0400 (пт, 26 июл 2013) | 7 lines

Implement certificate verification, and many other SSL-related
imrovements; complete details in the PR.

PR: kern/175514
Submitted by: Michael Gmelin <freebsd@grem.de>
MFC after: 1 week

---------------------------------------------------------------------
r253805 | des | 2013-07-30 17:07:55 +0400 (вт, 30 июл 2013) | 5 lines

Include an Accept header in requests.

PR: kern/180917
MFC after: 1 week

---------------------------------------------------------------------
r254650 | des | 2013-08-22 11:43:36 +0400 (чт, 22 авг 2013) | 8 lines

Even though it doesn't really make sense in the context of a CONNECT
request, RFC 2616 14.23 mandates the presence of the Host: header in
all HTTP 1.1 requests.

PR: kern/181445
Submitted by: Kimo <kimor79@yahoo.com>
MFC after: 3 days

---------------------------------------------------------------------

Reviewed by: des

MFH r233648: cosmetic

MFH r225813,r225814,r226537: cosmetic

MFH r225810 r225812: make passive mode the default.

Approved by: re (kib)

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Document HTTP digest authentication support.

Mention some prominent past contributors: Hajimu Umemoto (ipv6), Henry
Whincup (https), Jukka Ukkonen (if-modified-since) and Jean-François
Dockes (digest auth)

Add support for HTTP 1.1 If-Modified-Since behavior.

fetch(1) accepts a new argument -i <file> that if specified will cause
the file to be downloaded only if it is more recent than the mtime of
<file>.

libfetch(3) accepts the mtime in the url structure and a flag to
indicate when this behavior is desired.

PR: bin/87841
Submitted by: Jukka A. Ukkonen <jau@iki.fi> (partially)
Reviewed by: des, ru
MFC after: 3 weeks

s/MAXPATHLEN/PATH_MAX/ to reflect five-year old change to the code :)

Submitted by: Joerg Sonnenberger <joerg@britannica.bec.de>
MFC after: 2 weeks

Document NO_PROXY / no_proxy.

MFC after: 3 weeks

Enable FTP_COMBINE_CWDS.

Add back the original behavior of changing the entire directory path at
once (CWD a/b/c vs. 3 CWDs). If an error occurs, we fall back to the default
method of a single CWD per directory element. Since this is technically
a violation of the basic FTP RFC, this behavior is under a compile-time
option FTP_COMBINE_CWDS and is off by default. It should work with most
Unix-based FTP daemons and can save latency.

MFC after: 2 weeks

Fix misspelling in Poul-Henning Kamp's email address under AUTHORS, from
pkh@ to phk@.

Approved by: ade

Expand contractions.

Expand *n't contractions.

Update copyright years.

Mechanically kill hard sentence breaks.

I don't normally use my middle name, so remove it from attributions in
man pages (though not from copyright notices). While I'm here, add email
addresses where appropriate.

mdoc(7): Use the new feature of the .In macro.

Add an EXAMPLES section.

PR: 43899
Reviewed by: des, ru

Assorted mdoc(7) fixes.

Approved by: re (blanket)

Add and document support for a FETCH_BIND_ADDRESS environment variable
specifying a local address to bind sockets to. Caveat: lightly tested.

PR: bin/37572

Document .netrc support.

english(4) police.

Implement and document support for an HTTP_REFERER environment variable.

PR: 28171
Submitted by: Andre Albsmeier <andre.albsmeier@mchp.siemens.de>
Approved by: re (bmah)
MFC after: 1 week

Fix two typos.

mdoc(7) police: Use the new .In macro for #include statements.

Remove whitespace at EOL.

mdoc(7) police: remove extraneous .Pp before and/or after .Sh.

mdoc(7) police: fixed markup and typo.

Implement a HTTP_USER_AGENT environment variable.

PR: 27669
Submitted by: Eoin Lawless <eoin@maths.tcd.ie>

Document the authentication callback interface.
Update RFC references (should have done that ages ago...)

Apply 'const' liberally.

Fix some other minor glitches.

Grammar police: "its", not "it's", is the possessive form of "it".

Use high port range by default, and replace the 'h' option with an 'l' option
that forces the ftp code to use the low (default) port range instead.

Support lower-case versions of the proxy environment variables.

PR: bin/25494

mdoc(7) police: fixed broken references.

Prepare for mdoc(7)NG.

The user name for anonymous ftp is now "anonymous".
Remove the period after the last man page reference.
Add a reference to RFC1635, and sort the RFC references by number.

Document FTP_LOGIN.

Somewhere along the line, I misunderstood the whole FTP_PASSIVE_MODE debate
and had libfetch selecting passive mode even when FTP_PASSIVE_MODE was not
set at all, which is really quite surprising unless you know about it. So
change it to the agreed default behaviour of selecting passive mode if
FTP_PASSIVE_MODE is set, but not "no".

Remove some obsolete comments.

(null commit)

Previous commit has wrong log message. The correct one is:

mdoc(7) police: do not split author names in the AUTHORS section.

log

Spell the des's name correctly.

Use Fx macro wherever possible.

Document FTP_PASSIVE_MODE and FTP_PASSWORD.

PR: docs/20626
Submitted by: nik

Document recent changes in URL parsing and proxy handling

Make the ENVIRONMENT section more legible, and mention that HTTP_AUTH is
colon-separated.

Document the FTP_PROXY and HTTP_PROXY variables better.

Document the proxy support better.

Introduce fetchXGet*(), which combine the functionalities of fetchGet*() and
fetchStat*(). In most cases, either fetchGet*() or fetchXGet*() is a wrapper
around the other; in all cases, calling fetchGet*() is identical to calling
fetchXGet*() with the second argument set to NULL.

Fully document the HTTP authorization feature.
Update the BUGS section to reflect the HTTP rewrite.

Document that basic authorization now kinda works.

Add (and document) fetchMakeURL()

Dump com_err, it's a stinking crock of shit.

Document struct url.
Document the default values for fetchStat*().

Fix hard sentence break.

Submitted by: sheldonh

Reorganize some of the http code and split it into more functions.
Implement fetchStatHTTP().
Unbungle struct url, and add fetchFreeURL().
Document it.

(null commit)

The previous commit was discussed with the maintainer (des).

Supply only one author name per instance of %A, as per mdoc.samples(7).

PR: 18465
Submitted by: Kazu TAKAMUNE <takamune@avrl.mei.co.jp>

Introduce .Lb macro to libfetch manpage
Sort .Nm values
Change first column width in errors list (table look much better now)
Remove redundant comma

Remove single-space hard sentence breaks. These degrade the quality
of the typeset output, tend to make diffs harder to read and provide
bad examples for new-comers to mdoc.

Patches I've had lying around for several months:
* Add the 'h' ftp flag (allocate local port in high range)
* Add the 'd' flag (use direct connection even if proxy is defined)
* Make sure flags != NULL before calling strchr().
* Minor changes to some comments.

$Id$ -> $FreeBSD$

Fixed prototype for fetchParseURL().

Implement and document file list retrieval.

Implement and document fetchStatFTP.
Update description of struct url_stat in the man page.
Clean up error handling in ftp.c.

Add verbose flag, and support functions.
Brucify the Makefile.
Differentiate atime and mtime in fetch*Stat().
Fix a few pointer bugs.
Tweak some error messages.
Don't #include sys/param.h and stdio.h in fetch.h.
Document that sys/param.h and stdio.h must be #included before fetch.h.

Second of a series of cleanups to libfetch.

This commit introduces the following features:

a) the fetchStat*() functions, which return meta-information for a
document, such as size, modification time, etc.

b) the use of the com_err(3) facilities to report errors.

It also fixes a bunch of style bugs and a few logic bugs and somewhat
improves the man page.

Changed files, in alphabetical order:

Makefile:
Don't generate macros in {ftp,http}err.c.

Generate category fields for the error message lists.

Compile the error table.

Install fetch_err.h along with fetch.h.

common.c:
Remove the _netdb_errstring() macro, and add FETCH_ERR_NETDB to the
error code in the _netdb_seterr() macro.

Add categories to the _netdb_errlist table.

Report errors through the Common Error library.

common.h:
Add the DEBUG macros.

Add prototype for fetchConnect().

Remove the prototype for _fetch_errstring(), which is local to common.c

Add a categroy field to struct fetcherr, and define constants for
error categories.

Define macros for _{url,netdb,ftp,http}_seterr().

errors.et: (new file)
List error categories.

fetch.3:
Document the fetchStat*() functions.

Move the "unimplemented functionality" comments from NOTES to BUGS.

Document that applications which use libfetch must also use
libcom_err, and list existing error codes.

Undocument fetchLastErr{Code,String}.

Remove the (empty) DIAGNOSTICS section.

Mention Eugene Skepner in the AUTHORS section.

fetch.c:
Move the DEBUG macros to common.c

Add fetchStat() and fetchStatURL().

Generate error messages for URL parser errors, and fix a minor bug
in the parser.

Use 'struct url' instead of 'url_t'.

Remove fetchLastErr{Code,String}.

fetch.h:
Use 'struct url' instead of 'url_t', and remove the typedef.

Define struct url_stat (used by fetchStat()).

Add prototypes for fetchStat*().

Remove the declarations for fetchLastErr{Code,String}.

Include fetch_err.h.

fetch_err.et: (new file)
Error table for libfetch.

file.c:
Add fetchStatFile().

Use 'struct url' instead of 'url_t'.

ftp.c:
Add fetchStatFTP().

Use 'struct url' instead of 'url_t'.

Don't use fetchLastErrCode.

ftp.errors:
Add categories to all error messages.

http.c:
Add fetchStatHTTP().

Use 'struct url' instead of 'url_t'.

Don't use fetchLastErr{Code,Text}.

http.errors:
Add categories to all error messages.

Prompted by: jkh and Eugene Skepner
Numerous sugestions from: Garett Wollman and Eugene Skepner

First of a series of cleanups to libfetch. Changed files, in
alphabetical order:

Makefile:
Add common.c to SRCS.

Make debugging easier by making 'CFLAGS += -DNDEBUG' conditional on DEBUG

Don't declare struct {ftp,http}err in {ftp,http}err.c; use struct fetcherr
instead.

README:
Remove the todo list, which is out of date anyway.

common.c: (new file)
Gather utility functions in this file.

Merge the error reporting functions intp _fetch_errstring(),
_fetch_seterr() and _fetch_syserr().

Set fetchLastErrCode and fetchLastErrText appropriately when fetchConnect
fails.

common.h: (new file)
Gather internal prototypes and structures in this files.

fetch.3:
Undocument fetchFreeURL().

Document a few more known bugs.

Document fetchLastErrCode and fetchLastErrText.

fetch.c:
Add descriptive comments to all functions that lacked them.

Move fetchConnect() to common.c.

Obviate the need for fetchFreeURL(), and remove it.

fetch.h:
Modify struct url_t so the document part is at the end.

ftp.c:
Remove code that is duplicated elsewhere.

http.c:
Remove code that is duplicated elsewhere.

Prompted by: jkh

Bring the man page more or less up to date with the code.

This commit was generated by cvs2svn to compensate for changes in r37535,
which included commits to RCS files with non-trunk default branches.

Imported libfetch into the tree. It compiles, but there's still some
work to do. I especially need help with the man page.