Remove svn:mergeinfo carried over from stable/9.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFH (r260904): fix format string
MFH (r261230,r261263): fix issues with buffering / stalling
MFH (r261284): bump copyright dates
MFH (r266291): look for root certs in /usr/local first

Approved by: re (gjb)

MFC r263021:

Support Last-Modified behind proxies which return UTC instead of GMT.

MFC r258347,r258349:

Support SNI in libfetch

SNI is Server Name Indentification which is a protocol for TLS that
indicates the host that is being connected to at the start of the
handshake. It allows to use Virtual Hosts on HTTPS.

PR: kern/183583
Approved by: bapt (implicit)

Merge from head r253514,r253680,r253805,r254650:
---------------------------------------------------------------------
r253514 | des | 2013-07-21 10:59:56 +0400 (вс, 21 июл 2013) | 5 lines

Use the correct request syntax for proxied (tunneled) HTTPS requests.

PR: bin/180666
MFC after: 3 days

---------------------------------------------------------------------
r253680 | des | 2013-07-26 19:53:43 +0400 (пт, 26 июл 2013) | 7 lines

Implement certificate verification, and many other SSL-related
imrovements; complete details in the PR.

PR: kern/175514
Submitted by: Michael Gmelin <freebsd@grem.de>
MFC after: 1 week

---------------------------------------------------------------------
r253805 | des | 2013-07-30 17:07:55 +0400 (вт, 30 июл 2013) | 5 lines

Include an Accept header in requests.

PR: kern/180917
MFC after: 1 week

---------------------------------------------------------------------
r254650 | des | 2013-08-22 11:43:36 +0400 (чт, 22 авг 2013) | 8 lines

Even though it doesn't really make sense in the context of a CONNECT
request, RFC 2616 14.23 mandates the presence of the Host: header in
all HTTP 1.1 requests.

PR: kern/181445
Submitted by: Kimo <kimor79@yahoo.com>
MFC after: 3 days

---------------------------------------------------------------------

Reviewed by: des

MFH (r252375): remove useless side effect which triggers a clang bug.

MFH (r243149): fix indentation
MFH (r249431): use CONNECT to proxy HTTPS over HTTP

PR: bin/80176

MFH (r240496): use libmd if and only if OpenSSL is not available

MFC r241841:
Implement HTTP 305 redirect handling.

PR: 172452
Approved by: cperciva (implicit)

MFC r241840:
Don't deny non-temporary redirects if the -A option is set (per
the man page) [0]

While here add support for draft-reschke-http-status-308-07

PR: 172451 [0]
Approved by: cperciva (implicit)

MFC r241839:
Be a bit more lenient in the maximum number of redirects allowed.
Chrome and Firefox have a limit of 20. IE has a limit of 8.

Approved by: cperciva (implicit)

MFC r240495:
Adding missing return statements during error conditions.

PR: kern/171187
Approved by: cperciva (implicit)

MFC r236193: libfetch: Avoid SIGPIPE on network connections.

To avoid unexpected process termination from SIGPIPE when writing to a
closed network connection, enable SO_NOSIGPIPE on all network connections.

The POSIX standard MSG_NOSIGNAL is not used since it requires modifying all
send calls to add this flag. This is particularly nasty for SSL connections.

MFH r234837: avoid busy-loop on slow connections when no timeout is set.
MFH r234838: don't reuse credentials when redirected to a different host.

MFH r233648: cosmetic

MFH r230478: fix nits in already-merged r230307.

MFH r225813,r225814,r226537: cosmetic

MFC r234138:

Support percent-encoded user and password

RFC 1738 specifies that any ":", "@", or "/" within a user name or
password in a URL is percent-encoded, to avoid ambiguity with the use
of those characters as URL component separators.

MFH r230307: fix SIGINFO-related data corruption bug

Approved by: des (mentor)

MFC: r227006, r227281, r227282

Add a PCI front-end to esp(4) allowing it to support AMD Am53C974 and
replace amd(4) with the former in the amd64, i386 and pc98 GENERIC kernel
configuration files. Besides duplicating functionality, amd(4), which
previously also supported the AMD Am53C974, unlike esp(4) is no longer
maintained and has accumulated enough bit rot over time to always cause
a panic during boot as long as at least one target is attached to it
(see PR 124667).

PR: 124667
Approved by: re (kib)
Obtained from: NetBSD (based on)

MFH r225810 r225812: make passive mode the default.

Approved by: re (kib)

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Mark all socket and file descriptors close-on-exec.

PR: bin/151866
MFC after: 3 weeks

Builds cleanly at the default WARNS level (WARNS=6).

Increase WARNS to 4.

Mechanical whitespace cleanup.

Increase WARNS to 3.

Fix a bug related to connection caching which could cause a crash after
a STAT command.

PR: kern/153748 (different patch)
Submitted by: Mark Johnston <markjdb@gmail.com>
MFC after: 2 weeks

Move variable declarations into the conditional block where they are
used, to fix warning if WITH_SSL is not set.

Submitted by: Sean Bruno
MFC after: 1 week

Redo fetch_read() using non-blocking sockets. This is necessary to
avoid a hang in the SSL case if the server sends a close notification
before we are done reading. In the non-SSL case, it can provide a
minor (but probably not noticeable) performance improvement for small
transfers.

MFC after: 3 weeks

Fix a couple of embarrassing mistakes in the previous commit.

Submitted by: Dimitry Andric <dimitry@andric.com>

If the A flag is supplied, http_request() will attempt the request only
once, even if authentication is required, instead of retrying with the
proper credentials. Fix this by bumping the countdown if the origin or
proxy server requests authentication so that the initial unauthenticated
request does not count as an attempt.

PR: 148087
Submitted by: Tom Evans <tevans.uk@googlemail.com>
MFC after: 2 weeks

Document HTTP digest authentication support.

Mention some prominent past contributors: Hajimu Umemoto (ipv6), Henry
Whincup (https), Jukka Ukkonen (if-modified-since) and Jean-François
Dockes (digest auth)

Add missing DPADD and fix whitespace.

Submitted by: ru

Add HTTP digest authentication.

Submitted by: Jean-Francois Dockes <jf@dockes.org>
Forgotten by: des (repeatedly)

Fix a socket leak in ftp_request() after that a connection is established.

Submitted by: Sandvine Incorporated
Reviewed by: des, emaste
Sponsored by: Sandvine Incorporated
MFC: 1 week

PR: 139751
Approved by: des
Obtained from: Xavier Heiny <xavier.heiny@netasq.com>
MFC after: 3 weeks

Bump the version of all non-symbol-versioned shared libraries in
preparation for 8.0-RELEASE. Add the previous version of those
libraries to ObsoleteFiles.inc and bump __FreeBSD_Version.

Reviewed by: kib
Approved by: re (rwatson)

1. Update fetch to consistently return 1 on error, as the man page states,
rather than usually returning 1 but in a few instances using a sysexits(3)
return value.

2. Remove a few unused variables from libfetch.

PR: docs/122470 (1, only)
Reviewed by: des
> Description of fields to fill in above: 76 columns --|
> PR: If a GNATS PR is affected by the change.
> Submitted by: If someone else sent in the change.
> Reviewed by: If someone else reviewed your modification.
> Approved by: If you needed approval for this commit.
> Obtained from: If the change is from a third party.
> MFC after: N [day[s]|week[s]|month[s]]. Request a reminder email.
> Security: Vulnerability reference (one per line) or description.
> Empty fields above will be automatically removed.

M usr.bin/fetch/fetch.c
M lib/libfetch/fetch.c

Add support for HTTP 1.1 If-Modified-Since behavior.

fetch(1) accepts a new argument -i <file> that if specified will cause
the file to be downloaded only if it is more recent than the mtime of
<file>.

libfetch(3) accepts the mtime in the url structure and a flag to
indicate when this behavior is desired.

PR: bin/87841
Submitted by: Jukka A. Ukkonen <jau@iki.fi> (partially)
Reviewed by: des, ru
MFC after: 3 weeks

Don't fail mistakenly with -r when we already have the whole file.

Reviewed by: des

Fix one-byte buffer overflow: NUL gets written to the buffer, but isn't
counted in the width specification in scanf.

This is not a security problem, since this function is only used to
parse a user's configuration file.

Submitted by: Joerg Sonnenberger
Obtained from: dragonflybsd
MFC after: 1 week

s/wait/delta/ to avoid namespace collision.

MFC after: 2 weeks

Use memcpy(3) instead of the BSD-specific bcopy(3).

Submitted by: Joerg Sonnenberger <joerg@britannica.bec.de>
MFC after: 2 weeks

s/MAXPATHLEN/PATH_MAX/ to reflect five-year old change to the code :)

Submitted by: Joerg Sonnenberger <joerg@britannica.bec.de>
MFC after: 2 weeks

Add necessary cast for tolower() argument.

Submitted by: Joerg Sonnenberger <joerg@britannica.bec.de>
MFC after: 1 week

Fix a regression introduced in rev 1.99: replace fclose(f) with a comment
explaining why f cannot possibly be a valid FILE * at this point.

MFC after: 1 day

Add missing dependencies on Makefile (fixes NO_CLEAN builds).

As several people pointed out, I did all the ctype casts the wrong
way (not for the first time...)

Noticed by: bde, ru ++
MFC after: 1 week

Document NO_PROXY / no_proxy.

MFC after: 3 weeks

Add support for the NO_PROXY / no_proxy environment variable as used by
lynx, curl etc. Note that this patch differs significantly from that
in the PR, as the submitter refined it after submitting the PR.

PR: 110388
Submitted by: Alexander Pohoyda <alexander.pohoyda@gmx.net>
MFC after: 3 weeks

Old patch I had lying around: correctly cast the argument to is*().
IWBNI gcc could warn about this the way it warns about printf() abuse.

MFC after: 1 week

Clean up namespace violations.

MFC after: 1 week

Enable FTP_COMBINE_CWDS.

Bump library versions in preparation for 7.0.

Ok'd by: kan

Handle temporary redirects (HTTP status code 307)

PR: kern/112515
Submitted by: Ryan C. Gordon <icculus@icculus.org>
MFC after: 3 weeks

Add back the original behavior of changing the entire directory path at
once (CWD a/b/c vs. 3 CWDs). If an error occurs, we fall back to the default
method of a single CWD per directory element. Since this is technically
a violation of the basic FTP RFC, this behavior is under a compile-time
option FTP_COMBINE_CWDS and is off by default. It should work with most
Unix-based FTP daemons and can save latency.

MFC after: 2 weeks

Respect MK_INET6_SUPPORT.

Respect FETCH_BIND_ADDRESS when opening the data connection.

PR: misc/98872

Fix a bug introduced in rev 1.92, where, when changing from one directory
to another, the first CWD after a string of CDUPs would incorrectly include
a slash before the directory name.

Reported by: obrien
PR: bin/83278

Reimplementation of world/kernel build options. For details, see:

http://lists.freebsd.org/pipermail/freebsd-current/2006-March/061725.html

The src.conf(5) manpage is to follow in a few days.

Brought to you by: imp, jhb, kris, phk, ru (all bugs are mine)

In order to maintain interoperability with certain broken FTP servers,
ignore a MODE failure if and only if the mode we attempted to set was S
(which is supposed to be the default).

PR: bin/91973

Fix misspelling in Poul-Henning Kamp's email address under AUTHORS, from
pkh@ to phk@.

Approved by: ade

Ignore HTTP_PROXY if it is defined but empty. This was already handled
correctly in the case of FTP_PROXY, because an empty FTP_PROXY has a
specific meaning ("don't use any proxy at all for ftp, even if HTTP_PROXY
is defined"), while an empty HTTP_PROXY has no meaning at all.

PR: bin/85185
Submitted by: Conall O'Brien <conallob=freebsd@maths.tcd.ie>
MFC after: 2 weeks

Change directory one level at a time, and use CDUP to back out. This is a
work in progress; it partially fixed bin/83278 and is a prerequisite to
fixing bin/83277.

PR: bin/83277, bin/83278

Bump the shared library version number of all libraries that have not
been bumped since RELENG_5.

Reviewed by: ru
Approved by: re (not needed for commit check but in principle...)

Set the TCP_NODELAY socket option and clear TCP_NOPUSH in order to flush
any pending HTTP request rather than calling shutdown(2) with SHUT_WR.
This makes libfetch (and thus fetch(1)) work again with Squid proxies
configured to not allow half-closed connections.

Reported by: Pawel Worach (pawel.worach AT telia DOT com)

Fix a memory leak: when freeing the connection structure, don't forget to
free the connection buffer as well.

PR: bin/76153
MFC after: 1 week

Set TCP_NOPUSH on HTTP requests, reducing the number of round-trips
necessary to establish each connection.

MFC after: 2 weeks

Expand contractions.

Expand *n't contractions.

NOCRYPT -> NO_CRYPT

Update copyright years.

Don't forget to allocate space for the terminating NUL when converting to
base 64.

PR: misc/70022
Submitted by: Herve Masson <herve-bsdbt@mindstep.com>
MFC after: 3 days

Join the 21st century: Cryptography is no longer an optional component
of releases. The -DNOCRYPT build option still exists for anyone who
really wants to build non-cryptographic binaries, but the "crypto"
release distribution is now part of "base", and anyone installing from a
release will get cryptographic binaries.

Approved by: re (scottl), markm
Discussed on: freebsd-current, in late April 2004

Mechanically kill hard sentence breaks.

Slight tweak to previous commit: don't forget to call _http_seterr() if
we get a genuine 416 reply.

When restarting a transfer that has already completed, the server will
reply with a 416 error code (requested range not satisfiable) because
we ask it to start at the end of the file. Handle this gracefully by
considering a 416 reply a success if the requested offset exactly
matches the length of the file and the requested length is zero.

Re-wrap some comments.

I don't normally use my middle name, so remove it from attributions in
man pages (though not from copyright notices). While I'm here, add email
addresses where appropriate.

preparation for RFC3493. EAI_NODATA was deprecated.

mdoc(7): Use the new feature of the .In macro.

Don't close a FILE * which we know is bogus.

Some servers respond to RETR in active mode with 125 (connection already
open) rather than 150 (opening connection). There's no reason why we
shouldn't accept that.

PR: misc/42172
MFC in: 3 days

Make sure the crypto versions of libfetch and fetch(1) appear in
the "crypto" distribution.

Approved by: des

Add an EXAMPLES section.

PR: 43899
Reviewed by: des, ru

Assorted mdoc(7) fixes.

Approved by: re (blanket)

Retire the useless NOSECURE knob.

Approved by: re (scottl)

Mark libfetch as c99-clean. Always build at WARNS level 2 rather than
juggling with levels 2 and 3, as this has lead to world breakage for
NOCRYPT users in the past.

Don't dereference flags if NULL (see http.c rev 1.87)

The flags passed in to _ftp_get_proxy may be null

Approved by: des, markm (mentor)(implicit)

Don't parse the proxy URL unless we're actually going to use it. No real
functional difference, but debugging output will be less confusing.

Add and document support for a FETCH_BIND_ADDRESS environment variable
specifying a local address to bind sockets to. Caveat: lightly tested.

PR: bin/37572

No need for the OPENSSL_NO_KRB5 switch anymore.

Fixed by: nectar

Adjust for OpenSSL 0.9.7.

style(9): add parentheses to sizeof even when not strictly required.

MFC after: 3 days

Fix signed/unsigned comparison warning/error from 'make release'

Document .netrc support.

Hook in the .netrc code + don't use pointers as if they were booleans.

Experimental support for .netrc.

Initialization paranoia.

A negative offset means "get it all".

Set auto-retry mode to avoid some spurious errors.

Submitted by: Andre Albsmeier <andre.albsmeier@siemens.com>
MFC after: 7 days

english(4) police.

Reduce WARNS level to 3 in the non-crypto case to fix make release on sparc64.
This may only be turning up now to changes in the cpp predefines for sparc64,
which may be turning on more strict alignment checking.

Approved by: re

Implement and document support for an HTTP_REFERER environment variable.

PR: 28171
Submitted by: Andre Albsmeier <andre.albsmeier@mchp.siemens.de>
Approved by: re (bmah)
MFC after: 1 week

Fix two typos.

Use __func__ and break a long line.

Fix a bug in fenner's _fetch_writev() patch (rev 1.36)

Submitted by: fenner

Recognize the (incorrect) error code a MediaHawk server sends in reply to
unrecognized commands such as MDTM.

Requested by: Stephen Roome <stephen_roome@pepcross.com>

Clarify my feelings towards fetch / libfetch.

Recommit the non-broken parts of 1.34 and 1.37.
Change the type and name of a variable introduced in 1.33.

Don't forget to set the error flag when _fetch_read() returns -1.

Reinstate revs 1.35-36 and 1.38. Revisions 1.34 and 1.37 were specifically
the root cause of the bus errors I was experiencing.

Submitted by: fenner
Tested by: obrien
Prompted by: peter

Fix `pkg_add -r' by backing out revs 1.34-1.38.
Revs 1.37-8 produce a bus error in some environments.
Revs 1.34-6 do not bus error, but write corrupted files.

libfetch is now WARNS5 clean in the non-SSL case.

Fix an off-by-one error (> where >= should have been used) which caused
_fetch_writev() to incorrectly report EPIPE in certain cases.

Also fix a number of const warnings by using __DECONST(), plus a signed /
unsigned comparison by casting the rhs to ssize_t.

Submitted by: fenner, Craig Rodrigues <rodrigc@attbi.com>

Up WARNS to 3 if not building with crypto.

Approved by: des

Slight amendment to rev 1.34: instead of considering any short read an
error, only report an error if no data was read at all (unless len was
0 to start with). Otherwise, the final read of practically any transfer
will end in a fatal error.

Introduce _fetch_writev(), which is the conn_t version of writev(2). In
the SSL case, it is no different from the old _fetch_write(), but in the
non-SSL case it uses writev(2) to send the entire vector as a single
packet (provided it can fit in one packet). Implement _fetch_write()
and _fetch_putln() in terms of _fetch_writev().

This should improve performance in the non-SSL case (by reducing protocol
overhead) and solve the problem where too-smart-for-their-own-good
firewalls reject FTP packets that do not end in CRLF.

PR: bin/44123
Submitted by: fenner

Eliminate two cases of undefined behaviour: total in _fetch_write() was
not initialized before use, and _http_growbuf() did not return a value
on success.

Reported by: Peter Edwards <pmedwards@eircom.net>
MFC after: 2 weeks

Back out the previous commit, and fix the bug rather than try to hide its
symptoms: make timeouts and short transfers fatal, and set errno to an
appropriate value (ETIMEDOUT for a timeout, EPIPE for a short transfer).

MFC after: 2 weeks

The FTP connection caching needs a better interface -- connections are
closed through _fetch_close() which is the only one who knows the connection
REALLY was closed (since ref -> 0). However, FTP keeps its own local
cached_connection and checks if it is valid by comparing it to NULL. This
is bogus since it may have been freed elsewhere by _fetch_close().

This change checks if we are closing the cached_connection and the ref is 1
(soon to be 0). If so, set cached_connection to NULL so we don't
accidentally reuse it. The REAL fix should be to move connection caching
to the common.c level (_fetch_* functions) and NULL the cache(s) in
_fetch_close(). Then all layers could benefit from caching.

Use a warns setting we can catch regressions with.

When recycling a cached connection, increment the reference count so that
the heap block does not get freed and reused. This should fix the
pkg_add -r crashes that have been happening for months.

HTTP authentication got broken in rev. 1.49.

Zap now-unused SHLIB_MINOR

Fix an infinite loop when _fetch_read() can return 0 (if the
connection is broken), take this into account and return at this
point.

Make _fetch_connect() always set the error code.
Tell ftp that _fetch_connect() always sets the error code (http already knew)

Surround prototypes with __BEGIN_DECLS / __END_DECLS.

PR: misc/40399
MFC after: 3 days

Cope with 2292bis-01 getaddrinfo (no NI_WITHSCOPEID, always attach
scope identifier).

Approved by: des
MFC after: 3 weeks

Reintroduce debugging code that somehow got lost in a previous revision.

Fix incorrect library ordering. I thought I'd committed this already...

Don't try to dereference conn when we know it's NULL.

Add a reference count to struct fetchconn so we don't prematurely close and
free a cached FTP connection.

libfetch now depends on libcrypto and libssl.

Make SSL support conditional on NOCRYPT.

During buildworld, "regular" libraries are built before crypto stuff, so
libfetch can't depend on lib{crypto,ssl}. Move the dependency to fetch
until we can figure out how to fix this.

Add SSL support + slight cleanup.

Submitted by: Henry Whincup <henry@techiebod.com> (in principle)

Wrap everything in struct connection, and enforce timeouts everywhere
(except for DNS operations). Always use funopen() for HTTP, to support
both timeouts and SSL.

Add the necessary dependencies for SSL.

Rename struct cookie to struct httpio to avoid confusion (it's not an HTTP
cookie) and increase symmetry with equivalent FTP code.

Add comments to struct cookie.

Fix a bug I introduced in the chunk decoder in the previous commit..

First step towards SSL support: wrap connections in a 'struct connection'
which contains the socket descriptor, the input buffer and (yet unused)
SSL state variables. This has the neat side effect of greatly improving
reentrance (though we're not *quite* there yet) and opening the door to
HTTP connection caching.

This commit is inspired by email conversations with and patches from
Henry Whincup <henry@techiebod.com> last fall.

Do not use __progname directly (except in [gs]etprogname(3)).
Also, make an internal _getprogname() that is used only inside
libc. For libc, getprogname(3) is a weak symbol in case a
function of the same name is defined in userland.

Modernize my email address

Switch to a self-starting allocation scheme.

Reindent, and add parentheses to return statements. Some functions in
ftp.c and http.c now have exceedingly long lines due to deep nesting;
this will be corrected by reorganizing the code in a later revision.

Eliminate a redundant loop.

Check the return value from read() when reading the CR/LF at the end of a
chunk.

PR: bin/33608
MFC after: 2 weeks

Mark uploads as O_WRONLY, not O_RDONLY.

PR: misc/34043
MFC after: 2 weeks

Remove long-obsolete README.

In verbose mode, display the full error message from the server, stripping
it of HTML tags and comments.

PR: bin/32989
MFC after: 1 week

Remove VT100 escapes from debugging messages now that they're enabled by
default.

PR: 32988
MFC after: 3 days

Don't build with NDEBUG.

PR: bin/32615
MFC after: 1 week

Conditionalize some debugging code that didn't use the DEBUG macro.

MFC after: 1 week

Introduce a fetchDebug global. Change the DEBUG macro so it only runs the
debugging code if fetchDebug is set.

PR: bin/32615
MFC after: 1 week

Properly handle being redirected to an FTP URL.

Also fix a couple of not-so-minor bugs:
- missing case for HTTP_SEE_OTHER
- incorrect definition of HTTP_NEED_PROXY_AUTH

PR: 20259
MFC after: 2 weeks

Add prototype for _ftp_request().

Reorganize to reduce code duplication.

Add NO_WERROR so the build won't die because of discarded qualifiers

Back out part of previous commit which was gcc-centric

Tons of type, style and warning fixes that have been rotting in my tree for
ages - some of which wouldn't be necessary if gcc wasn't broken or TPTB were
willing to do something (-fno-builtin) about it.

mdoc(7) police: Use the new .In macro for #include statements.

Add __FBSDID()s to libfetch

Use fseeko() instead of fseek() (u->offset is already an off_t), and mark
some function arguments as unused.

Handle snprintf() returning < 0 (not just -1)

MFC after: 2 weeks

Handle snrintf overflows.

Spotted by: bde

Handle snprintf() returning -1.

MFC after: 2 weeks

The fix for schemeless and hostless URLs (rev. 1.27) broke the schemeless
proxy specification, which seems to be valid according to the man page.

Change the logic to consider "hostname:port" a hostname and port instead
of a file URL.

Approved by: des

Unbreak parsing URLs that have a host part but no document part.
While we're here, fix a snprintf() usage warning.

Remove whitespace at EOL.

mdoc(7) police: remove extraneous .Pp before and/or after .Sh.

Handle shemeless, hostless URLs correctly.

mdoc(7) police: fixed markup and typo.

Don't call _ftp_closefn() upon EOF in _ftp_readfn(); just return 0. This
fixes a bug in pkg_add(1) (which nobody noticed because of another bug).

Implement a HTTP_USER_AGENT environment variable.

PR: 27669
Submitted by: Eoin Lawless <eoin@maths.tcd.ie>

Document the authentication callback interface.
Update RFC references (should have done that ages ago...)

Bump major number.

Add rudimentary support for an authentication callback function.

MAXPATHLEN -> PATH_MAX

Plug memory leak.

PR: 27506

Add some error codes that were added in RFC2616.

Apply 'const' liberally.

Fix some other minor glitches.

Grammar police: "its", not "it's", is the possessive form of "it".

If the server's reply to the SIZE command is unparseable, reset us->size
to -1 so the caller will know it's invalid. This is an MFC candidate.

MAN[1-9] -> MAN.

Use high port range by default, and replace the 'h' option with an 'l' option
that forces the ftp code to use the low (default) port range instead.

Don't remember an EINTR, since the caller may want to restart the call.

Support lower-case versions of the proxy environment variables.

PR: bin/25494

mdoc(7) police: fixed broken references.

Prepare for mdoc(7)NG.

The user name for anonymous ftp is now "anonymous".
Remove the period after the last man page reference.
Add a reference to RFC1635, and sort the RFC references by number.

Back out previous (accidental) commit.

Use "anonymous" rather than "ftp" as login name for anonymous ftp.

Rather than have a separate (misnamed) FTP_ANONYMOUS_PASSWORD constant, use
FTP_ANONYMOUS_USER (i.e. "anonymous") to construct the anonymous ftp password
if getlogin() fails.

Document FTP_LOGIN.

Check the FTP_LOGIN environment variable before falling back on
FTP_ANONYMOUS_USER.

Avoid a segfault (due to an unitialized pointer) when parsing URLs that have
no scheme or host part.

Somewhere along the line, I misunderstood the whole FTP_PASSIVE_MODE debate
and had libfetch selecting passive mode even when FTP_PASSIVE_MODE was not
set at all, which is really quite surprising unless you know about it. So
change it to the agreed default behaviour of selecting passive mode if
FTP_PASSIVE_MODE is set, but not "no".

Remove some obsolete comments.

Clean up the whitespace encoding code.

Add SCHEME_HTTPS.

Fix old-style proxy specs: default to FTP if FTP_PROXY was set; only default
to HTTP if HTTP_PROXY was used instead.

Don't try to get the proxy port number from /etc/services.

Don't go haywire if the server closes the connection in the middle of a
multiline response (proper fix this time).

Revert previous commit, it was somewhat hasty.

Don't go haywire if the server closes the connection during a multiline
response.

log

Spell the des's name correctly.

Use Fx macro wherever possible.

Use the documented (and historical) defaults. Centralize the decision logic
in order to avoid this bug in the future.

Submitted by: se

Use CHECK_FLAG

Stricter error checking in the I/O functions.

Add CHECK_FLAGS, a macro for (safely) checking if a particular flag is set.

Document FTP_PASSIVE_MODE and FTP_PASSWORD.

PR: docs/20626
Submitted by: nik

Fix this my way. David had absolutely no call overriding MAINTAINER without
even giving me 24 hours to read his mail and find the bug.

Install per function manpages so one doesn't mistakenly think we don't
have manpages for libfetch's functions.

Repeat after me, "check to see that a pointer isn't NULL before
dereferenceing it". This fixes ``pkg_add -r''.

MFS: don't expect result code until you close the data connection

Use funopen() instead of fdopen(). This fixes three problems:

- ftpTimeout was not honored when reading actual data, as opposed to
talking protocol

- connection caching was broken because _ftp_cached_connect() would see
the result of the transfer instead of the result of the NOOP.

- if the RETR succeeded, but an error occurred later (as can happen
when talking to a proxy), the error would not be detected.

There still remains to register an atexit(3) callback to close the cached
connection gracefully instead of just dropping it on the floor.

If the scheme is HTTP or HTTPS, percent-escape whitespace in the document
part.

Submitted by: green

Argh! Fix passive mode selection (again)

Use the right user name

Document recent changes in URL parsing and proxy handling

ftp.c needs _http_request()

Rework proxy handling so that proxies can be specified as URLs.
As a side effect, remove a lot of duplicate and now redundant code.

Relax URL syntax so that schemeless URLs are supported.

Add macros for the names of the URL schemes we support.

Make the ENVIRONMENT section more legible, and mention that HTTP_AUTH is
colon-separated.

Work around buggy servers such as NCSA httpd which send an incomplete
HTTP-Version on the Status-Line (see RFC2616 sections 3.1 and 6.1).

Document the FTP_PROXY and HTTP_PROXY variables better.

Grok 125 replies to passive RETR. This fixes an interoperability bug with
Microsoft FTP Service.

Reported by: asmodai, eivind

Fix signedness bogon.

Don't fail if _ftp_stat() reports a protocol error, since that most likely
means that the server doesn't understand SIZE or MDTM, which should not be a
hard failure.

Submitted by: ume

If the server reports the size as 0, treat it as unknown. This works around
a bug in some ftp servers (most notably ftp.vmunix.com) which report the
size of a file correctly in ascii mode, but report it as 0 in binary mode.

Reported by: asmodai

Also remove an unneeded initialization.

Centralize the default port finding code.
Work around YA Apache bug: don't send port in Host: header if it's the
default port.

Ignore environment variables that are set but empty.

Sort out the size / length confusion. Always try to report the *real* file
size in the url_stat structure, no matter how much of it is actually being
sent, and try to detect inconsistencies between sizes.

Rearrange the request loop to avoid having to add meaningless code just to
silence compiler warnings.

Switch to a more sensible and consistent interface for the _http_parse*()
functions.

Document the proxy support better.

The decision to use an HTTP proxy for FTP documents belongs in the FTP code

Don't send port number to proxy unless it's not the standard FTP port

Fix the case where the file name is preceded by a lone slash.

Don't try to skip to the requested offset if the server returns more data
than requested. Instead, inform the caller of the real offset by modifying
the offset field in the original struct url, and let him decide how to handle
the situation.

Don't take a failure to stat a file to mean that it doesn't exist, as it may
be a dynamically created file, e.g. a tarball on servers that support sending
tarballs of directories.

Introduce fetchXGet*(), which combine the functionalities of fetchGet*() and
fetchStat*(). In most cases, either fetchGet*() or fetchXGet*() is a wrapper
around the other; in all cases, calling fetchGet*() is identical to calling
fetchXGet*() with the second argument set to NULL.

Don't forget to set file descriptor to -1 after closing it, since the code
outside the loop inspects it to determine whether or not we succeeded in
retrieving the requested document. This fixes a bug where fetchGetHTTP()
would return a FILE with an invalid file descriptor if it hit the redirect
limit without locating the requested document.

Use a symbolic constant instead of hardcoding 999

Don't forget to declare fetchRestartCalls.

Introduce the (undocumented) variable fetchRestartCalls, which controls whether
or not interrupted system calls will be restarted. This fixes a bug where
fetch(1) would hang (potentially forever) if a server stopped responding,
because the signal handler would absorb the user's efforts to interrupt the
transfer.

Remove fetchContentType(), which disappeared in the HTTP rewrite.

#ifdef DEBUG -> #ifndef NDEBUG

Pointed out by: ache

Reinsert the MIT copyright, which applies to the base64 code and was
left out by accident during the rewrite.

Fix a bug (misplaced continue) that caused redirects to fail. Lots of code
moved around, but the acutal functional changes are small.
Add support for site-internal redirects (where the Location: header gives a
path instead of an absolute URI)

Pointed out by: kuriyama

Fully document the HTTP authorization feature.
Update the BUGS section to reflect the HTTP rewrite.

Extensive rewrite of the HTTP code. The new code is significantly cleaner and
more robust, and somewhat more efficient. It also handles authorization and
redirects properly, and supports timeouts like the FTP code.

Many thanks to Umemoto-san for his assistance with IPv6 support, both here
and in other parts of libfetch.

Use fetch_putln()

Add _fetch_putln()

Document that basic authorization now kinda works.

Rework the authorization code.
Only send absolute URI if connected to a proxy, since Apache doesn't always
understand absolute URIs.
Clean up some of the debugging output.

Clean up the debugging output

Free chunks obtained by getaddrinfo(3).

Approved by: des

Make EPSV work again. Separate parsing code of 229 replies from
the code for 227 and 228.

Submitted by: des

Don't assume the remote address in a 227 reply is enclosed in parentheses.

Fix basic authentication, and add proxy authentication.

Submitted by: se

Make restart work in active mode, too.

PR: bin/18688
Submitted by: Rudolf Cejka <cejkar@dcse.fee.vutbr.cz>

Handle multiline replies properly, instead of kinda-right.

Don't incorrectly report a protocol error when we get a 302 and redirects
are disabled.

Back out wes' commit with extreme prejudice.

Add implementation of fetchListFTP.

Reviewed by: silence

Hackish support for 302 redirect.

Honor FTP_PASSIVE_MODE so caller doesn't need to.

Output newline after debugging message

Add (and document) fetchMakeURL()

Dump com_err, it's a stinking crock of shit.

Use $FTP_PASSWORD for FTP password. If $FTP_PASSWORD is not
found, `yourname@yourhost' is used.

IPv6 support.

Better handling of some boundary conditions.

Submitted by: ume

Make HTTP_PROXY work for FTP.

Reported by: Ben Smithurst <ben@scientia.demon.co.uk>
Reviewed by: des

Document struct url.
Document the default values for fetchStat*().

Initialize the struct url_stat at the beginning of _fetch_stat_file().

Initialize the struct url_stat at the beginning of fetchStatFTP().

Remove unused 'verbose'.
Initialize the struct url_stat at the beginning of fetchStatHTTP().

Add workaround for ftpds with the Y2K MDTM bug

Bump major number, since struct url has changed.

Discussed with: jdp

Fix hard sentence break.

Submitted by: sheldonh

Reorganize some of the http code and split it into more functions.
Implement fetchStatHTTP().
Unbungle struct url, and add fetchFreeURL().
Document it.

Supply only one author name per instance of %A, as per mdoc.samples(7).

PR: 18465
Submitted by: Kazu TAKAMUNE <takamune@avrl.mei.co.jp>

While we're at it, add a length field too in case we want to fully implement
partial fetches later.

Implement restart

Use sizeof more consistently.

Use sizeof more consistently.
Outline sanity checking of server port spec.

Implement restart.
Use sizeof more consistently.
Outline sanity checking of server port spec.

Implement restart

Add offset field to struct url

Introduce .Lb macro to libfetch manpage
Sort .Nm values
Change first column width in errors list (table look much better now)
Remove redundant comma

Remove single-space hard sentence breaks. These degrade the quality
of the typeset output, tend to make diffs harder to read and provide
bad examples for new-comers to mdoc.

Fix timestamp handling.

Submitted by: bde

Replace beforeinstall target with new variables used by .mk system.

Reviewed by: marcel, and make world

More old uncommitted patches: implement timeouts at the protocol level.
Currently only supported for ftp connections.

Add error codes for protocol errors.

Patches I've had lying around for several months:
* Add the 'h' ftp flag (allocate local port in high range)
* Add the 'd' flag (use direct connection even if proxy is defined)
* Make sure flags != NULL before calling strchr().
* Minor changes to some comments.

$Id$ -> $FreeBSD$

Fixed prototype for fetchParseURL().

Add MAINTAINER.

Implement and document file list retrieval.

Don't specify a language to compile_et.
Rename fetchConnect to _fetch_connect since it's internal.

Implement and document fetchStatFTP.
Update description of struct url_stat in the man page.
Clean up error handling in ftp.c.

Remove unused header files. Fix a few shadowing bugs. Improve error reporting.

Add verbose flag, and support functions.
Brucify the Makefile.
Differentiate atime and mtime in fetch*Stat().
Fix a few pointer bugs.
Tweak some error messages.
Don't #include sys/param.h and stdio.h in fetch.h.
Document that sys/param.h and stdio.h must be #included before fetch.h.

Fix the "Unknown HTTP error" message.
Make compile_et generate prototypes.

fetch_err.h is a generated file, so don't look for it in ${.CURDIR}.

Second of a series of cleanups to libfetch.

This commit introduces the following features:

a) the fetchStat*() functions, which return meta-information for a
document, such as size, modification time, etc.

b) the use of the com_err(3) facilities to report errors.

It also fixes a bunch of style bugs and a few logic bugs and somewhat
improves the man page.

Changed files, in alphabetical order:

Makefile:
Don't generate macros in {ftp,http}err.c.

Generate category fields for the error message lists.

Compile the error table.

Install fetch_err.h along with fetch.h.

common.c:
Remove the _netdb_errstring() macro, and add FETCH_ERR_NETDB to the
error code in the _netdb_seterr() macro.

Add categories to the _netdb_errlist table.

Report errors through the Common Error library.

common.h:
Add the DEBUG macros.

Add prototype for fetchConnect().

Remove the prototype for _fetch_errstring(), which is local to common.c

Add a categroy field to struct fetcherr, and define constants for
error categories.

Define macros for _{url,netdb,ftp,http}_seterr().

errors.et: (new file)
List error categories.

fetch.3:
Document the fetchStat*() functions.

Move the "unimplemented functionality" comments from NOTES to BUGS.

Document that applications which use libfetch must also use
libcom_err, and list existing error codes.

Undocument fetchLastErr{Code,String}.

Remove the (empty) DIAGNOSTICS section.

Mention Eugene Skepner in the AUTHORS section.

fetch.c:
Move the DEBUG macros to common.c

Add fetchStat() and fetchStatURL().

Generate error messages for URL parser errors, and fix a minor bug
in the parser.

Use 'struct url' instead of 'url_t'.

Remove fetchLastErr{Code,String}.

fetch.h:
Use 'struct url' instead of 'url_t', and remove the typedef.

Define struct url_stat (used by fetchStat()).

Add prototypes for fetchStat*().

Remove the declarations for fetchLastErr{Code,String}.

Include fetch_err.h.

fetch_err.et: (new file)
Error table for libfetch.

file.c:
Add fetchStatFile().

Use 'struct url' instead of 'url_t'.

ftp.c:
Add fetchStatFTP().

Use 'struct url' instead of 'url_t'.

Don't use fetchLastErrCode.

ftp.errors:
Add categories to all error messages.

http.c:
Add fetchStatHTTP().

Use 'struct url' instead of 'url_t'.

Don't use fetchLastErr{Code,Text}.

http.errors:
Add categories to all error messages.

Prompted by: jkh and Eugene Skepner
Numerous sugestions from: Garett Wollman and Eugene Skepner

First of a series of cleanups to libfetch. Changed files, in
alphabetical order:

Makefile:
Add common.c to SRCS.

Make debugging easier by making 'CFLAGS += -DNDEBUG' conditional on DEBUG

Don't declare struct {ftp,http}err in {ftp,http}err.c; use struct fetcherr
instead.

README:
Remove the todo list, which is out of date anyway.

common.c: (new file)
Gather utility functions in this file.

Merge the error reporting functions intp _fetch_errstring(),
_fetch_seterr() and _fetch_syserr().

Set fetchLastErrCode and fetchLastErrText appropriately when fetchConnect
fails.

common.h: (new file)
Gather internal prototypes and structures in this files.

fetch.3:
Undocument fetchFreeURL().

Document a few more known bugs.

Document fetchLastErrCode and fetchLastErrText.

fetch.c:
Add descriptive comments to all functions that lacked them.

Move fetchConnect() to common.c.

Obviate the need for fetchFreeURL(), and remove it.

fetch.h:
Modify struct url_t so the document part is at the end.

ftp.c:
Remove code that is duplicated elsewhere.

http.c:
Remove code that is duplicated elsewhere.

Prompted by: jkh

Bring the man page more or less up to date with the code.

Made this actually work when there's an object directory:
- replaced bogus dependencies of distribution sources on generated
sources by the same ones that bsd.lib.mk would generate if it knew
all the sources. We shoot ourself in the foot by not naming the
generated (included) sources *.h, so we can just put the generated
sources in SRCS.
- replaced -I${.CURDIR} by `-I.'. Here `.' is an alias for ${.OBJDIR}.
-I${.CURDIR} didn't do anything, since ${.CURDIR} is the default in
all cases here (it would be necessary for ""-style includes made from
sources in ${.OBJDIR}.

Don't use `+=' for setting CLEANFILES for the first time.

Added $Id$.

Commit a bunch of patches that have been accumulating:

- Fix the README to reflect the new status of the ftp code.
- Change tons of 'if (xxx < 0)' to 'if (xxx == -1)'
- Add two new interface functions
- Fix the Makefile so it actually works (yay!)

Now the manpage is lagging even further behind... :( Next on the todo
list is to clean up the http code.

Forgot to remove base64.c from SRCS in the Makefile.

Base64 code (and the MIT copyright) moved to http.c
FTP STORe and APPEnd added.
FTP proxy support added (untested).

FTP retrieve works.

Too many changes to list. Basically, FTP is nearly there and error
reporting is kinda sorted out. Now HTTP needs to catch up...

This commit was generated by cvs2svn to compensate for changes in r37535,
which included commits to RCS files with non-trunk default branches.