MFH (r314778): use reallocarray(3) for extra bounds checks
MFH (r333306): fix typo in man page
MFH (r333571, r333572): preserve if-modified-since across redirects
MFH (r334317): simplify the DEBUG macro
MFH (r334319): style bug roundup
MFH (r334326): fix netrc file location logic, improve netrcfd handling
MFH (r338572): fix end-of-transfer statistics, improve no-tty display

PR: 202424, 224426, 228017

Revert r330897:

This was intended to be a non-functional change. It wasn't. The commit
message was thus wrong. In addition it broke arm, and merged crypto
related code.

Revert with prejudice.

This revert skips files touched in r316370 since that commit was since
MFCed. This revert also skips files that require $FreeBSD$ property
changes.

Thank you to those who helped me get out of this mess including but not
limited to gonzo, kevans, rgrimes.

Requested by: gjb (re)

Partial merge of the SPDX changes

These changes are incomplete but are making it difficult
to determine what other changes can/should be merged.

No objections from: pfg

MFH (r313974,r314596): open .netrc early in case we want to drop privs
MFH (r314396,r315143): fix a crash caused by an incorrect format string
MFH (r314701): fix handling of 416 errors when requesting a range
MFH (r315455): fix parsing of IP literals (square brackets)

PR: 212065, 217723

MFH (r310823): fix multi-line CONNECT responses

PR: 194483

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

r169386 (PR 112515) was incomplete: it treated 307 as an error except
in verbose mode, and did not handle 308 at all.

r241840 (PR 172451) added support for 308, but with the same bug.

Correctly handle both by recognizing them as redirects in all places
where we check the HTTP result code.

PR: 112515 173451 209546
Submitted by: novel@
MFC after: 1 week

Use strlcpy() instead of strncpy() to copy the string returned by
setlocale() so that static analyzers know that the string is NUL
terminated. This was causing a false positive in Coverity even
though the longest string returned by setlocale() is ENCODING_LEN
(31) and we are copying into a 64 byte buffer. This change is also
a bit of an optimization since we don't need the strncpy() feature
of padding the rest of the destination buffer with NUL characters.

Reported by: Coverity
CID: 974654

Fix double-free error: r289419 moved all error handling in http_connect()
to the end of the function, but did not remove a fetch_close() call which
was made redundant by the one in the shared error-handling code.

PR: 206774
Submitted by: Christian Heckendorf <heckendorfc@gmail.com>
MFC after: 3 days

As a followup to r292330, standardize on size_t and add a few comments.

Reset bufpos to 0 immediately after refilling the buffer. Otherwise, we
risk leaving the connection in an indeterminate state if the server fails
to send a chunk delimiter. Depending on the application and on the sizes
of the preceding chunks, the result can be anything from missing data to a
segfault. With this patch, it will be reported as a protocol error.

PR: 204771
MFC after: 1 week

Fix buildworld after r291453, similar to r284346: url->user and url->pwd
are arrays, so they can never be NULL.

Reported by: many
Pointy hat to: des

Use .netrc for HTTP sites and proxies, not just FTP.

PR: 193740
Submitted by: TEUBEL György <tgyurci@gmail.com>
MFC after: 1 week

Fix two bugs in HTTPS tunnelling:
- If the proxy returns a non-200 result, set the error code accordingly
so the caller / user gets a somewhat meaningful error message.
- Consume and discard any HTTP response header following the result line.

PR: 194483
Tested by: Fabian Keil <fk@fabiankeil.de>
MFC after: 1 week

Remove unused variable to silence clang warning.

Differential Revision: D2683
Reviewed by: rodrigc, bapt

Fix the following clang 3.7.0 warnings in lib/libfetch/http.c:

lib/libfetch/http.c:1628:26: error: address of array 'purl->user'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.user = purl->user ?
~~~~~~^~~~ ~
lib/libfetch/http.c:1630:30: error: address of array 'purl->pwd'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.password = purl->pwd?
~~~~~~^~~~
lib/libfetch/http.c:1657:25: error: address of array 'url->user'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.user = url->user ?
~~~~~^~~~ ~
lib/libfetch/http.c:1659:29: error: address of array 'url->pwd'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.password = url->pwd ?
~~~~~^~~ ~
lib/libfetch/http.c:1669:25: error: address of array 'url->user'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.user = url->user ?
~~~~~^~~~ ~
lib/libfetch/http.c:1671:29: error: address of array 'url->pwd'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.password = url->pwd ?
~~~~~^~~ ~

Since url->user and url->pwd are arrays, they can never be NULL, so the
checks can be removed.

Reviewed by: bapt
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D2673

Add support for arbitrary http requests

Submitted by: Alex Hornung <alex@alexhornung.com>
Reviewed by: des
Obtained from: Dragonfly
MFC after: 3 week

Remove unnecessary semicolons

Patch by Sascha Wildner <saw@online.de> for Dragonfly

Reviewed by: des
Obtained from: Dragonfly
MFC after: 1 week

Use NULL instead of 0

Patch by Sascha Wildner <saw@online.de> for Dragonfly

Reviewed by: des
Obtained from: Dragonfly
MFC after: 1 week

If HTTP_USER_AGENT is defined but empty, don't send User-Agent at all.

PR: 184507
Submitted by: jbeich@tormail.org (with modifications)
MFC after: 1 week

Support Last-Modified behind proxies which return UTC instead of GMT.

The standard states that GMT must be used, but that UTC is equivalent. Still
parse UTC as otherwise this causes problems for pkg(8). It will refetch
the repository every time 'pkg update' or other remote operations
are used behind these proxies.

RFC2616: "All HTTP date/time stamps MUST be represented in Greenwich Mean
Time (GMT), without exception. For the purposes of HTTP, GMT is exactly equal
to UTC (Coordinated Universal Time).""

Approved by: bapt (mentor)
Reviewed by: des, peter
Sponsored by: EMC / Isilon Storage Division
MFC after: 1 week

Bump copyright dates

r261230 broke the cases where the amount of data to be read is not
known in advance, or where the caller doesn't care and just keeps
reading until it hits EOF.

In fetch_read(): the socket is non-blocking, so read() will return 0
on EOF, and -1 (errno == EAGAIN) when the connection is still open but
there is no data waiting. In the first case, we should immediately
return 0. The EINTR case was also broken, although not in a way that
matters.

In fetch_writev(): use timersub() and timercmp() as in fetch_read().

In http_fillbuf(): set errno to a sensible value when an invalid chunk
header is encountered.

In http_readfn(): as in fetch_read(), a zero return from down the
stack indicates EOF, not an error. Furthermore, when io->error is
EINTR, clear it (but no errno) before returning so the caller can
retry after dealing with the interrupt.

MFC after: 3 days

Solve http buffering issues and hangs once and for all (hopefully!) by
simply not trying to return exactly what the caller asked for - just
return whatever we got and let the caller be the judge of whether it
was enough. If an error occurs or the connection times out after we
already received some data, return a short read, under the assumption
that the next call will fail or time out before we read anything.

As it turns out, none of the code that calls fetch_read() assumes an
all-or-nothing result anyway, except for a couple of lines where we
read the CR LF at the end of a hunk in HTTP hunked encoding, so the
changes outside of fetch_read() and http_readfn() are minimal.

While there, replace select(2) with poll(2).

MFC after: 3 days

Even though it doesn't really make sense in the context of a CONNECT
request, RFC 2616 14.23 mandates the presence of the Host: header in
all HTTP 1.1 requests.

PR: kern/181445
Submitted by: Kimo <kimor79@yahoo.com>
MFC after: 3 days

Include an Accept header in requests.

PR: kern/180917
MFC after: 1 week

Implement certificate verification, and many other SSL-related
imrovements; complete details in the PR.

PR: kern/175514
Submitted by: Michael Gmelin <freebsd@grem.de>
MFC after: 1 week

Use the correct request syntax for proxied (tunneled) HTTPS requests.

PR: bin/180666
MFC after: 3 days

Use the CONNECT method to proxy HTTPS connections through HTTP proxies.

PR: bin/80176
Submitted by: Yuichiro NAITO <naito.yuichiro@gmail.com>

Fix weird indentation.

Implement HTTP 305 redirect handling.

PR: 172452
Submitted by: gcooper
Reviewed by: des
Approved by: cperciva
MFC after: 1 week

Don't deny non-temporary redirects if the -A option is set (per
the man page) [0]

While here add support for draft-reschke-http-status-308-07

PR: 172451 [0]
Submitted by: gcooper [0]
Reviewed by: des
Approved by: cperciva
MFC after: 1 week

Be a bit more lenient in the maximum number of redirects allowed.
Chrome and Firefox have a limit of 20. IE has a limit of 8.

Reviewed by: des
Approved by: cperciva
MFC after: 3 days

Use libmd if and only if OpenSSL is not available.

PR: bin/171402
MFC after: 3 days

Don't reuse credentials if redirected to a different host.

Submitted by: Niels Heinen <heinenn@google.com>
MFC after: 3 weeks

Fix two issues related to the use of SIGINFO in fetch(1) to display
progress information. The first is that fetch_read() (used in the HTTP
code but not the FTP code) can enter an infinite loop if it has previously
been interrupted by a signal. The second is that when it is interrupted,
fetch_read() will discard any data it may have read up to that point.
Luckily, both bugs are extremely timing-sensitive and therefore difficult
to trigger.

PR: bin/153240
Submitted by: Mark <markjdb@gmail.com>
MFC after: 3 weeks

latin1 -> utf8

Update copyright dates and strip my middle name.

Increase WARNS to 4.

Mechanical whitespace cleanup.

Increase WARNS to 3.

Fix a couple of embarrassing mistakes in the previous commit.

Submitted by: Dimitry Andric <dimitry@andric.com>

If the A flag is supplied, http_request() will attempt the request only
once, even if authentication is required, instead of retrying with the
proper credentials. Fix this by bumping the countdown if the origin or
proxy server requests authentication so that the initial unauthenticated
request does not count as an attempt.

PR: 148087
Submitted by: Tom Evans <tevans.uk@googlemail.com>
MFC after: 2 weeks

Add HTTP digest authentication.

Submitted by: Jean-Francois Dockes <jf@dockes.org>
Forgotten by: des (repeatedly)

Add support for HTTP 1.1 If-Modified-Since behavior.

fetch(1) accepts a new argument -i <file> that if specified will cause
the file to be downloaded only if it is more recent than the mtime of
<file>.

libfetch(3) accepts the mtime in the url structure and a flag to
indicate when this behavior is desired.

PR: bin/87841
Submitted by: Jukka A. Ukkonen <jau@iki.fi> (partially)
Reviewed by: des, ru
MFC after: 3 weeks

Don't fail mistakenly with -r when we already have the whole file.

Reviewed by: des

Use memcpy(3) instead of the BSD-specific bcopy(3).

Submitted by: Joerg Sonnenberger <joerg@britannica.bec.de>
MFC after: 2 weeks

Add necessary cast for tolower() argument.

Submitted by: Joerg Sonnenberger <joerg@britannica.bec.de>
MFC after: 1 week

As several people pointed out, I did all the ctype casts the wrong
way (not for the first time...)

Noticed by: bde, ru ++
MFC after: 1 week

Add support for the NO_PROXY / no_proxy environment variable as used by
lynx, curl etc. Note that this patch differs significantly from that
in the PR, as the submitter refined it after submitting the PR.

PR: 110388
Submitted by: Alexander Pohoyda <alexander.pohoyda@gmx.net>
MFC after: 3 weeks

Old patch I had lying around: correctly cast the argument to is*().
IWBNI gcc could warn about this the way it warns about printf() abuse.

MFC after: 1 week

Clean up namespace violations.

MFC after: 1 week

Handle temporary redirects (HTTP status code 307)

PR: kern/112515
Submitted by: Ryan C. Gordon <icculus@icculus.org>
MFC after: 3 weeks

Ignore HTTP_PROXY if it is defined but empty. This was already handled
correctly in the case of FTP_PROXY, because an empty FTP_PROXY has a
specific meaning ("don't use any proxy at all for ftp, even if HTTP_PROXY
is defined"), while an empty HTTP_PROXY has no meaning at all.

PR: bin/85185
Submitted by: Conall O'Brien <conallob=freebsd@maths.tcd.ie>
MFC after: 2 weeks

Set the TCP_NODELAY socket option and clear TCP_NOPUSH in order to flush
any pending HTTP request rather than calling shutdown(2) with SHUT_WR.
This makes libfetch (and thus fetch(1)) work again with Squid proxies
configured to not allow half-closed connections.

Reported by: Pawel Worach (pawel.worach AT telia DOT com)

Set TCP_NOPUSH on HTTP requests, reducing the number of round-trips
necessary to establish each connection.

MFC after: 2 weeks

Update copyright years.

Don't forget to allocate space for the terminating NUL when converting to
base 64.

PR: misc/70022
Submitted by: Herve Masson <herve-bsdbt@mindstep.com>
MFC after: 3 days

Slight tweak to previous commit: don't forget to call _http_seterr() if
we get a genuine 416 reply.

When restarting a transfer that has already completed, the server will
reply with a 416 error code (requested range not satisfiable) because
we ask it to start at the end of the file. Handle this gracefully by
considering a 416 reply a success if the requested offset exactly
matches the length of the file and the requested length is zero.

Re-wrap some comments.

Don't dereference flags if NULL (see http.c rev 1.87)

Don't parse the proxy URL unless we're actually going to use it. No real
functional difference, but debugging output will be less confusing.

style(9): add parentheses to sizeof even when not strictly required.

MFC after: 3 days

A negative offset means "get it all".

Implement and document support for an HTTP_REFERER environment variable.

PR: 28171
Submitted by: Andre Albsmeier <andre.albsmeier@mchp.siemens.de>
Approved by: re (bmah)
MFC after: 1 week

Use __func__ and break a long line.

Don't forget to set the error flag when _fetch_read() returns -1.

Eliminate two cases of undefined behaviour: total in _fetch_write() was
not initialized before use, and _http_growbuf() did not return a value
on success.

Reported by: Peter Edwards <pmedwards@eircom.net>
MFC after: 2 weeks

HTTP authentication got broken in rev. 1.49.

Don't try to dereference conn when we know it's NULL.

Make SSL support conditional on NOCRYPT.

Add SSL support + slight cleanup.

Submitted by: Henry Whincup <henry@techiebod.com> (in principle)

Wrap everything in struct connection, and enforce timeouts everywhere
(except for DNS operations). Always use funopen() for HTTP, to support
both timeouts and SSL.

Rename struct cookie to struct httpio to avoid confusion (it's not an HTTP
cookie) and increase symmetry with equivalent FTP code.

Add comments to struct cookie.

Fix a bug I introduced in the chunk decoder in the previous commit..

First step towards SSL support: wrap connections in a 'struct connection'
which contains the socket descriptor, the input buffer and (yet unused)
SSL state variables. This has the neat side effect of greatly improving
reentrance (though we're not *quite* there yet) and opening the door to
HTTP connection caching.

This commit is inspired by email conversations with and patches from
Henry Whincup <henry@techiebod.com> last fall.

Do not use __progname directly (except in [gs]etprogname(3)).
Also, make an internal _getprogname() that is used only inside
libc. For libc, getprogname(3) is a weak symbol in case a
function of the same name is defined in userland.

Reindent, and add parentheses to return statements. Some functions in
ftp.c and http.c now have exceedingly long lines due to deep nesting;
this will be corrected by reorganizing the code in a later revision.

Check the return value from read() when reading the CR/LF at the end of a
chunk.

PR: bin/33608
MFC after: 2 weeks

In verbose mode, display the full error message from the server, stripping
it of HTML tags and comments.

PR: bin/32989
MFC after: 1 week

Remove VT100 escapes from debugging messages now that they're enabled by
default.

PR: 32988
MFC after: 3 days

Conditionalize some debugging code that didn't use the DEBUG macro.

MFC after: 1 week

Properly handle being redirected to an FTP URL.

Also fix a couple of not-so-minor bugs:
- missing case for HTTP_SEE_OTHER
- incorrect definition of HTTP_NEED_PROXY_AUTH

PR: 20259
MFC after: 2 weeks

Tons of type, style and warning fixes that have been rotting in my tree for
ages - some of which wouldn't be necessary if gcc wasn't broken or TPTB were
willing to do something (-fno-builtin) about it.

Add __FBSDID()s to libfetch

Implement a HTTP_USER_AGENT environment variable.

PR: 27669
Submitted by: Eoin Lawless <eoin@maths.tcd.ie>

Add rudimentary support for an authentication callback function.

Apply 'const' liberally.

Fix some other minor glitches.

Support lower-case versions of the proxy environment variables.

PR: bin/25494

Use the documented (and historical) defaults. Centralize the decision logic
in order to avoid this bug in the future.

Submitted by: se

Use CHECK_FLAG

Rework proxy handling so that proxies can be specified as URLs.
As a side effect, remove a lot of duplicate and now redundant code.

Work around buggy servers such as NCSA httpd which send an incomplete
HTTP-Version on the Status-Line (see RFC2616 sections 3.1 and 6.1).

Fix signedness bogon.

Centralize the default port finding code.
Work around YA Apache bug: don't send port in Host: header if it's the
default port.

Ignore environment variables that are set but empty.

Sort out the size / length confusion. Always try to report the *real* file
size in the url_stat structure, no matter how much of it is actually being
sent, and try to detect inconsistencies between sizes.

Rearrange the request loop to avoid having to add meaningless code just to
silence compiler warnings.

Switch to a more sensible and consistent interface for the _http_parse*()
functions.

Don't try to skip to the requested offset if the server returns more data
than requested. Instead, inform the caller of the real offset by modifying
the offset field in the original struct url, and let him decide how to handle
the situation.

Introduce fetchXGet*(), which combine the functionalities of fetchGet*() and
fetchStat*(). In most cases, either fetchGet*() or fetchXGet*() is a wrapper
around the other; in all cases, calling fetchGet*() is identical to calling
fetchXGet*() with the second argument set to NULL.

Don't forget to set file descriptor to -1 after closing it, since the code
outside the loop inspects it to determine whether or not we succeeded in
retrieving the requested document. This fixes a bug where fetchGetHTTP()
would return a FILE with an invalid file descriptor if it hit the redirect
limit without locating the requested document.

#ifdef DEBUG -> #ifndef NDEBUG

Pointed out by: ache

Reinsert the MIT copyright, which applies to the base64 code and was
left out by accident during the rewrite.

Fix a bug (misplaced continue) that caused redirects to fail. Lots of code
moved around, but the acutal functional changes are small.
Add support for site-internal redirects (where the Location: header gives a
path instead of an absolute URI)

Pointed out by: kuriyama

Extensive rewrite of the HTTP code. The new code is significantly cleaner and
more robust, and somewhat more efficient. It also handles authorization and
redirects properly, and supports timeouts like the FTP code.

Many thanks to Umemoto-san for his assistance with IPv6 support, both here
and in other parts of libfetch.

Rework the authorization code.
Only send absolute URI if connected to a proxy, since Apache doesn't always
understand absolute URIs.
Clean up some of the debugging output.

Forced commit to note that I meant authorization, not authentication.

Fix basic authentication, and add proxy authentication.

Submitted by: se

Don't incorrectly report a protocol error when we get a 302 and redirects
are disabled.

Hackish support for 302 redirect.

IPv6 support.

Better handling of some boundary conditions.

Submitted by: ume

Make HTTP_PROXY work for FTP.

Reported by: Ben Smithurst <ben@scientia.demon.co.uk>
Reviewed by: des

Remove unused 'verbose'.
Initialize the struct url_stat at the beginning of fetchStatHTTP().

Reorganize some of the http code and split it into more functions.
Implement fetchStatHTTP().
Unbungle struct url, and add fetchFreeURL().
Document it.

Implement restart

Use sizeof more consistently.
Outline sanity checking of server port spec.

Patches I've had lying around for several months:
* Add the 'h' ftp flag (allocate local port in high range)
* Add the 'd' flag (use direct connection even if proxy is defined)
* Make sure flags != NULL before calling strchr().
* Minor changes to some comments.

$Id$ -> $FreeBSD$

Implement and document file list retrieval.

Don't specify a language to compile_et.
Rename fetchConnect to _fetch_connect since it's internal.

Remove unused header files. Fix a few shadowing bugs. Improve error reporting.

Add verbose flag, and support functions.
Brucify the Makefile.
Differentiate atime and mtime in fetch*Stat().
Fix a few pointer bugs.
Tweak some error messages.
Don't #include sys/param.h and stdio.h in fetch.h.
Document that sys/param.h and stdio.h must be #included before fetch.h.

Second of a series of cleanups to libfetch.

This commit introduces the following features:

a) the fetchStat*() functions, which return meta-information for a
document, such as size, modification time, etc.

b) the use of the com_err(3) facilities to report errors.

It also fixes a bunch of style bugs and a few logic bugs and somewhat
improves the man page.

Changed files, in alphabetical order:

Makefile:
Don't generate macros in {ftp,http}err.c.

Generate category fields for the error message lists.

Compile the error table.

Install fetch_err.h along with fetch.h.

common.c:
Remove the _netdb_errstring() macro, and add FETCH_ERR_NETDB to the
error code in the _netdb_seterr() macro.

Add categories to the _netdb_errlist table.

Report errors through the Common Error library.

common.h:
Add the DEBUG macros.

Add prototype for fetchConnect().

Remove the prototype for _fetch_errstring(), which is local to common.c

Add a categroy field to struct fetcherr, and define constants for
error categories.

Define macros for _{url,netdb,ftp,http}_seterr().

errors.et: (new file)
List error categories.

fetch.3:
Document the fetchStat*() functions.

Move the "unimplemented functionality" comments from NOTES to BUGS.

Document that applications which use libfetch must also use
libcom_err, and list existing error codes.

Undocument fetchLastErr{Code,String}.

Remove the (empty) DIAGNOSTICS section.

Mention Eugene Skepner in the AUTHORS section.

fetch.c:
Move the DEBUG macros to common.c

Add fetchStat() and fetchStatURL().

Generate error messages for URL parser errors, and fix a minor bug
in the parser.

Use 'struct url' instead of 'url_t'.

Remove fetchLastErr{Code,String}.

fetch.h:
Use 'struct url' instead of 'url_t', and remove the typedef.

Define struct url_stat (used by fetchStat()).

Add prototypes for fetchStat*().

Remove the declarations for fetchLastErr{Code,String}.

Include fetch_err.h.

fetch_err.et: (new file)
Error table for libfetch.

file.c:
Add fetchStatFile().

Use 'struct url' instead of 'url_t'.

ftp.c:
Add fetchStatFTP().

Use 'struct url' instead of 'url_t'.

Don't use fetchLastErrCode.

ftp.errors:
Add categories to all error messages.

http.c:
Add fetchStatHTTP().

Use 'struct url' instead of 'url_t'.

Don't use fetchLastErr{Code,Text}.

http.errors:
Add categories to all error messages.

Prompted by: jkh and Eugene Skepner
Numerous sugestions from: Garett Wollman and Eugene Skepner

First of a series of cleanups to libfetch. Changed files, in
alphabetical order:

Makefile:
Add common.c to SRCS.

Make debugging easier by making 'CFLAGS += -DNDEBUG' conditional on DEBUG

Don't declare struct {ftp,http}err in {ftp,http}err.c; use struct fetcherr
instead.

README:
Remove the todo list, which is out of date anyway.

common.c: (new file)
Gather utility functions in this file.

Merge the error reporting functions intp _fetch_errstring(),
_fetch_seterr() and _fetch_syserr().

Set fetchLastErrCode and fetchLastErrText appropriately when fetchConnect
fails.

common.h: (new file)
Gather internal prototypes and structures in this files.

fetch.3:
Undocument fetchFreeURL().

Document a few more known bugs.

Document fetchLastErrCode and fetchLastErrText.

fetch.c:
Add descriptive comments to all functions that lacked them.

Move fetchConnect() to common.c.

Obviate the need for fetchFreeURL(), and remove it.

fetch.h:
Modify struct url_t so the document part is at the end.

ftp.c:
Remove code that is duplicated elsewhere.

http.c:
Remove code that is duplicated elsewhere.

Prompted by: jkh

Commit a bunch of patches that have been accumulating:

- Fix the README to reflect the new status of the ftp code.
- Change tons of 'if (xxx < 0)' to 'if (xxx == -1)'
- Add two new interface functions
- Fix the Makefile so it actually works (yay!)

Now the manpage is lagging even further behind... :( Next on the todo
list is to clean up the http code.

Base64 code (and the MIT copyright) moved to http.c
FTP STORe and APPEnd added.
FTP proxy support added (untested).

FTP retrieve works.

Too many changes to list. Basically, FTP is nearly there and error
reporting is kinda sorted out. Now HTTP needs to catch up...

This commit was generated by cvs2svn to compensate for changes in r37535,
which included commits to RCS files with non-trunk default branches.

Imported libfetch into the tree. It compiles, but there's still some
work to do. I especially need help with the man page.