lib/libfetch/common.c: remove an extra semicolon

Signed-off-by: rilysh <nightquick@proton.me>
Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/959

lib: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

libfetch: don't rely on ca_root_nss for certificate validation

Before certctl(8), there was no system trust store, and libfetch
relied on the CA certificate bundle from the ca_root_nss port to
verify peers.

We now have a system trust store and a reliable mechanism for
manipulating it (to explicitly add, remove, or revoke certificates),
but if ca_root_nss is installed, libfetch will still prefer that to
the system trust store.

With this change, unless explicitly overridden, libfetch will rely on
OpenSSL to pick up the default system trust store.

PR: 256902
MFC after: 3 days
Reviewed by: kevans
Differential Revision: https://reviews.freebsd.org/D42059

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

libfetch: remove all old OpenSSL support

This change removes pre-OpenSSL 1.1 supporting code and removes/adjusted
preprocessor conditionals which were tautilogically true as FreeBSD main
has shipped with OpenSSL 1.1+ for some time.

Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D40711

libfetch: do not call deprecated OpenSSL functions

As of OpenSSL 1.1 SSL_library_init() and SSL_load_error_strings() are
deprecated. There are replacement initialization functions but they do
not need to be called: "As of version 1.1.0 OpenSSL will automatically
allocate all resources that it needs so no explicit initialisation is
required."

Wrap both calls in an OPENSSL_VERSION_NUMBER block.

PR: 271615
Reviewed by: Pierre Pronchery <pierre@freebsdfoundation.org>
Event: Kitchener-Waterloo Hackathon 202305
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40265

fetch: support EAI_ADDRFAMILY error, correct two error messages

With the change to return EAI_ADDRFAMILY from getaddrinfo(), fetch
would print "Unknown resolver error" for that error. Add that error
and its string to libfetch's table, using an #ifdef just in case.
Correct error strings for EAI_NODATA (although it is currently unused)
and EAI_NONAME. Should maybe rework the code to use gai_strerror(3),
but that doesn't map directly, and the current strings are shortened.

Reviewed in https://reviews.freebsd.org/D37139 with related changes.

Reviewed by: bz
MFC after: 1 month

libfetch: Use memcpy in place of an odd strncpy.

The length passed to strncpy is the length of the source string, not
the destination buffer. This triggers a non-fatal warning in GCC 12.
Hoewver, the code is also odd. It is really just a memcpy of the
string without its nul terminator. For that use case, memcpy is
clearer.

Reviewed by: imp, emaste
Differential Revision: https://reviews.freebsd.org/D36824

Remove support for SSLv3 from fetch(3).

Support for SSLv3 was already removed from OpenSSL (r361392).

Differential Revision: https://reviews.freebsd.org/D24947

Replace literal uses of /usr/local in C sources with _PATH_LOCALBASE

Literal references to /usr/local exist in a large number of files in
the FreeBSD base system. Many are in contributed software, in configuration
files, or in the documentation, but 19 uses have been identified in C
source files or headers outside the contrib and sys/contrib directories.

This commit makes it possible to set _PATH_LOCALBASE in paths.h to use
a different prefix for locally installed software.

In order to avoid changes to openssh source files, LOCALBASE is passed to
the build via Makefiles under src/secure. While _PATH_LOCALBASE could have
been used here, there is precedent in the construction of the path used to
a xauth program which depends on the LOCALBASE value passed on the compiler
command line to select a non-default directory.

This could be changed in a later commit to make the openssh build
consistently use _PATH_LOCALBASE. It is considered out-of-scope for this
commit.

Reviewed by: imp
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D26942

fetch(3): plug some leaks

In the successful case, sockshost is not freed prior to return.

The failure case can now be hit after fetch_reopen(), which was not true
before. Thus, we need to make sure to clean up all of the conn resources
which will also close sd. For all of the points prior to fetch_reopen(), we
continue to just close sd.

CID: 1419598, 1419616

fetch(3): don't leak sockshost on failure

fetch_socks5_getenv will allocate memory for the host (or set it to NULL) in
all cases through the function; the caller is responsible for freeing it if
we end up allocating.

While I'm here, I've eliminated a label that just jumps to the next line...

fetch(3): fix regression in IPv6:port spec from r357977

In case the port was specified, we never actually populated *host. Do so
now.

Pointy hat: kevans

fetch(3): move bits of fetch_socks5_getenv around

This commit separates out port parsing and validation from grabbing the host
from the env var. The only related bit really is that we need to be more
specific with the delimiter in the IPv6 case.

fetch(3): Add SOCKS5 support

This change adds SOCKS5 support to the library fetch(3) and updates the man
page.

Details: Within the fetch_connect() function, fetch(3) checks if the
SOCKS5_PROXY environment variable is set. If so, it connects to this host
rather than the end-host. It then initializes the SOCKS5 connection in
accordance with RFC 1928 and returns the resulting conn_t (file descriptor)
for usage by the regular FTP/HTTP handlers.

Design Decision: This change defaults all DNS resolutions through the proxy
by sending all IPs as hostnames. Going forward, another feature might be to
create another environmental variable to toggle resolutions through the
proxy or not..

One may set the SOCKS5_PROXY environment variable in any of the formats:

SOCKS5_PROXY=proxy.example.com
SOCKS5_PROXY=proxy.example.com:1080
SOCKS5_PROXY=192.0.2.0
SOCKS5_PROXY=198.51.100.0:1080
SOCKS5_PROXY=[2001:db8::1]
SOCKS5_PROXY=[2001:db8::2]:1080

Then perform a request with fetch(1).

(note by kevans)
I've since been informed that Void Linux/xbps has a fork of libfetch that
also implements SOCKS5. I may compare/contrast the two in the mid-to-near
future.

Submitted by: Farhan Khan <farhan farhan codes>
Differential Revision: https://reviews.freebsd.org/D18908

[libfetch] Fix compilation with WITHOUT_CRYPT.

Improve URL parsing. In particular, convert scheme and host to lowercase.

MFC after: 1 week

Make libfetch buildable.

Fix an inverted conditional in the netrc code, which would ignore the
value of $HOME and always use the home directory from the passwd
database, unless $HOME was unset, in which case it would use (null).

While there, clean up handling of netrcfd and add debugging aids.

MFC after: 3 weeks

Use __VA_ARGS__ to simplify the DEBUG macro.

MFC after: 3 weeks

lib: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

In fetch_resolve(), if the port number or service name is included in
the host argument (e.g. "www.freebsd.org:443"), the service pointer,
which is supposed to point to the port or service part, instead points
to the separator, causing getaddrinfo() to fail.

Note that I have not been able to trigger this bug with fetch(1), nor
do I believe it is possible, as libfetch always parses the host:port
specification itself. I discovered it when I copied fetch_resolve()
into an unrelated project.

MFC after: 3 days

r308996 broke IP literals by assuming that a colon could only occur as
a separator between host and port, and using strchr() to search for it.
Rewrite fetch_resolve() so it handles bracketed literals correctly, and
remove similar code elsewhere to avoid passing unbracketed literals to
fetch_resolve(). Remove #ifdef INET6 so we still parse IP literals
correctly even if we do not have the ability to connect to them.

While there, fix an off-by-one error which caused HTTP 400 errors to be
misinterpreted as redirects.

PR: 217723
MFC after: 1 week
Reported by: bapt, bz, cem, ngie

libfetch: extra bounds checking through reallocarray(3).

Reviewed by: des
MFC after: 1 week

Properly indent a default: label and avoid crashing when running
under -v but cannot connect due to trying to print an int as %s [1].

Reported by: andrew [1]
MFC after: 3 days

Add a file descriptor in struct url for netrc

When using libfetch in an application that drops privileges when fetching
like pkg(8) then user complain because the application does not read anymore
${HOME}/.netrc. Now a caller can prepare a fd to the said file and manually
assign it to the structure.

It is also a first step to allow to capsicumize libfetch applications

Reviewed by: allanjude, des
Approved by: des
Differential Revision: https://reviews.freebsd.org/D9678

More debugging code I missed in r309051.

Reported by: jbeich, jkim

Remove debugging code.

Refactor fetch_connect() and fetch_bind() to improve readability and avoid
repeating the same DNS lookups.

MFC after: 3 weeks

Don't leak addrinfo in fetch_bind()

Reported by: Coverity
CID: 1225038
MFC after: 1 week

lib: minor spelling fixes in comments.

No functional change.

Test for /etc/ssl/cert.pem existence to avoid masking SSL_CA_CERT_PATH

Prior to this patch, unless SSL_CA_CERT_FILE is set in the environment,
libfetch will set the CA file to "/usr/local/etc/cert.pem" if it exists,
and to "/etc/ssl/cert.pem" otherwise. This has the consequence of
masking SSL_CA_CERT_PATH, because OpenSSL will ignore the CA path if a CA
file is set but fails to load (see X509_STORE_load_locations()).

While here, fall back to OpenSSL defaults if neither SSL_CA_CERT_FILE nor
SSL_CA_CERT_PATH are set in the environment, and if neither of the
libfetch default CA files exists.

PR: 193871
Submitted by: John W. O'Brien <john@saltant.com>
Approved by: des
MFC after: 1 week

Fix non-POSIX-compliant use of getaddrinfo in libfetch

Submitted by: Boris Kolpackov <boris@codesynthesis.com>
Reviewed by: bapt
Approved by: bapt
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D3724

Remove defunct SSLv2 support from fetch(1) and fetch(3).

As pointed out by several people, r273114 was incorrect: it unconditionally
disabled everything except TLS 1.0. Replace it with a more carefully
wrought patch:

- Switch the default for SSLv3 from on to off
- Add environment variables to control TLS 1.1 and 1.2
- In verbose mode, report which version is used
- Update the man page to reflect these changes.

MFC after: 1 week

Drop support for SSLv3.

MFC after: 1 week

Add a comment to explain the EAGAIN is only there for POSIX compliance

Resquested by: kib
Reviewed by: des

Support EAGAIN in fetch_writev

Reviewed by: des
Approved by: des

Look for root certificates in /usr/local/etc/ssl before /etc/ssl.

MFH: 1 week

Bump copyright dates

r261230 broke the cases where the amount of data to be read is not
known in advance, or where the caller doesn't care and just keeps
reading until it hits EOF.

In fetch_read(): the socket is non-blocking, so read() will return 0
on EOF, and -1 (errno == EAGAIN) when the connection is still open but
there is no data waiting. In the first case, we should immediately
return 0. The EINTR case was also broken, although not in a way that
matters.

In fetch_writev(): use timersub() and timercmp() as in fetch_read().

In http_fillbuf(): set errno to a sensible value when an invalid chunk
header is encountered.

In http_readfn(): as in fetch_read(), a zero return from down the
stack indicates EOF, not an error. Furthermore, when io->error is
EINTR, clear it (but no errno) before returning so the caller can
retry after dealing with the interrupt.

MFC after: 3 days

Solve http buffering issues and hangs once and for all (hopefully!) by
simply not trying to return exactly what the caller asked for - just
return whatever we got and let the caller be the judge of whether it
was enough. If an error occurs or the connection times out after we
already received some data, return a short read, under the assumption
that the next call will fail or time out before we read anything.

As it turns out, none of the code that calls fetch_read() assumes an
all-or-nothing result anyway, except for a couple of lines where we
read the CR LF at the end of a hunk in HTTP hunked encoding, so the
changes outside of fetch_read() and http_readfn() are minimal.

While there, replace select(2) with poll(2).

MFC after: 3 days

Fix format string.

Submitted by: Jörg Sonnenberger <joerg@NetBSD.org>
MFC after: 1 week

Fix build with GCC

SSL_set_tlsext_host_name(3) internally does not modify the host buffer
pased to it. So it is safe to DECONST the struct url* here.

Reported by: gjb
Approved by: bapt (implicit)
MFC after: 1 week
X-MFC-With: r258347

Support SNI in libfetch

SNI is Server Name Indentification which is a protocol for TLS that
indicates the host that is being connected to at the start of the
handshake. It allows to use Virtual Hosts on HTTPS.

Submitted by: sbz
Submitted by: Michael Gmelin <freebsd@grem.de> [1]
PR: kern/183583 [1]
Reviewed by: des
Approved by: bapt
MFC after: 1 week

Implement certificate verification, and many other SSL-related
imrovements; complete details in the PR.

PR: kern/175514
Submitted by: Michael Gmelin <freebsd@grem.de>
MFC after: 1 week

libfetch: Avoid SIGPIPE on network connections.

To avoid unexpected process termination from SIGPIPE when writing to a
closed network connection, enable SO_NOSIGPIPE on all network connections.

The POSIX standard MSG_NOSIGNAL is not used since it requires modifying all
send calls to add this flag. This is particularly nasty for SSL connections.

Reviewed by: des
Tested by: bapt
MFC after: 5 days

Since the socket is non-blocking, it is necessary to use select(2) even
when there is no timeout, because read(2) will return immediately if there
is no data waiting in the TCP buffer, causing fetch_read() to busy-loop on
slow connections.

MFC after: 3 weeks
Noticed by: Yanhui Shen <shen.elf@gmail.com>

Fix two nits in previous commit pointed out by pjd@.

MFC after: 3 weeks

Fix two issues related to the use of SIGINFO in fetch(1) to display
progress information. The first is that fetch_read() (used in the HTTP
code but not the FTP code) can enter an infinite loop if it has previously
been interrupted by a signal. The second is that when it is interrupted,
fetch_read() will discard any data it may have read up to that point.
Luckily, both bugs are extremely timing-sensitive and therefore difficult
to trigger.

PR: bin/153240
Submitted by: Mark <markjdb@gmail.com>
MFC after: 3 weeks

latin1 -> utf8

Update copyright dates and strip my middle name.

Mark all socket and file descriptors close-on-exec.

PR: bin/151866
MFC after: 3 weeks

Move variable declarations into the conditional block where they are
used, to fix warning if WITH_SSL is not set.

Submitted by: Sean Bruno
MFC after: 1 week

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Redo fetch_read() using non-blocking sockets. This is necessary to
avoid a hang in the SSL case if the server sends a close notification
before we are done reading. In the non-SSL case, it can provide a
minor (but probably not noticeable) performance improvement for small
transfers.

MFC after: 3 weeks

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

MFC 198339:
Fix the NO_PROXY handling.

PR: 139751

PR: 139751
Approved by: des
Obtained from: Xavier Heiny <xavier.heiny@netasq.com>
MFC after: 3 weeks

- Import the HEAD csup code which is the basis for the cvsmode work.

Fix one-byte buffer overflow: NUL gets written to the buffer, but isn't
counted in the width specification in scanf.

This is not a security problem, since this function is only used to
parse a user's configuration file.

Submitted by: Joerg Sonnenberger
Obtained from: dragonflybsd
MFC after: 1 week

s/wait/delta/ to avoid namespace collision.

MFC after: 2 weeks

Use memcpy(3) instead of the BSD-specific bcopy(3).

Submitted by: Joerg Sonnenberger <joerg@britannica.bec.de>
MFC after: 2 weeks

As several people pointed out, I did all the ctype casts the wrong
way (not for the first time...)

Noticed by: bde, ru ++
MFC after: 1 week

Add support for the NO_PROXY / no_proxy environment variable as used by
lynx, curl etc. Note that this patch differs significantly from that
in the PR, as the submitter refined it after submitting the PR.

PR: 110388
Submitted by: Alexander Pohoyda <alexander.pohoyda@gmx.net>
MFC after: 3 weeks

Clean up namespace violations.

MFC after: 1 week

Fix a memory leak: when freeing the connection structure, don't forget to
free the connection buffer as well.

PR: bin/76153
MFC after: 1 week

Update copyright years.

preparation for RFC3493. EAI_NODATA was deprecated.

Add and document support for a FETCH_BIND_ADDRESS environment variable
specifying a local address to bind sockets to. Caveat: lightly tested.

PR: bin/37572

style(9): add parentheses to sizeof even when not strictly required.

MFC after: 3 days

Fix signed/unsigned comparison warning/error from 'make release'

Experimental support for .netrc.

Set auto-retry mode to avoid some spurious errors.

Submitted by: Andre Albsmeier <andre.albsmeier@siemens.com>
MFC after: 7 days

Fix a bug in fenner's _fetch_writev() patch (rev 1.36)

Submitted by: fenner

Recommit the non-broken parts of 1.34 and 1.37.
Change the type and name of a variable introduced in 1.33.

Reinstate revs 1.35-36 and 1.38. Revisions 1.34 and 1.37 were specifically
the root cause of the bus errors I was experiencing.

Submitted by: fenner
Tested by: obrien
Prompted by: peter

Fix `pkg_add -r' by backing out revs 1.34-1.38.
Revs 1.37-8 produce a bus error in some environments.
Revs 1.34-6 do not bus error, but write corrupted files.

Fix an off-by-one error (> where >= should have been used) which caused
_fetch_writev() to incorrectly report EPIPE in certain cases.

Also fix a number of const warnings by using __DECONST(), plus a signed /
unsigned comparison by casting the rhs to ssize_t.

Submitted by: fenner, Craig Rodrigues <rodrigc@attbi.com>

Slight amendment to rev 1.34: instead of considering any short read an
error, only report an error if no data was read at all (unless len was
0 to start with). Otherwise, the final read of practically any transfer
will end in a fatal error.

Introduce _fetch_writev(), which is the conn_t version of writev(2). In
the SSL case, it is no different from the old _fetch_write(), but in the
non-SSL case it uses writev(2) to send the entire vector as a single
packet (provided it can fit in one packet). Implement _fetch_write()
and _fetch_putln() in terms of _fetch_writev().

This should improve performance in the non-SSL case (by reducing protocol
overhead) and solve the problem where too-smart-for-their-own-good
firewalls reject FTP packets that do not end in CRLF.

PR: bin/44123
Submitted by: fenner

Eliminate two cases of undefined behaviour: total in _fetch_write() was
not initialized before use, and _http_growbuf() did not return a value
on success.

Reported by: Peter Edwards <pmedwards@eircom.net>
MFC after: 2 weeks

Back out the previous commit, and fix the bug rather than try to hide its
symptoms: make timeouts and short transfers fatal, and set errno to an
appropriate value (ETIMEDOUT for a timeout, EPIPE for a short transfer).

MFC after: 2 weeks

Fix an infinite loop when _fetch_read() can return 0 (if the
connection is broken), take this into account and return at this
point.

Make _fetch_connect() always set the error code.
Tell ftp that _fetch_connect() always sets the error code (http already knew)

Reintroduce debugging code that somehow got lost in a previous revision.

Add a reference count to struct fetchconn so we don't prematurely close and
free a cached FTP connection.

Make SSL support conditional on NOCRYPT.

Add SSL support + slight cleanup.

Submitted by: Henry Whincup <henry@techiebod.com> (in principle)

Wrap everything in struct connection, and enforce timeouts everywhere
(except for DNS operations). Always use funopen() for HTTP, to support
both timeouts and SSL.

First step towards SSL support: wrap connections in a 'struct connection'
which contains the socket descriptor, the input buffer and (yet unused)
SSL state variables. This has the neat side effect of greatly improving
reentrance (though we're not *quite* there yet) and opening the door to
HTTP connection caching.

This commit is inspired by email conversations with and patches from
Henry Whincup <henry@techiebod.com> last fall.

Switch to a self-starting allocation scheme.

Reindent, and add parentheses to return statements. Some functions in
ftp.c and http.c now have exceedingly long lines due to deep nesting;
this will be corrected by reorganizing the code in a later revision.

Remove VT100 escapes from debugging messages now that they're enabled by
default.

PR: 32988
MFC after: 3 days

Back out part of previous commit which was gcc-centric

Tons of type, style and warning fixes that have been rotting in my tree for
ages - some of which wouldn't be necessary if gcc wasn't broken or TPTB were
willing to do something (-fno-builtin) about it.

Add __FBSDID()s to libfetch

MAXPATHLEN -> PATH_MAX

Apply 'const' liberally.

Fix some other minor glitches.

Don't try to get the proxy port number from /etc/services.

Use the documented (and historical) defaults. Centralize the decision logic
in order to avoid this bug in the future.

Submitted by: se

Introduce the (undocumented) variable fetchRestartCalls, which controls whether
or not interrupted system calls will be restarted. This fixes a bug where
fetch(1) would hang (potentially forever) if a server stopped responding,
because the signal handler would absorb the user's efforts to interrupt the
transfer.

Add _fetch_putln()

Clean up the debugging output

Free chunks obtained by getaddrinfo(3).

Approved by: des

Output newline after debugging message

Dump com_err, it's a stinking crock of shit.

IPv6 support.

Use sizeof more consistently.

More old uncommitted patches: implement timeouts at the protocol level.
Currently only supported for ftp connections.

$Id$ -> $FreeBSD$

Implement and document file list retrieval.

Don't specify a language to compile_et.
Rename fetchConnect to _fetch_connect since it's internal.

Add verbose flag, and support functions.
Brucify the Makefile.
Differentiate atime and mtime in fetch*Stat().
Fix a few pointer bugs.
Tweak some error messages.
Don't #include sys/param.h and stdio.h in fetch.h.
Document that sys/param.h and stdio.h must be #included before fetch.h.

Second of a series of cleanups to libfetch.

This commit introduces the following features:

a) the fetchStat*() functions, which return meta-information for a
document, such as size, modification time, etc.

b) the use of the com_err(3) facilities to report errors.

It also fixes a bunch of style bugs and a few logic bugs and somewhat
improves the man page.

Changed files, in alphabetical order:

Makefile:
Don't generate macros in {ftp,http}err.c.

Generate category fields for the error message lists.

Compile the error table.

Install fetch_err.h along with fetch.h.

common.c:
Remove the _netdb_errstring() macro, and add FETCH_ERR_NETDB to the
error code in the _netdb_seterr() macro.

Add categories to the _netdb_errlist table.

Report errors through the Common Error library.

common.h:
Add the DEBUG macros.

Add prototype for fetchConnect().

Remove the prototype for _fetch_errstring(), which is local to common.c

Add a categroy field to struct fetcherr, and define constants for
error categories.

Define macros for _{url,netdb,ftp,http}_seterr().

errors.et: (new file)
List error categories.

fetch.3:
Document the fetchStat*() functions.

Move the "unimplemented functionality" comments from NOTES to BUGS.

Document that applications which use libfetch must also use
libcom_err, and list existing error codes.

Undocument fetchLastErr{Code,String}.

Remove the (empty) DIAGNOSTICS section.

Mention Eugene Skepner in the AUTHORS section.

fetch.c:
Move the DEBUG macros to common.c

Add fetchStat() and fetchStatURL().

Generate error messages for URL parser errors, and fix a minor bug
in the parser.

Use 'struct url' instead of 'url_t'.

Remove fetchLastErr{Code,String}.

fetch.h:
Use 'struct url' instead of 'url_t', and remove the typedef.

Define struct url_stat (used by fetchStat()).

Add prototypes for fetchStat*().

Remove the declarations for fetchLastErr{Code,String}.

Include fetch_err.h.

fetch_err.et: (new file)
Error table for libfetch.

file.c:
Add fetchStatFile().

Use 'struct url' instead of 'url_t'.

ftp.c:
Add fetchStatFTP().

Use 'struct url' instead of 'url_t'.

Don't use fetchLastErrCode.

ftp.errors:
Add categories to all error messages.

http.c:
Add fetchStatHTTP().

Use 'struct url' instead of 'url_t'.

Don't use fetchLastErr{Code,Text}.

http.errors:
Add categories to all error messages.

Prompted by: jkh and Eugene Skepner
Numerous sugestions from: Garett Wollman and Eugene Skepner

First of a series of cleanups to libfetch. Changed files, in
alphabetical order:

Makefile:
Add common.c to SRCS.

Make debugging easier by making 'CFLAGS += -DNDEBUG' conditional on DEBUG

Don't declare struct {ftp,http}err in {ftp,http}err.c; use struct fetcherr
instead.

README:
Remove the todo list, which is out of date anyway.

common.c: (new file)
Gather utility functions in this file.

Merge the error reporting functions intp _fetch_errstring(),
_fetch_seterr() and _fetch_syserr().

Set fetchLastErrCode and fetchLastErrText appropriately when fetchConnect
fails.

common.h: (new file)
Gather internal prototypes and structures in this files.

fetch.3:
Undocument fetchFreeURL().

Document a few more known bugs.

Document fetchLastErrCode and fetchLastErrText.

fetch.c:
Add descriptive comments to all functions that lacked them.

Move fetchConnect() to common.c.

Obviate the need for fetchFreeURL(), and remove it.

fetch.h:
Modify struct url_t so the document part is at the end.

ftp.c:
Remove code that is duplicated elsewhere.

http.c:
Remove code that is duplicated elsewhere.

Prompted by: jkh