Add -b <buflen> to specify the buffer size.

Don't finish downloading an empty file with 'already done' before it is
created locally.

Fix HTTPS through Proxy.

While a regular HTTP Proxy, requires the absolute URL with protocol
and host part, yyou must only send the relative URL through a
CONNECT tunnel (you are talking to the target server).

Add option sslnoverify to control validation of SSL certificates.
Add netrc processing to fetch-mode (URL on command line) to enable options and autologin
via netrc.
Fix SSL cleanup in some error paths.

Certificate validation is now enabled by default. Set FTPSSLNOVERIFY=1 in environment
or configure a corresponding init macro via netrc to not validate certs (required if
you haven't installed a required CA certificate for OpenSSL).

Discussed with lukem@ on icb.

branches: 1.235.2;
PR/57003: Handle relative URLs (patch by kim@)

fix typos in word "otherwise".

Use raw write(2) instead of fwrite(3) to avoid stream corruption because
of the progress bar interrupts. From RVP.

fetch_url: improve signal handler restoration

Use SIG_ERR not NULL as the indicator that a signal handler
hasn't been changed, so that SIG_DFL (equivalent to NULL)
will be restored.

Fix restoration of SIGQUIT; use the old handler not SIGPIPE's.

branches: 1.231.2;
Make fetch_read() return size_t like fread() does. It is bogus to
have one backing implementation that returns different values and
types than the other. Handle error setting properly; i.e. bail
out if the internal read returned an error. Now we get a proper
error message when the the server resets our connection instead of
a warning that the right failed with an invalid argument.

The server used for testing was:
http://capeweather.dyndns.org:8080/graphs/3474.png
Which seems to be unreliable :-)

branches: 1.230.4;
more volatile to appease gcc.

Make outfile always allocated, free it to set it to NULL, and don't move it
around.

branches: 1.228.4;
ftp(1): split the auth processing function.

Use the first name we requested the http/https URL for, not any name we ended
up with after random redirects.

branches: 1.226.2;
handle proxy authentication correctly.

PR/51558: ast@: ftp dumps core after usage message when IPv6 URL lacks a slash.
Initialize variable so that we don't get random behavior on cleanup.

Do globbing for FTP URLs of the form ftp://... too

ok christos

branches: 1.223.2;
PR/51043: Yorick Hardy: ftp(1) should use the port number for CONNECT

sprinkle more volatile (distribution build with gcc-5.3)

Initialize the token match pointer.

Fix downloads of local files using file:// URLs

Previously it would error out in copyurlinfo() when copying a NULL port.

mark function as only needed with ssl.

Split the position/size parsing into a separate function.

Simplify and factor out connect message

- Fix to connect https via proxy.
- Fix ttyout message.

PR/50438: NONAKA Kimihiro: ftp(1): CONNECT method support
Please test!

more refactoring:
- introduce authinfo and urlinfo structures
- split negotiation code out.

Fix compile failure without WITH_SSL.

> /tmp/bracket/build/2015.12.15.21.01.27-i386/src/usr.bin/ftp/fetch.c: In function 'fetch_url':
> /tmp/bracket/build/2015.12.15.21.01.27-i386/src/usr.bin/ftp/fetch.c:823:18: error: 'HTTPS_URL_T' undeclared (first use in this function)
> urltype == HTTPS_URL_T ? &ssl : NULL);

Factor the proxy handling code out.

Separate no_proxy handling.

Try to factor out some code, this is completely out of control.

(Hopefully) fix build without IPv6 support

Use the proper format "[IPv6 address]:port" when reporting connection
attempts to IPv6 endpoints.

Add Server Name Indication (SNI) support for https.

Needed for e.g. some github URLs.

don't pay attention to special characters if they don't come from the command
line (from jmcneill)

branches: 1.205.4;
more volatile for m68k

let progressmeter deal with the timeout once we've started transferring.

PR/34796: Hauke Fath: ftp does not timeout on http fetches.

fix restart from anon ymous

compile without SSL

document https support, mentioning certificates are not validated, and obey
https_proxy.

PR/47276: Add https support

branches: 1.198.2;
As discussed on tech-net@: Don't display expected EHOSTUNREACH for all but
the last connect attempts in terse mode.

subtracting two pointers yields ptrdiff_t, so cast it to int.

When given an URL that contains "://" but is not recognised,
print an error message. Now "ftp https://foo/bar" prints

ftp: Unsupported URL scheme `https'

instead of

ftp: Can't lookup `https:ftp': No address associated with hostname
ftp: Can't connect or login to host `https:?'

branches: 1.195.2; 1.195.6; 1.195.8;
Move determination of socket buffer sizes from startup to the first
time a socket is used, as the previous logic assumed AF_INET sockets
were available (which they may not be in an IPv6-only system).
Per discussion with Maxim Konovalov and the FreeBSD problem 162661.

branches: 1.194.2;
Use __dead

Back to using 'RFC xxxx' instead of 'RFCxxxx'

Parse HTTP 'Date' entries in the `C' locale rather than the user's.
Fix from [bin/42917] (with minor changes), from KAMADA Ken'ichi.

back out previous; luke says:
'@' is a reserved URI char per RFC 3986, use %40

use strrchr to find the last @ because we might want the username to contain
user@domain.

avoid NULL dereference in log output if the command line parser
failed to extract a port number from the URL

Rename internal getline() function to get_line() so it does
conflict with the soon to be added getline(3) libc function.

Fix numerous WARNS=4 issues (-Wcast-qual -Wsign-compare).

fix -Wshadow issues

branches: 1.185.6; 1.185.8; 1.185.14;
Remove clause 3 and 4 from TNF licenses

Use the service name to getaddrinfo() (along with the host name), so that
features such as DNS Service Discovery have a better chance of working.
Suggested by David Young <dyoung>.

Display the service name in various status & error messages.

Don't getservbyname() the :port component of a URL; RFC 3986 says it's
just an unsigned number, not a service name.

branches: 1.183.6;
DPRINTF() consistency tweaks

branches: 1.182.2;
Cast the field precision calculation to int.
Should fix the amd64 build problem noticed by Paul Goyette.

Improve parsing of chunked transfer chunks per RFC2616:
* more stringent chunk-size parsing
* ignore optional trailing ';chunk-ext' stuff, instead of barfing
* detect EOF before final \r\n.

Enforce restriction that (http) proxied URL fetchs don't support
being restarted at this time.
PR #28697.

Display times in RFC2822 form rather than using ctime(3), since
the former is more explicit about the timezone offset.

* main: call tzset() to ensure TZ is setup for other <time.h> functions.
* remotemodtime(): use strptime() to parse the reply.
* fetch_url(): ensure struct tm is zeroed before calling strptime().

* Modify parse_url() to consistently strip the leading `/' off ftp URLs.
Fixes PR 17617.
* Use 'RFCnnnn' (with leading 0) instead of 'RFC nnnn', to be
consistent with the style in the RFC index.
* Refer to RFC3916 instead of 1738 or 2732.
* Expand the list of supported RFCs in ftp(1) to contain the document
name as well.

Switch from fparseln() to the internal getline() when parsing HTTP headers.
Makes ftp a bit more portable (not needing fparseln()) at the expense of not
supporting arbitrary long header lines, which I'm not concerned about
because we don't support header line continuation either...

* Implement -s srcaddr; uses srcaddr as the local IP address for all
connections.
Based on code in the version of ftp that FreeBSD had before they
replaced it with lukemftp.
* Move error message handling into ftp_connect() rather than in the
caller, so that more specific error reporting can occur.
* Improve consistency of various warning and error messages.

getpass() can return NULL upon error in some implementations
(as documented in older standards documents, before the API was obsoleted).
Problem observed in tnftp on Solaris by Emil Mikulic.

gcc4 does not care about &foo; use volatile instead. From Anon Ymous

spell precede; from Zafer

PR/34540: b1ff at fr33 dot b33r dot net: bogus free() in ftp(1)
Applied patch, thanks!

If a file upload (via -u) fails, return an non-zero exit value based on the
index of the file that caused the problem (a la auto-fetch retrieval).
Problem noted by A P Garcia in private email.

Coverity CID 2194: Don't forget to free port.

Coverity CID 2195: Free path in all code paths, not some.

rename debug to ftp_debug. grr libssh.

Rename xfoo() to ftp_foo() to avoid collisions with libssh. Don't ask.

recognize 307 redirect code.

Don't clear the trailing character on the auth_url() username;
we now use getline() and that newline strips for us.
Problem found & fixed by Mark Davies.

Add NO_USAGE and NO_DEBUG so that we can fit in the floppies again.

Implement:
int getline(FILE *stream, char *buf, size_t buflen, const char **errormsg)
Read a line from the FILE stream into buf/buflen using fgets(), so up
to buflen-1 chars will be read and the result will be NUL terminated.
If the line has a trailing newline it will be removed.
If the line is too long, excess characters will be read until
newline/EOF/error.
Various -ve return values indicate different errors, and errormsg
will be changed to an error description if it's not NULL.

Convert to use getline() instead of fgets() whenever reading user input
to ensure that an overly long input line doesn't leave excess characters
for the next input operation to accidentally use as input.

Zero out the password & account after we've finished with it.

Consistently use getpass(3) (i.e, character echo suppressed) when
reading the account data. For some reason, historically the "login"
code suppressed echo for Account: yet the "user" command did not!

Display the hostname in the "getaddrinfo failed" warning.

Appease some -Wcast-qual warnings. Fixing all of these requires
significant code refactoring. (mmm, legacy code).

* Only print the "Trying <address>..." message if verbose and
there's more than one struct addrinfo in the getaddrinfo() result.
* Don't use non-standard "u_int".

* fetch_ftp(): preserve 'anonftp' across a disconnect() so that multiple
ftp auto-fetches on the same command line login automatically.
* auto_fetch(): use an initialized volatile int to appease IRIX cc.

Use size_t instead of int where appropriate.

Fix some cast issues highlighted by Scott Reynolds using gcc 4 on OSX.4

gratuitous whitespace cleanup (before someone else jumps the gun...)

In fetch_url(), don't call freeaddrinfo(res0) too early, as we use pointers
to its contents later in the function.
Problem found by Onno van der Linden.

branches: 1.155.2;
Fix ftp url reget when globs are being used.
Provided by Mathieu Arnold <mat@FreeBSD.org>.

* Always decode %xx in a url's user & pass components.
* Only remember {WWW,Proxy}-Authenticate "Basic" challenges; no point
in tracking any others since ftp doesn't support them.
* Improve the parsing of HTTP responses.

Add (unsigned char) cast to ctype functions

Don't base64 encode the trailing NUL in the HTTP basic auth response.
Problem noted by Eric Haszlakiewicz.

Slightly rework SIGINT handling; if we're exiting the auto-fetch stuff
and sigint_raised is non-zero, reset the handler for SIGINT to SIG_DFL
and raise(SIGINT) so that the appropriate wait(3) status is setup.
Based on solution proposed by Ognyan Kulev.
This should really fix PR [pkg/26351].

Improve parsing of HTTP response headers to be more RFC2616 compliant, and
skip LWS (linear white space; CR, LF, space, tab) and the end of lines and
between the field name and the field value. This still isn't 100% compliant,
since we don't support "multi line" responses at this time.
This should fix PR [bin/22611] from TAMURA Kent (although I can't easily
find a http server to reproduce the problem against.)

Fix a minor memory leak when parsing HTTP response headers.

Don't unnecessarily display a 401/407 error when running with -V.
Fix from PR [bin/18535] by Jeremy Reed.

If an ftp auto-fetch transfer is interrupted by SIGINT (usually ^C),
exit with 130 instead of 1 (or rarely, 0).
This allows an ftp auto-fetch in a shell loop to correctly terminate the loop.
Should fix PR [pkg/26351], and possibly others.

Save approximately 8K by not including http authentication, extended status
messages and help strings when the appropriate options are set.

Don't warn about "ignored setsockopt" failures unless debugging is
enabled. Suggested by Todd Vierling.

Allow empty passwords in ftp://user:@host/file auto-fetch URLs,
per RFC 1738. Requested by Simon Poole.

Update version.

correct URL syntax in comment

* $FTPUSERAGENT overrides the HTTP User-Agent header.
Based on patch from Douwe Kiela.
* Add about:tnftp
* Fix URL in about:netbsd
* Crank version

netbsd.org->NetBSD.org

strlcpy

Uppercase URL.

fix a bogus error message when given a HTTP URL with a trailing
slash.

crank dates

add a cast to appease SUNWspro cc. noted by grant@

tweaks for fparseln(3) move from libutil to libc:
- remove #include <util.h> if nothing else needed it
- remove LDFLAGS+=-lutil if nothing else needed it

- when showing the final progress bar, replace "00:00 ETA" with the
elapsed time. (suggested by simonb)
- actually display transfer stats after a URL fetch. (bug introduced a
*long* time ago)
- update copyright & version

Handle URLs without files correctly (e.g, when using '-o -').
Fix from Anders Dinsen <anders@dinsen.net> in [bin/13768]

- Only send port number in HTTP/1.1 Host: request if port != 80.
Fixes [bin/15415] from Takahiro Kambe <taca@sky.yamashina.kyoto.jp>
- Fix bad mode passed by mls() to recvrequest().
Fixes [bin/16642] from <steve.mcclellan@radisys.com>

portnum is unsigned, use %u instead of %d

update copyrights

Add -4 to force IPv4 and -6 to force IPv6 address usage.
From Hajimu UMEMOTO, via Mike Heffner of FreeBSD.

(FreeBSD has imported NetBSD's ftp as their ftp client;
Mike is sending back some of their local changes).

use u_char instead of char in base64_encode().
problem noticed by Jorgen Lundman in private mail.

don't make broken file with -R option.

handle "*" in Content-Range properly.

If no_proxy condition is true && urltype == FTP_URL_T, use fetch_ftp to retrieve.

convert to use getprogname()

explicitly use SOCK_STREAM with socket() instead of res->ai_socktype,
because it appears that linux with glibc doesn't set the latter
correctly after one of getaddrinfo() or getnameinfo().

base64_encode should be static. picked up by hp/ux(!) compiler

It appears that whilst Apache 1.3.9 incorrectly puts a trailing space
after the chunksize (before the \r\n), Apache 1.3.11 puts *multiple*
trailing spaces after the chunksize. I 'm fairly certain that this is
contrary to RFC 2068 section 3.6, but whatever...
Found by David Brownlee <abs@mono.org>

* implement parseport(), which takes a string and attempts to convert
it to a numeric port number
* use parseport() in parse_url() and hookup()
* don't try and lookup the port number using getaddrinfo(), as it's too hard
to separate a failed host name lookup from a failed service name lookup.
this was causing lossage on systems that don't have `http' in services(5)
(such as solaris), but only crept in when we started using getaddrinfo()
unconditionally.

- rename NO_QUAD to NO_LONG_LONG, QUAD* -> LL* and add ULL* (unsigned)
equivalents. name change suggested by Klaus Klein <kjk@netbsd.org>
- change defined(BSD4_4) || HAVE_SIN_LEN tests into HAVE_SOCKADDR_SA_LEN,
and set the latter if BSD4_4 exists

* always use getaddrinfo() and getnameinfo() instead of maintaining two code
paths. (lukemftp will provide replacements for these on older systems)
* rename __USE_SELECT to USE_SELECT
* rename BSD4_4 to HAVE_SIN_LEN
* replace union sockunion {} with struct sockinet {}, and modify the code
accordingly. this is possibly more portable, as it doesn't rely upon the
structure alignment within the union for our own stuff.
(XXX: haven't tested the ipv6 stuff)

clean up NO_QUAD support: create helper #defines and use as appropriate:
#define NOQUAD ! NOQUAD
------- ------ - ------
QUADF "%ld" "%lld"
QUADFP(x) "%" x "ld" "%" x "lld"
QUADT long long long
STRTOL(x,y,z) strtol(x,y,z) strtoll(x,y,z)

minor knf. call setbinary()/setascii() with non-NULL 2nd arg

errx?/warnx? audit. do not pass variable alone, use %s. idea from openbsd

branches: 1.116.2;
* migrate the SYST parsing from setpeer() into a separate remotesyst().
call remotesyst() only when login has been successful
some servers don't let you run SYST until you've successfully logged in.
* in fetch_ftp(), always call setpeer() with autologin disabled, and use
the following ftp_login() to DTRT.
this prevents ftp from trying to login a second time if the first autologin
fails when connecting to a remote site anonymously using autofetch.
* reset unix_proxy and unix_server in cleanuppeer()
* missed a function conversion in the KNF sweep...

- fix ai_unmapped() to be a no-op in the !def INET6 case
- display `(-INET6)' at the end of the version string if !def INET6
- clarify in the man page that IPv6 support may not be present (for lukemftp :)

Add support for 'ftp -u url file ...', to upload a list of files to given url.
Mostly based on [bin/10019] by Scott Aaron Bamford <sab@ansic.net>

convert IPv4 mapped address (::ffff:10.1.1.1) into real IPv4 address
before touching it. IPv4 mapped address complicates too many things
in FTP protocol handling.

branches: 1.112.2;
do not pass scoped IPv6 address notation on Host: directive, since
scope identifier is local to the originating node.
do not allow scoped IPv6 address notation in URL, if it is via proxy.

convert to ANSI KNF

* Add support for `fget localfile', which reads a list of filenames to
retrieve from localfile. Based on work by Darren Reed.
* Crank version.
* Update copyright dates.

fixes from cgd:
* sanity check a length (otherwise certain bogus responses can crash ftp)
* allow a transfer encoding type of `binary'; certain firewall vendors
return this bogus type...

s/strtoq/strtoll/ (the latter is standardised)

make debugging output unambiguous on IPv6 numeric addrs (don't use host:port)

http://[::1]:8080/ is legal.

send Host: directive with RFC2732 bracket notation for IPv6 numeric,
otherwise "host:port" is ambiguous to servers (clarification will be submitted
as update to RFC2732).

only use getaddrinfo() et al if both NI_NUMERICHOST *and* INET6 are defined...
(allows --disable-ipv6 in lukemftp's configure script to disable this as
well, which is good for testing when it appears getaddrinfo() is borken)

define private type `sigfunc' as
typedef void (*sigfunc) __P((int));
and replace use of sig_t and void (*)(int).

certain other OSes define sig_t differently to that (they add extra arguments),
and it causes problems due to function mismatches, etc...

work around bug in apache 1.3.9 which incorrectly puts a trailing
space after the chunksize.
noted by Jun-ichiro itojun Hagino <itojun@itojun.org> in [bin/9096]

change references from draft-ietf-ipngwg-url-literal-01.txt to RFC 2732

Fix chunked support; probably broke after rate limiting was added.
Problem noticed/debugging assisted by giles lean <giles@nemeton.com.au>.

XXX: rate limiting with chunked xfers might not limit correctly (i.e,
the limit may be too high or too low); fixing this is non trivial,
and will probably occur if i ever rototill fetch_url()

move version into separate file to reduce recompilation after version crank.

* remove unnecessary freeaddrinfo(res), since res0 was changed to be
freed earlier in itojun's last commit. fixes [bin/8948].
* remove `const char *reason'; it was being assigned but not used.

fix memory leak in fetch_url (no freeaddrinfo was there).
sync with recent KAME.

* complete_remote(): use remglob("", ...) instead of remglob(".", ...),
for listings of the current working directory; some ftp servers don't
like `NLST .'.
[noted by Giles Lean <giles@nemeton.com.au>]
* recvrequest(): treat remote=="" as remote==NULL when calling command().
(to support the above change)
* support `[user@]' in `[user@]host' and `[user@]host[:][path]'.
[based on idea (and initial code) from David Maxwell <david@fundy.ca>]
* `idle' may be invoked without any args
* reformat some comments
* reformat usage string in program and man page
* call updateremotepwd() after successful login, not after successful connect
* always call setsockopt(, IPPROTO_IP, IP_TOS, ) (et al); using #if
defined(IPPROTO_IP) doesn't work on certain foreign systems where
enums instead of #defines are used...
[noted by Matthias Pfaller <leo@dachau.marco.de>]

whitespace nits

- implement restarting file:/// non-proxied http:// URLs (with -R).
- fix a semicolono which stopped file:/// from working

- split the version string into product and version
- be consistent about reporting the version between:
+ status command
+ about:version URL fetch
+ User-agent sent in http requests

when using http/1.1 for unproxied http requests, send the port as a number
(e.g, `:80') instead of a name (e.g, `:http'), because the former is the
supported method that actually works... noted by hubertf@netbsd.org

support `about:version'. also display the version in the output of `status'.

new features:
- add `usage'; displays the usage of a command.
implemented by calling the c_handler() with argc = 0, argv = "funcname".
- add `passive auto'; does the same as $FTPMODE=auto.
- add `set [option value]'; display all options, or set an option to a value.
- add `unset option'; unset an option.
- add getoptionvalue() to retrieve an option's value, and replace a few
global variables with calls to this.
- implement cleanuppeer(), which resets various bits of state back to
`disconnected'. call in disconnect() and lostpeer().
- support completing on `options'.
- improve recovery after a SIGINT may have closed the connection.
XXX: there's still a couple to fix

other stuff:
- various consistency fixes in the man page.
- ensure that the command usage strings in the code and man page match reality.
- mput/mget: check that the connection still exists before each xfer.
- minor cosmetic changes in confirm().
- set code correctly in sizecmd() and modtime()
- don't need \n in err() strings.
- change lostpeer to take an argument (rather than casting (sig_t)lostpeer
in signal handlers)
- knf and whitespace police.

a few user interface and cosmetic tweaks:
* confirm(): move from util.c to cmds.c. display mnemonic string in its prompt.
add support for `q' (terminate current xfer), `?' (show help list)
* in various signal handlers, output a linefeed only if fromatty.
* if fgets(stdin) returned NULL (i.e, EOF), clearerr(stdin) because you don't
want future fgets to fail. this is not done for the fgets() in the main
command loop, since ftp will quit at that point.
* unless ftp is invoked with -a, don't retain the anonftp setting between
hosts (`ftp somehost:' sets anonftp, but you don't want that to `stick'
if you close that connection and open a new one).

use sigjmp_buf for sigsetjmp(), instead of jmp_buf.
noted by Havard.Eidnes@runit.sintef.no.

* use sigsetjmp()/siglongjump() instead of setjmp()/longjmp(); the latter
don't save the signal mask on some foreign systems.
* ensure signal handlers don't use stdio and do reset errno if they
don't exit with siglongjmp()
* use a common SIGINT handler for {send,recv}request()

more propaganda :)

* fetch_url(): specifically set SIGQUIT to psummary before each xfer.
(work around editline's override)
* minor cleanup of signal handler (along the lines of similar work in
recvrequest()). the handlers should now be reset everytime the cleanup
handler was callled.

* factor out SIGINFO setting into a handler that is always active (but only
prints out info if bytes > 0). only set the handler if SIGINFO is defined
* hijack SIGQUIT to be the same as SIGINFO (foreign ports have this, and it's
annoying to have SIGQUIT dump core on netbsd when it prints info on other
systems)
* in {recv,send}request(), factor a lot of duplicated code out into a
`cleanup' section at the end
* rework shell() a bit

add TNFi copyright to all files i've done more than a minor amount of work to...

enhancments from Marc Horowitz <marc@mit.edu> to improve connection timeouts:
* implement xsignal_restart(), which only sets the SA_RESTART flag if
specifically requested
* xsignal() is now a wrapper to xsignal_restart(). INFO, USR1, USR2 and WINCH
are restartable, ALRM, INT, PIPE and QUIT are not.
* improve getreply()'s timeout code to take advantage of the above.

other changes:
* improve wording of how globbing works for `classic' URLs (host:path).
suggested by John Refling <johnr@imageworks.com> in relation to PRs
[bin/8519] and [bin/8520]
* always compile in the `edit' command even if NO_EDITCOMPLETE defined.
it's just a no-op in the latter case, which is more consistent to
the users.
* always compile in about: support (i.e, remove NO_ABOUT).
i'm entitled to some vanity in this program...
* clean up some whitespace

* In the !NI_NUMERICHOST case (i.e, getaddrinfo() challenged systems), portnum
should be in host order. found/fixed by Matthias Pfaller <leo@dachau.marco.de>
* parse_url(): improve checking of portnum, and add an extra argument to pass
back the parsed portnum to the caller (reduces a bit of code duplication)
* Move the KAME/WIDE copyrights after the BSD/TNFi ones. Since there was
significantly less code added under the former, it's only fair on the latter.

* fix initialisation of home[]
* fetch_url(): if path would be NULL, return strdup("")

* consistentnly use memset(a,0,c); there were some cases of memset(a,'\0',c)
* remove explicit extern int h_errno; it's in <netdb.h>
* add <termios.h> back to util.c; it contains struct winsize on some systems

remove debugging cruft

* add new commands:
lpage page local files
pdir as dir, but through your $PAGER
pls as ls, but through your $PAGER
* implement docase() (a la dotrans() et al) and use appropriately, rototilling
some duplicated code
* globulize(): modify to return a pointer to the strdup()ed result in all cases,
and hack the code that calls it to take this into account
* replace strcpy() and strncpy() with strlcpy()
* put(), getit(): use some aptly named local vars instead of argv[...]
* delint

* idle(): rename to idlecmd(). certain linux distributions have an incompatible
prototype for idle() in <unistd.h> (which i thought was against namespace
and sensibility guidelines, but...)
* consistently use xsignal() instead of signal(). we get known behaviour
in all cases (SA_RESTART), which is good for some borken foreign systems.
* remove signal.h from most files; it's unnecessary now
* fetch_url(): use `long chunksize' instead of ssize_t; it's more portable, and
we're setting chunksize with strtol() anyway
* xsignal(): only use SA_RESTART if it exists. SunOS 4.x doesn't have it
but has the inverse (SA_INTERRUPT). the original function i was inspired
from had this support (lib/signal.c, W. Richard Stevens' `UNP 2nd ed Vol 1').
* remove <termios.h> from util.c; it should be unnecessary now

* replace ifdefs against __SVR4 and __linux__ with DIRENT_MISSING_D_NAMLEN;
it's more portable and more obvious
* remove the mkgmtime() && HAVE_TIMEGM stuff:
a) why should netbsd have to define HAVE_TIMEGM to compile cleanly?
b) foreign compiles of ftp should just be linked with working
timegm function

a more portable version of this ftp client will be released as a 3rdparty
product; no use polluting our code with half-baked attempts...

* use %lld instead of %qd to print out (long long) vars.
(slightly more portable; e.g, solaris supports this)
* remove some fluff (lint)

fix a couple of thinkos in my recent work:
* abort_remote(): replace borken MIN(4,BUFSIZ) with just BUFSIZ; it
should have been MAX(4,BUFSIZ), but it's probably safe to assume that
BUFSIZ is at least 3... (fix from simonb)
* auth_url(): use the correct variable when calculating a buffer size.

branches: 1.73.2;
* add support for `xferbuf', which sets both `sndbuf' and `rcvbuf'
* document the above three commands
* rototill the way the sndbuf and rcvbuf work. remove resetsockbufsize()
* use the appropriate socket buffer size as the size of the buffer that
the read()/write() loops use. speeds up things in some cases.

replace snprintf() with strlcpy(), strlcat(), or direct assignment
where appropriate. (strlc*() are easier to port to foriegn systems).

XXX; there's still a few snprintf's in the progress meter stuff to convert

* protect more of the AF_INET6 stuff with #ifdef INET6 (for portability)
* in the main data moving loops only call the initial gettimeofday() if
rate throttling is enabled (saves a system call per loop when not
throttling).

add trailing . to message

warn()->warnx() in a couple of places.

The port number might have changed in the proxy case. Re-evaluate it.

fix proxy code. strtol() does not work very well for parsing port names.

make ftp work again with the traditional gethostbyname/getservbyname
interfaces.

fetch_url() fixes:
- just display the hostname:port of the proxy url, rather than the full url.
this prevents someone `shoulder surfing' a proxy username/password
in $http_proxy. [suggested by perry]
- compact verbose notes for http fetchs; now displays
(via host:port, with authorization, with proxy authorization)
with each component being optional.
(and a couple introduced with the ipv6 mods...)
- don't override host with the canonical name; this prevented fetches from
http/1.1 virtual hosts from working if the virtual host was a CNAME.
[noted by bernd]
- call freeaddrinfo() if res was built with getaddrinfo()

fix problem with www authentication: len passed to snprintf() for
auth_url() strings was 1 too small. noted by veego@

* change fetch_ftp() to be fully rfc 1738 compliant; if the URL contains
the an empty directory (e.g, between `some' and `path' in
`ftp://host/some//path'), then execute `CWD ' (without a path).
This command will probably fail on rfc 959 compliant servers, so
issue a warning in this case and bail. [noted by cgd].
(i wonder if the people who wrote rfc 1738 actually realised that this
requirement appears to contravene the spec for `cwd' in rfc 959 ?)
* replace isurl() with isipv6addr(), and use appropriately. fixes
auto-login with `classic ftp URLs' (e.g, `ftp somehost:')
* cleanup and rework some of the ipv6 stuff in parse_url()
* prevent potential coredump in fetch_ftp() when parsing `;type=X'
* KNF a few lines
* fix a couple of comments
* cleanup the man page a bit

Make FTP downloads via HTTP proxy servers work again.

fix 'junk pointer free' issue.

add dual-stack (IPv4/v6) support. hope I broke no other part...

[fear this; more ftp hacking from lukem :-]

features:

don't search for trailing `;type=' in a NULL path...
fixes [bin/7800] by Mason Loring Bliss <mason@acheron.middleboro.ma.us>

include time.h explicitly, don't depend on other include side-effects

split the "SMALL" #ifdefs into ones for NO_ABOUT, NO_EDITCOMPLETE, and
NO_PROGRESS. -DSMALL still implies all of those. progress meter support
isn't necessary for the smallest possible ftp client, but it adds very
little space and makes users' lives much better. Therefore, it should
be enabled for installation media if at all possible.

* fix gate mode to login as `user@realhost' rather than using PASSERVE;
the latter only seemed to work for TIS Gauntlet and not TIS fwtk.
thanks to simonb@netbsd.org for testing this. fixes [bin/5556].
* if EOF (e.g, ^D) is entered at a username/password/account prompt which
happens to use fgets(), exit the login rather than treating EOF as CR.
* don't use the comma operator where separate statements are valid
* always use snprintf to copy stuff into malloced buffers, just in case
typos creep in and mean that the buffer ends up being overflowed

parse http:// urls (and the $http_proxy variable) for [user:[pass]@]
elements, which are used for the initial authentication attempt (if
requested by the server). in the case of $http_proxy, use the values
for proxy authentication.

* make parsing of ftp:// urls more RFC 1738 compliant;
- the path is split on `/', and each directory is CWD-ed into separately.
(from [standards/7484] by Alan Barrett <apb@iafrica.com>)
- support a trailing `;type=X' suffix, where X is a,i, or d. (d isn't
implemented, but it is recognised)
- the only non-compliant behaviour is that empty directories sections
(e.g `//') aren't run as `CWD ' - as a lot of ftpds don't like that.
Instead, treat this as a no-op.
* don't support globbing for ftp urls, since that's technically not
RFC compliant.
* fix a couple of man-page nits

branches: 1.52.2;
* implement -R; restart non-proxied command-line FTP xfers
* fix fetch_ftp() so that hcode parsing is not done for file:// urls
(a } in the wrong place, and code at the wrong indent level...)
* change outfile to being a global (so it gets correctly reset)
* change parse_url to not remove leading '/' for non ftp urls.
whilst this is not totally rfc1738 compliant, other code kinda
assumes this is the case, and it doesn't hurt

Add a few more variables that end up in registers in gcc-2.8.1

* add url_decode() - `in-place' decode %xx escapes in a given url component
* parse_url()
- only look for user[:pass] for an ftp url (per rfc1738)
- strip leading /'s in an ftp url. (almost per rfc 1738)
* fetch_url()
- decode a copy of the path and use that to build local filenames
- send port in http Host: header (suggested by cgd@netbsd.org)
* fetch_ftp()
- url_decode() the user, pass and path
- fix splitting of path into dir & file (partially from [bin/7073])
- don't bother caching the last host; it can cause problems when
using ftp:// transfers, or when the user changes between xfers
* improve documentation of auto-fetched url arguments (especially regarding
escape sequences in ftp:// urls)
* some whitespace & copyright updates

this should fix [bin/7073] William O Ferry <woferry@warp.wofme.com>,
as well as the metaissues raised in that PR.

only print "Requesting..." messages if (verbose)

* -v enables verbose & progress, -V disables both
* set setvbuf(ttyout, NULL, _IOLBF, 0) and remove a bunch of fflush(ttyout).
* use fwrite() instead of write() for progressmeter (don't intermix stdio
with non stdio ops)

support -f, which forces a cache flush for http xfers using either
`Pragma: no-cache' (http/1.0) or `Cache-Control: no-cache' (http/1.1)

add missing braces (this is C not python...). fixes coredump on solaris

support http/1.1's `transfer-encoding: chunked' mode. [noted by hubertf & bad]

* implement auth_url(); given a challenge, decode it, ask the user for
a response, and return the base64 encoded response. only the `Basic'
scheme is supported.
* implement base64_enc(), which base64 encodes the given buffer
* add support for http response codes 401 "authorization required" and 407
"proxy authorization required", using auth_url(). [requested by veego@]

use uname(3) to determine info to send in User-agent: header. (suggested by cgd)

* major code reorg; gut auto_fetch() and url_get() into a smaller auto_fetch(),
do_fetch() - which retrieves one file calling fetch_url() or fetch_ftp()
as necessary.
* don't http redirect more than 5 times for a given url
* send `User-Agent: NetBSD-ftp/1.4' header in http requests. (suggested
by Christoph Badura <bad@ora.de>)
* cleanup http return code parser, and add support for:
- 300 `Multiple Choices' - but only if the server returns a
preferred url in a Location: header because i'm *not* adding
a html parser to provide the user with options.
- 305 `Use Proxy [given in Location: header]'. (XXX: not tested)
* support http redirects to non-proxied ftp://urls. (bug discovered by
Chris Demetriou <cgd@netbsd.org>)
* auto-login to an ftp site (using the FTP protocol) if an ftp://host/dir/
style url is given and ftp_proxy is set. whilst this is less orthoganol
with other ftp://host/file urls it's *much* more convenient.

be saner about the method to determine which $proxy to use

Send an Accept: */* header, since some sites require content negotiation.
Grr.

Temporary fix of UTC -> time_t conversion:
1. changed to use timegm(3) on NetBSD and
2. supply alternative code for other Unix-like OSs
(NetBSD ftp shall be portable for some reasons :-).

This fix closes PR #6448.

NOTE: This should be fixed again if a portable UTC to time_t
conversion method is specified in some standards.

* fix previous in a more portable manner (timegm() is not portable/standard).
use mktime(), and adjust to GMT as per code in util.c::remotemodtime()

HTTP uses UTC time, and changed to use timegm(3) instead of mktime(3)
for conversion of HTTP time.

XXX Unfortunately, timegm(3) is not portable (ex. SunOS 4 have, but 5 not).

* hash & progress are mutually exclusive. [noted by mrg@netbsd.org]
* in autofetch mode, don't attempt to 'cd /' on first xfer (only on
subsequent xfers), as some ftp daemons don't permit that.
[noted by dbj@netbsd.org].
XXX: code could be smarter (but a lot more complicated to boot); this
works for now.

fix handling of port on non-proxied auto-fetch ftp:// urls

some compilers bitch about int used when enum expected...

* fix skipping of leading / in dir, which may have resulted in
parsing junk memory (picked up whilst porting to solaris)
* remove superfluous var assignments

pull in <sys/time.h> for utimes(), reset port to 0 in auto_fetch loop

use utimes() instead of futimes() - the former is more portable

technically, you need <sys/stat.h> for struct stat ....

cleaner version of fvdl's previous fix, using xstrdup as well

Avoid using freed memory when using URLs and a proxy.

features:
* support $no_proxy, which is a comma or space separated list of
host[:port] elements for which proxying is to be disabled.
(asked for by cgd in [bin/5027])
* if $FTPANONPASS is defined, use that as the anon ftp password
(instead of "`whoami`@")
* allow http URL's without a filename as long as an output file
is specified.

other stuff:
* implement parse_url(), which breaks up a URL into its bits, and use.
* simplify url_get() and auto_fetch() to use parse_url() and to not
modify the supplied URL or a copy of it.
* implement xmalloc() and xstrdup(); error-checked malloc()/strdup()
* add more consistency to messages, quoting strings in output as `%s'

minor grammar fix

* spell `retrieved' correctly!
* parse Last-Modified: HTTP headers for date, and change time of
retrieved file if parsing succeeded
* support file:/// and file://localhost/ URLs
* in url_get(), re-write byte moving code to consistently use fread()
and fwrite(), and check errors at end with ferror()
* add about:* easter-egg (#ifndef SMALL :-)

* send 'Connection: close' in HTTP/1.1 headers, preventing the
remove server from using a persistent connection, which speeds
up such requests.
* support http 301 and 302 redirects
* rewrite guts of url_get() to use fparseln() et al instead of
read(s,&p,1)... enables each in the header to be parsed
as necessary
* rename login to ftp_login, to remove conflict with util.h::login
* cleanup verbose messages during http proxy requests

Add "sndbuf" and "rcvbuf" commands for setting the socket buffer sizes,
which in turn can allow the use of larger TCP windows. This is a work in
progress; there is not yet support for specifying global defaults or
user prefrences on a host/network basis.

some fixes & enhancements from openbsd's ftp, with extra fixes by me:
* default to passive with active fallback. $FTPMODE modifies this behaviour.
-A forces active connection.
* support '-o outfile' for auto-fetched files. outfile can be a file,
`-' (for stdout), or '|command' (to output each file through command).
* support '-r waittime', which retries the connection after waittime seconds
if it fails.
* fix 'page file' when restart is non-zero.
* try all ip-addresses of a host in a http fetch (as the normal ftp fetch
does).

XXX: a ``broken pipe'' error sometimes occurs with -o '|command';
i haven't tracked this down yet.

Add the ability to work with HTTP-1.1-style virtual hosts, using the
HTTP/1.0 backwards compatible "Host:" field method.

- cast arg to is*() to unsigned char.

define MAX_IN_PORT_T and use instead of USHRT_MAX. suggested by cgd@netbsd.org

* ensure buffer for username is initialised, so ^D on username prompt
doesn't use garbage for the username. from "Soren S. Jorvang" <soren@t.dk>
in [bin/4559]
* use in_port_t for ports, and USHRT_MAX instead of 0xffff
(from millert@openbsd.org)
* use `NULL' instead of `(.... *)0' where appropriate.

* in recvrequest(), ignore restart_point unless "RETR"ieving. fixes problems
where a remote completion or `mget' would confuse the client a `restart'
had been issued beforehand. now, `restart' is remembered until an operation
that can actually use it is invoked.
* in sendrequest(), don't reset restart_point upon entry. fixes `restart'
for `put' operations.
* if `restart' is invoked with no arguments, print current setting instead
of displaying a usage
* consistently use printf("%qd", (long long)restart_point) when displaying
restart_point
* use strto[lq]() instead of atol() when parsing `mark' and `restart' values
* remove unnecessary strlen()s when result of previous snprintf() will do
* replace a few malloc()/strcpy()s with strdup()s
* use SECSPERHOUR instead of '3600'

branches: 1.16.2;
More cleanups from Todd Miller <Todd.Miller@courtesan.com>:
* Use an int, not "union wait".
* Move the "parsed_url" label so that the next statement is not an "else" --
some ansi C compilers don't like it the old way (SGI's for example).
* Deal with the possibility of getlogin(2) failing.
* Don't error out if the remote server doesn't support the "MDTM" command.

Fixes from Todd Miller <Todd.Miller@courtesan.com>:
* use size_t instead of int in places
* use symbolic constants when using access()

bugs fixed:
* don't interpret '-' or '|' when a local filename is determined from
the remote name (i.e, in mget, and in get with only one argument).
This is implemented using an extra argument to recvrequest().
Fixes a major security hole.
* clean up memory leak when using globulize()
* clean up a couple of comments
* fix wording in TNF copyright

features added:
* support for TIS fwtk gate-ftp servers:
* read defaults from $FTPSERVER && $FTPSERVERPORT
* start in gate-ftp mode if invoked as 'gate-ftp'
* toggle or set with 'gate [host [port]]'

Send \r with \n in http requests. Allow "content-length" to be missing.
From [bin/3891] by Krister Walfridsson <cato@ulysses.df.lth.se>

* use RCSID() && COPYRIGHT() macros
* cleanup for WARNS=1 (including some ugly '(void)&var' bits wrapped in
#ifdef __GNUC__ to shut up gcc warnings WRT setjmp/longjmp)
* use strtol() instead of atoi(), and more extensively check result of
conversion
* use u_int16_t instead of short or int for TCP port addresses

Provide a different error message than `invalid url' when an
auto-login ftp URL is used when $ftp_proxy is defined. It now prints:
Auto-login using ftp URLs isn't supported when using $ftp_proxy
Should solve rest of [bin/3643].

Whilst this is inconsistant with the behaviour when $ftp_proxy isn't
defined, the following constraints apply:
* it's not possible to support ftp URL auto-login when $ftp_proxy is
defined, since it uses http not ftp, and you can't `login' to http
servers; fudging this would require a major rewrite of ftp anyway)
* silently ignoring $ftp_proxy and not using it if an ftp auto-login
URL is given is bad user interface design)
* mrg & others will harrass me if I remove support for autologin ftp URLs
when $ftp_proxy isn't defined, even though it made the behaviour
consistant whether $ftp_proxy was set or not.

url_get(): use origline in some messages, simplifying them, and free up copy
of origline before returning. also, previous commit (fix for [bin/3643]) was
inspired by suggested fix in said PR by Alan Barrett <apb@iafrica.com>.

Actually report why certain URLs are invalid, rather than just exit
with a non-zero exit code. fixes most of [bin/3643].

* cleanup parser of ftp://[user:pass@]host[:port]/[dir/][file]. should
be more robust now. fixes [bin/3520]. (missed this in previous commit msg)
* rename bad_url: to bad_ftp_url:

i

More enhancements/bugfixes (when will it end?)
* differentiate between being connected, and being logged in
* cleanup some text messages
* support username & password ftp URLs (ftp://user:pass@host/) in non-proxy
situations; assume proxy supports it for proxy situations.
* cd to / before performing any autofetch transfers
* use strncasecmp in URL parsing. fix from <Todd.Miller@courtesan.com>

* support $ftp_proxy for ftp:// transfers [bin/3245]
* add "more" & "less" as synonyms for "page"
* move editline setup code into controlediting(), and call appropriately.
only setup setup terminal if going into interactive mode. inspired
by Todd Miller <todd.miller@courtesan.com>

Fixes from <Todd.Miller@cs.colorado.edu>, with some cleanup and reworks by me:
* only echo "Passive mode" in verbose mode; scripts that use ftp
may get unwanted output otherwise
* disable progress bar and modification time preservation when
retreiving to a non-regular files. fixes progress bar getting in
way of "get file /dev/tty"
* setup el_init() et al if editing is set, not if fromatty.
TODO: migrate this to a function, and call if editing is turned on later
in the session. also implement edit_cleanup if editing is turned off
* call el_set() after setting SIGWINCH handler. This fixes the problem
when suspending in a non-cbreak shell (e.g, csh) would trash your tty mode.
* reset interactive mode correctly in auto_fetch() mget mode

Features:
* support remglobbing of auto_fetch arguments
* new flag - '-e'; disable editing
* "page file" == "get file |${PAGER-less}"

Bugfixes/cleanup:
* consistently use a trailing '.' on messages
* code cleanup, including buffer overrun fixes, use puts
and putchar in places, etc (inspired by OpenBSD mods)
* disable progress bar when local-file is a pipe or '-'
* skip \r in http headers
* fix remote ftpd slash bug more elegantly (so it works with ////)
* abort_remote(): check if cout==NULL before using it. should fix [bin/3273]
* fixed up cosmetic problems when complete_remote() generated errors from the
remote server (such as "no files found", "login with user and pass", ...)
done by adding extra argument to remglob(), which is a pointer to an error
buffer to put messages in rather than printing to stdout.

[Yet Another Huge Ftp Commit - hopefully the last for a while,
barring any more little things people want added ...]

New features:
* progressmeter is now asynchronous, so "stalled" transfers can be
detected. "- stalled -" is displayed instead of the ETA in this case.
When the xfer resumes, the time that the xfer was stalled for is
factored out of the ETA. It is debatable whether this is better than
not factoring it out, but I like it this way (I.e, if it stalls for 8
seconds and the ETA was 30 seconds, when it resumes the ETA will still
be 30 seconds).
* verbosity can be disabled on the command line (-V), so that in auto-fetch
mode the only lines displayed will be a description of the file, and
the progress bar (if possible)
* if the screen is resized (and detected via the SIGWINCH signal), the
progress bar will rescale automatically.

Bugs fixed:
* progress bar will not use the last character on the line, as this can
cause problems on some terminals
* screen dimensions (via ioctl(TIOCWINSZ)) should use stdout not stdin
* progressmeter() used some vars before initialising them
* ^D will quit now. [fixes bin/3162]
* use hstrerror() to generate error message for host name lookup failure.
* use getcwd instead of getwd (it should have been OK, but why tempt fate?)
* auto-fetch transfers will always return a positive exit value upon failure
or interruption, relative to the file's position in argv[].
* remote completion of / will work, without putting a leading "///".
This is actually a bug in ftpd(1), where "NLST /" prefixes all names
with "//", but fixing every ftpd(1) is not an option...

New features:
* Command line editing via editline(3) library.
* Context sensitive command and file completion, including remote files.

Enhancements to auto-fetch feature:
* Support for http:// URLs using the http protocol, including proxy HTTP
support via $htty_proxy if it's defined.
* The connection is kept open between successive files on the same host.
(obviously, this does not count for http requests.)
* Return value of ftp is 0 on no error, or the offset in argv[] of the file
which failed (i.e., argv[x] failed, ftp returns x).
* If the path in an ftp URL or classic format line has a trailing '/',
cd to the path and enter interactive mode. Fixes [bin/3011], albiet
requiring the user to help ftp in determining the operation.

Other changes:
* '-P port' works for normal ftp, and is the default for all classic style
auto-fetch transfers and for ftp URLs that don't specify the port.
(previously it would just work for the first xfer.)
* Some code moved into separated files along logical divisions.
* Editing and completion can be compiled out with -DSMALLFTP.

Don't finish downloading an empty file with 'already done' before it is
created locally.

Fix HTTPS through Proxy.

While a regular HTTP Proxy, requires the absolute URL with protocol
and host part, yyou must only send the relative URL through a
CONNECT tunnel (you are talking to the target server).

Add option sslnoverify to control validation of SSL certificates.
Add netrc processing to fetch-mode (URL on command line) to enable options and autologin
via netrc.
Fix SSL cleanup in some error paths.

Certificate validation is now enabled by default. Set FTPSSLNOVERIFY=1 in environment
or configure a corresponding init macro via netrc to not validate certs (required if
you haven't installed a required CA certificate for OpenSSL).

Discussed with lukem@ on icb.

branches: 1.235.2;
PR/57003: Handle relative URLs (patch by kim@)

fix typos in word "otherwise".

Use raw write(2) instead of fwrite(3) to avoid stream corruption because
of the progress bar interrupts. From RVP.

fetch_url: improve signal handler restoration

Use SIG_ERR not NULL as the indicator that a signal handler
hasn't been changed, so that SIG_DFL (equivalent to NULL)
will be restored.

Fix restoration of SIGQUIT; use the old handler not SIGPIPE's.

branches: 1.231.2;
Make fetch_read() return size_t like fread() does. It is bogus to
have one backing implementation that returns different values and
types than the other. Handle error setting properly; i.e. bail
out if the internal read returned an error. Now we get a proper
error message when the the server resets our connection instead of
a warning that the right failed with an invalid argument.

The server used for testing was:
http://capeweather.dyndns.org:8080/graphs/3474.png
Which seems to be unreliable :-)

branches: 1.230.4;
more volatile to appease gcc.

Make outfile always allocated, free it to set it to NULL, and don't move it
around.

branches: 1.228.4;
ftp(1): split the auth processing function.

Use the first name we requested the http/https URL for, not any name we ended
up with after random redirects.

branches: 1.226.2;
handle proxy authentication correctly.

PR/51558: ast@: ftp dumps core after usage message when IPv6 URL lacks a slash.
Initialize variable so that we don't get random behavior on cleanup.

Do globbing for FTP URLs of the form ftp://... too

ok christos

branches: 1.223.2;
PR/51043: Yorick Hardy: ftp(1) should use the port number for CONNECT

sprinkle more volatile (distribution build with gcc-5.3)

Initialize the token match pointer.

Fix downloads of local files using file:// URLs

Previously it would error out in copyurlinfo() when copying a NULL port.

mark function as only needed with ssl.

Split the position/size parsing into a separate function.

Simplify and factor out connect message

- Fix to connect https via proxy.
- Fix ttyout message.

PR/50438: NONAKA Kimihiro: ftp(1): CONNECT method support
Please test!

more refactoring:
- introduce authinfo and urlinfo structures
- split negotiation code out.

Fix compile failure without WITH_SSL.

> /tmp/bracket/build/2015.12.15.21.01.27-i386/src/usr.bin/ftp/fetch.c: In function 'fetch_url':
> /tmp/bracket/build/2015.12.15.21.01.27-i386/src/usr.bin/ftp/fetch.c:823:18: error: 'HTTPS_URL_T' undeclared (first use in this function)
> urltype == HTTPS_URL_T ? &ssl : NULL);

Factor the proxy handling code out.

Separate no_proxy handling.

Try to factor out some code, this is completely out of control.

(Hopefully) fix build without IPv6 support

Use the proper format "[IPv6 address]:port" when reporting connection
attempts to IPv6 endpoints.

Add Server Name Indication (SNI) support for https.

Needed for e.g. some github URLs.

don't pay attention to special characters if they don't come from the command
line (from jmcneill)

branches: 1.205.4;
more volatile for m68k

let progressmeter deal with the timeout once we've started transferring.

PR/34796: Hauke Fath: ftp does not timeout on http fetches.

fix restart from anon ymous

compile without SSL

document https support, mentioning certificates are not validated, and obey
https_proxy.

PR/47276: Add https support

branches: 1.198.2;
As discussed on tech-net@: Don't display expected EHOSTUNREACH for all but
the last connect attempts in terse mode.

subtracting two pointers yields ptrdiff_t, so cast it to int.

When given an URL that contains "://" but is not recognised,
print an error message. Now "ftp https://foo/bar" prints

ftp: Unsupported URL scheme `https'

instead of

ftp: Can't lookup `https:ftp': No address associated with hostname
ftp: Can't connect or login to host `https:?'

branches: 1.195.2; 1.195.6; 1.195.8;
Move determination of socket buffer sizes from startup to the first
time a socket is used, as the previous logic assumed AF_INET sockets
were available (which they may not be in an IPv6-only system).
Per discussion with Maxim Konovalov and the FreeBSD problem 162661.

branches: 1.194.2;
Use __dead

Back to using 'RFC xxxx' instead of 'RFCxxxx'

Parse HTTP 'Date' entries in the `C' locale rather than the user's.
Fix from [bin/42917] (with minor changes), from KAMADA Ken'ichi.

back out previous; luke says:
'@' is a reserved URI char per RFC 3986, use %40

use strrchr to find the last @ because we might want the username to contain
user@domain.

avoid NULL dereference in log output if the command line parser
failed to extract a port number from the URL

Rename internal getline() function to get_line() so it does
conflict with the soon to be added getline(3) libc function.

Fix numerous WARNS=4 issues (-Wcast-qual -Wsign-compare).

fix -Wshadow issues

branches: 1.185.6; 1.185.8; 1.185.14;
Remove clause 3 and 4 from TNF licenses

Use the service name to getaddrinfo() (along with the host name), so that
features such as DNS Service Discovery have a better chance of working.
Suggested by David Young <dyoung>.

Display the service name in various status & error messages.

Don't getservbyname() the :port component of a URL; RFC 3986 says it's
just an unsigned number, not a service name.

branches: 1.183.6;
DPRINTF() consistency tweaks

branches: 1.182.2;
Cast the field precision calculation to int.
Should fix the amd64 build problem noticed by Paul Goyette.

Improve parsing of chunked transfer chunks per RFC2616:
* more stringent chunk-size parsing
* ignore optional trailing ';chunk-ext' stuff, instead of barfing
* detect EOF before final \r\n.

Enforce restriction that (http) proxied URL fetchs don't support
being restarted at this time.
PR #28697.

Display times in RFC2822 form rather than using ctime(3), since
the former is more explicit about the timezone offset.

* main: call tzset() to ensure TZ is setup for other <time.h> functions.
* remotemodtime(): use strptime() to parse the reply.
* fetch_url(): ensure struct tm is zeroed before calling strptime().

* Modify parse_url() to consistently strip the leading `/' off ftp URLs.
Fixes PR 17617.
* Use 'RFCnnnn' (with leading 0) instead of 'RFC nnnn', to be
consistent with the style in the RFC index.
* Refer to RFC3916 instead of 1738 or 2732.
* Expand the list of supported RFCs in ftp(1) to contain the document
name as well.

Switch from fparseln() to the internal getline() when parsing HTTP headers.
Makes ftp a bit more portable (not needing fparseln()) at the expense of not
supporting arbitrary long header lines, which I'm not concerned about
because we don't support header line continuation either...

* Implement -s srcaddr; uses srcaddr as the local IP address for all
connections.
Based on code in the version of ftp that FreeBSD had before they
replaced it with lukemftp.
* Move error message handling into ftp_connect() rather than in the
caller, so that more specific error reporting can occur.
* Improve consistency of various warning and error messages.

getpass() can return NULL upon error in some implementations
(as documented in older standards documents, before the API was obsoleted).
Problem observed in tnftp on Solaris by Emil Mikulic.

gcc4 does not care about &foo; use volatile instead. From Anon Ymous

spell precede; from Zafer

PR/34540: b1ff at fr33 dot b33r dot net: bogus free() in ftp(1)
Applied patch, thanks!

If a file upload (via -u) fails, return an non-zero exit value based on the
index of the file that caused the problem (a la auto-fetch retrieval).
Problem noted by A P Garcia in private email.

Coverity CID 2194: Don't forget to free port.

Coverity CID 2195: Free path in all code paths, not some.

rename debug to ftp_debug. grr libssh.

Rename xfoo() to ftp_foo() to avoid collisions with libssh. Don't ask.

recognize 307 redirect code.

Don't clear the trailing character on the auth_url() username;
we now use getline() and that newline strips for us.
Problem found & fixed by Mark Davies.

Add NO_USAGE and NO_DEBUG so that we can fit in the floppies again.

Implement:
int getline(FILE *stream, char *buf, size_t buflen, const char **errormsg)
Read a line from the FILE stream into buf/buflen using fgets(), so up
to buflen-1 chars will be read and the result will be NUL terminated.
If the line has a trailing newline it will be removed.
If the line is too long, excess characters will be read until
newline/EOF/error.
Various -ve return values indicate different errors, and errormsg
will be changed to an error description if it's not NULL.

Convert to use getline() instead of fgets() whenever reading user input
to ensure that an overly long input line doesn't leave excess characters
for the next input operation to accidentally use as input.

Zero out the password & account after we've finished with it.

Consistently use getpass(3) (i.e, character echo suppressed) when
reading the account data. For some reason, historically the "login"
code suppressed echo for Account: yet the "user" command did not!

Display the hostname in the "getaddrinfo failed" warning.

Appease some -Wcast-qual warnings. Fixing all of these requires
significant code refactoring. (mmm, legacy code).

* Only print the "Trying <address>..." message if verbose and
there's more than one struct addrinfo in the getaddrinfo() result.
* Don't use non-standard "u_int".

* fetch_ftp(): preserve 'anonftp' across a disconnect() so that multiple
ftp auto-fetches on the same command line login automatically.
* auto_fetch(): use an initialized volatile int to appease IRIX cc.

Use size_t instead of int where appropriate.

Fix some cast issues highlighted by Scott Reynolds using gcc 4 on OSX.4

gratuitous whitespace cleanup (before someone else jumps the gun...)

In fetch_url(), don't call freeaddrinfo(res0) too early, as we use pointers
to its contents later in the function.
Problem found by Onno van der Linden.

branches: 1.155.2;
Fix ftp url reget when globs are being used.
Provided by Mathieu Arnold <mat@FreeBSD.org>.

* Always decode %xx in a url's user & pass components.
* Only remember {WWW,Proxy}-Authenticate "Basic" challenges; no point
in tracking any others since ftp doesn't support them.
* Improve the parsing of HTTP responses.

Add (unsigned char) cast to ctype functions

Don't base64 encode the trailing NUL in the HTTP basic auth response.
Problem noted by Eric Haszlakiewicz.

Slightly rework SIGINT handling; if we're exiting the auto-fetch stuff
and sigint_raised is non-zero, reset the handler for SIGINT to SIG_DFL
and raise(SIGINT) so that the appropriate wait(3) status is setup.
Based on solution proposed by Ognyan Kulev.
This should really fix PR [pkg/26351].

Improve parsing of HTTP response headers to be more RFC2616 compliant, and
skip LWS (linear white space; CR, LF, space, tab) and the end of lines and
between the field name and the field value. This still isn't 100% compliant,
since we don't support "multi line" responses at this time.
This should fix PR [bin/22611] from TAMURA Kent (although I can't easily
find a http server to reproduce the problem against.)

Fix a minor memory leak when parsing HTTP response headers.

Don't unnecessarily display a 401/407 error when running with -V.
Fix from PR [bin/18535] by Jeremy Reed.

If an ftp auto-fetch transfer is interrupted by SIGINT (usually ^C),
exit with 130 instead of 1 (or rarely, 0).
This allows an ftp auto-fetch in a shell loop to correctly terminate the loop.
Should fix PR [pkg/26351], and possibly others.

Save approximately 8K by not including http authentication, extended status
messages and help strings when the appropriate options are set.

Don't warn about "ignored setsockopt" failures unless debugging is
enabled. Suggested by Todd Vierling.

Allow empty passwords in ftp://user:@host/file auto-fetch URLs,
per RFC 1738. Requested by Simon Poole.

Update version.

correct URL syntax in comment

* $FTPUSERAGENT overrides the HTTP User-Agent header.
Based on patch from Douwe Kiela.
* Add about:tnftp
* Fix URL in about:netbsd
* Crank version

netbsd.org->NetBSD.org

strlcpy

Uppercase URL.

fix a bogus error message when given a HTTP URL with a trailing
slash.

crank dates

add a cast to appease SUNWspro cc. noted by grant@

tweaks for fparseln(3) move from libutil to libc:
- remove #include <util.h> if nothing else needed it
- remove LDFLAGS+=-lutil if nothing else needed it

- when showing the final progress bar, replace "00:00 ETA" with the
elapsed time. (suggested by simonb)
- actually display transfer stats after a URL fetch. (bug introduced a
*long* time ago)
- update copyright & version

Handle URLs without files correctly (e.g, when using '-o -').
Fix from Anders Dinsen <anders@dinsen.net> in [bin/13768]

- Only send port number in HTTP/1.1 Host: request if port != 80.
Fixes [bin/15415] from Takahiro Kambe <taca@sky.yamashina.kyoto.jp>
- Fix bad mode passed by mls() to recvrequest().
Fixes [bin/16642] from <steve.mcclellan@radisys.com>

portnum is unsigned, use %u instead of %d

update copyrights

Add -4 to force IPv4 and -6 to force IPv6 address usage.
From Hajimu UMEMOTO, via Mike Heffner of FreeBSD.

(FreeBSD has imported NetBSD's ftp as their ftp client;
Mike is sending back some of their local changes).

use u_char instead of char in base64_encode().
problem noticed by Jorgen Lundman in private mail.

don't make broken file with -R option.

handle "*" in Content-Range properly.

If no_proxy condition is true && urltype == FTP_URL_T, use fetch_ftp to retrieve.

convert to use getprogname()

explicitly use SOCK_STREAM with socket() instead of res->ai_socktype,
because it appears that linux with glibc doesn't set the latter
correctly after one of getaddrinfo() or getnameinfo().

base64_encode should be static. picked up by hp/ux(!) compiler

It appears that whilst Apache 1.3.9 incorrectly puts a trailing space
after the chunksize (before the \r\n), Apache 1.3.11 puts *multiple*
trailing spaces after the chunksize. I 'm fairly certain that this is
contrary to RFC 2068 section 3.6, but whatever...
Found by David Brownlee <abs@mono.org>

* implement parseport(), which takes a string and attempts to convert
it to a numeric port number
* use parseport() in parse_url() and hookup()
* don't try and lookup the port number using getaddrinfo(), as it's too hard
to separate a failed host name lookup from a failed service name lookup.
this was causing lossage on systems that don't have `http' in services(5)
(such as solaris), but only crept in when we started using getaddrinfo()
unconditionally.

- rename NO_QUAD to NO_LONG_LONG, QUAD* -> LL* and add ULL* (unsigned)
equivalents. name change suggested by Klaus Klein <kjk@netbsd.org>
- change defined(BSD4_4) || HAVE_SIN_LEN tests into HAVE_SOCKADDR_SA_LEN,
and set the latter if BSD4_4 exists

* always use getaddrinfo() and getnameinfo() instead of maintaining two code
paths. (lukemftp will provide replacements for these on older systems)
* rename __USE_SELECT to USE_SELECT
* rename BSD4_4 to HAVE_SIN_LEN
* replace union sockunion {} with struct sockinet {}, and modify the code
accordingly. this is possibly more portable, as it doesn't rely upon the
structure alignment within the union for our own stuff.
(XXX: haven't tested the ipv6 stuff)

clean up NO_QUAD support: create helper #defines and use as appropriate:
#define NOQUAD ! NOQUAD
------- ------ - ------
QUADF "%ld" "%lld"
QUADFP(x) "%" x "ld" "%" x "lld"
QUADT long long long
STRTOL(x,y,z) strtol(x,y,z) strtoll(x,y,z)

minor knf. call setbinary()/setascii() with non-NULL 2nd arg

errx?/warnx? audit. do not pass variable alone, use %s. idea from openbsd

branches: 1.116.2;
* migrate the SYST parsing from setpeer() into a separate remotesyst().
call remotesyst() only when login has been successful
some servers don't let you run SYST until you've successfully logged in.
* in fetch_ftp(), always call setpeer() with autologin disabled, and use
the following ftp_login() to DTRT.
this prevents ftp from trying to login a second time if the first autologin
fails when connecting to a remote site anonymously using autofetch.
* reset unix_proxy and unix_server in cleanuppeer()
* missed a function conversion in the KNF sweep...

- fix ai_unmapped() to be a no-op in the !def INET6 case
- display `(-INET6)' at the end of the version string if !def INET6
- clarify in the man page that IPv6 support may not be present (for lukemftp :)

Add support for 'ftp -u url file ...', to upload a list of files to given url.
Mostly based on [bin/10019] by Scott Aaron Bamford <sab@ansic.net>

convert IPv4 mapped address (::ffff:10.1.1.1) into real IPv4 address
before touching it. IPv4 mapped address complicates too many things
in FTP protocol handling.

branches: 1.112.2;
do not pass scoped IPv6 address notation on Host: directive, since
scope identifier is local to the originating node.
do not allow scoped IPv6 address notation in URL, if it is via proxy.

convert to ANSI KNF

* Add support for `fget localfile', which reads a list of filenames to
retrieve from localfile. Based on work by Darren Reed.
* Crank version.
* Update copyright dates.

fixes from cgd:
* sanity check a length (otherwise certain bogus responses can crash ftp)
* allow a transfer encoding type of `binary'; certain firewall vendors
return this bogus type...

s/strtoq/strtoll/ (the latter is standardised)

make debugging output unambiguous on IPv6 numeric addrs (don't use host:port)

http://[::1]:8080/ is legal.

send Host: directive with RFC2732 bracket notation for IPv6 numeric,
otherwise "host:port" is ambiguous to servers (clarification will be submitted
as update to RFC2732).

only use getaddrinfo() et al if both NI_NUMERICHOST *and* INET6 are defined...
(allows --disable-ipv6 in lukemftp's configure script to disable this as
well, which is good for testing when it appears getaddrinfo() is borken)

define private type `sigfunc' as
typedef void (*sigfunc) __P((int));
and replace use of sig_t and void (*)(int).

certain other OSes define sig_t differently to that (they add extra arguments),
and it causes problems due to function mismatches, etc...

work around bug in apache 1.3.9 which incorrectly puts a trailing
space after the chunksize.
noted by Jun-ichiro itojun Hagino <itojun@itojun.org> in [bin/9096]

change references from draft-ietf-ipngwg-url-literal-01.txt to RFC 2732

Fix chunked support; probably broke after rate limiting was added.
Problem noticed/debugging assisted by giles lean <giles@nemeton.com.au>.

XXX: rate limiting with chunked xfers might not limit correctly (i.e,
the limit may be too high or too low); fixing this is non trivial,
and will probably occur if i ever rototill fetch_url()

move version into separate file to reduce recompilation after version crank.

* remove unnecessary freeaddrinfo(res), since res0 was changed to be
freed earlier in itojun's last commit. fixes [bin/8948].
* remove `const char *reason'; it was being assigned but not used.

fix memory leak in fetch_url (no freeaddrinfo was there).
sync with recent KAME.

* complete_remote(): use remglob("", ...) instead of remglob(".", ...),
for listings of the current working directory; some ftp servers don't
like `NLST .'.
[noted by Giles Lean <giles@nemeton.com.au>]
* recvrequest(): treat remote=="" as remote==NULL when calling command().
(to support the above change)
* support `[user@]' in `[user@]host' and `[user@]host[:][path]'.
[based on idea (and initial code) from David Maxwell <david@fundy.ca>]
* `idle' may be invoked without any args
* reformat some comments
* reformat usage string in program and man page
* call updateremotepwd() after successful login, not after successful connect
* always call setsockopt(, IPPROTO_IP, IP_TOS, ) (et al); using #if
defined(IPPROTO_IP) doesn't work on certain foreign systems where
enums instead of #defines are used...
[noted by Matthias Pfaller <leo@dachau.marco.de>]

whitespace nits

- implement restarting file:/// non-proxied http:// URLs (with -R).
- fix a semicolono which stopped file:/// from working

- split the version string into product and version
- be consistent about reporting the version between:
+ status command
+ about:version URL fetch
+ User-agent sent in http requests

when using http/1.1 for unproxied http requests, send the port as a number
(e.g, `:80') instead of a name (e.g, `:http'), because the former is the
supported method that actually works... noted by hubertf@netbsd.org

support `about:version'. also display the version in the output of `status'.

new features:
- add `usage'; displays the usage of a command.
implemented by calling the c_handler() with argc = 0, argv = "funcname".
- add `passive auto'; does the same as $FTPMODE=auto.
- add `set [option value]'; display all options, or set an option to a value.
- add `unset option'; unset an option.
- add getoptionvalue() to retrieve an option's value, and replace a few
global variables with calls to this.
- implement cleanuppeer(), which resets various bits of state back to
`disconnected'. call in disconnect() and lostpeer().
- support completing on `options'.
- improve recovery after a SIGINT may have closed the connection.
XXX: there's still a couple to fix

other stuff:
- various consistency fixes in the man page.
- ensure that the command usage strings in the code and man page match reality.
- mput/mget: check that the connection still exists before each xfer.
- minor cosmetic changes in confirm().
- set code correctly in sizecmd() and modtime()
- don't need \n in err() strings.
- change lostpeer to take an argument (rather than casting (sig_t)lostpeer
in signal handlers)
- knf and whitespace police.

a few user interface and cosmetic tweaks:
* confirm(): move from util.c to cmds.c. display mnemonic string in its prompt.
add support for `q' (terminate current xfer), `?' (show help list)
* in various signal handlers, output a linefeed only if fromatty.
* if fgets(stdin) returned NULL (i.e, EOF), clearerr(stdin) because you don't
want future fgets to fail. this is not done for the fgets() in the main
command loop, since ftp will quit at that point.
* unless ftp is invoked with -a, don't retain the anonftp setting between
hosts (`ftp somehost:' sets anonftp, but you don't want that to `stick'
if you close that connection and open a new one).

use sigjmp_buf for sigsetjmp(), instead of jmp_buf.
noted by Havard.Eidnes@runit.sintef.no.

* use sigsetjmp()/siglongjump() instead of setjmp()/longjmp(); the latter
don't save the signal mask on some foreign systems.
* ensure signal handlers don't use stdio and do reset errno if they
don't exit with siglongjmp()
* use a common SIGINT handler for {send,recv}request()

more propaganda :)

* fetch_url(): specifically set SIGQUIT to psummary before each xfer.
(work around editline's override)
* minor cleanup of signal handler (along the lines of similar work in
recvrequest()). the handlers should now be reset everytime the cleanup
handler was callled.

* factor out SIGINFO setting into a handler that is always active (but only
prints out info if bytes > 0). only set the handler if SIGINFO is defined
* hijack SIGQUIT to be the same as SIGINFO (foreign ports have this, and it's
annoying to have SIGQUIT dump core on netbsd when it prints info on other
systems)
* in {recv,send}request(), factor a lot of duplicated code out into a
`cleanup' section at the end
* rework shell() a bit

add TNFi copyright to all files i've done more than a minor amount of work to...

enhancments from Marc Horowitz <marc@mit.edu> to improve connection timeouts:
* implement xsignal_restart(), which only sets the SA_RESTART flag if
specifically requested
* xsignal() is now a wrapper to xsignal_restart(). INFO, USR1, USR2 and WINCH
are restartable, ALRM, INT, PIPE and QUIT are not.
* improve getreply()'s timeout code to take advantage of the above.

other changes:
* improve wording of how globbing works for `classic' URLs (host:path).
suggested by John Refling <johnr@imageworks.com> in relation to PRs
[bin/8519] and [bin/8520]
* always compile in the `edit' command even if NO_EDITCOMPLETE defined.
it's just a no-op in the latter case, which is more consistent to
the users.
* always compile in about: support (i.e, remove NO_ABOUT).
i'm entitled to some vanity in this program...
* clean up some whitespace

* In the !NI_NUMERICHOST case (i.e, getaddrinfo() challenged systems), portnum
should be in host order. found/fixed by Matthias Pfaller <leo@dachau.marco.de>
* parse_url(): improve checking of portnum, and add an extra argument to pass
back the parsed portnum to the caller (reduces a bit of code duplication)
* Move the KAME/WIDE copyrights after the BSD/TNFi ones. Since there was
significantly less code added under the former, it's only fair on the latter.

* fix initialisation of home[]
* fetch_url(): if path would be NULL, return strdup("")

* consistentnly use memset(a,0,c); there were some cases of memset(a,'\0',c)
* remove explicit extern int h_errno; it's in <netdb.h>
* add <termios.h> back to util.c; it contains struct winsize on some systems

remove debugging cruft

* add new commands:
lpage page local files
pdir as dir, but through your $PAGER
pls as ls, but through your $PAGER
* implement docase() (a la dotrans() et al) and use appropriately, rototilling
some duplicated code
* globulize(): modify to return a pointer to the strdup()ed result in all cases,
and hack the code that calls it to take this into account
* replace strcpy() and strncpy() with strlcpy()
* put(), getit(): use some aptly named local vars instead of argv[...]
* delint

* idle(): rename to idlecmd(). certain linux distributions have an incompatible
prototype for idle() in <unistd.h> (which i thought was against namespace
and sensibility guidelines, but...)
* consistently use xsignal() instead of signal(). we get known behaviour
in all cases (SA_RESTART), which is good for some borken foreign systems.
* remove signal.h from most files; it's unnecessary now
* fetch_url(): use `long chunksize' instead of ssize_t; it's more portable, and
we're setting chunksize with strtol() anyway
* xsignal(): only use SA_RESTART if it exists. SunOS 4.x doesn't have it
but has the inverse (SA_INTERRUPT). the original function i was inspired
from had this support (lib/signal.c, W. Richard Stevens' `UNP 2nd ed Vol 1').
* remove <termios.h> from util.c; it should be unnecessary now

* replace ifdefs against __SVR4 and __linux__ with DIRENT_MISSING_D_NAMLEN;
it's more portable and more obvious
* remove the mkgmtime() && HAVE_TIMEGM stuff:
a) why should netbsd have to define HAVE_TIMEGM to compile cleanly?
b) foreign compiles of ftp should just be linked with working
timegm function

a more portable version of this ftp client will be released as a 3rdparty
product; no use polluting our code with half-baked attempts...

* use %lld instead of %qd to print out (long long) vars.
(slightly more portable; e.g, solaris supports this)
* remove some fluff (lint)

fix a couple of thinkos in my recent work:
* abort_remote(): replace borken MIN(4,BUFSIZ) with just BUFSIZ; it
should have been MAX(4,BUFSIZ), but it's probably safe to assume that
BUFSIZ is at least 3... (fix from simonb)
* auth_url(): use the correct variable when calculating a buffer size.

branches: 1.73.2;
* add support for `xferbuf', which sets both `sndbuf' and `rcvbuf'
* document the above three commands
* rototill the way the sndbuf and rcvbuf work. remove resetsockbufsize()
* use the appropriate socket buffer size as the size of the buffer that
the read()/write() loops use. speeds up things in some cases.

replace snprintf() with strlcpy(), strlcat(), or direct assignment
where appropriate. (strlc*() are easier to port to foriegn systems).

XXX; there's still a few snprintf's in the progress meter stuff to convert

* protect more of the AF_INET6 stuff with #ifdef INET6 (for portability)
* in the main data moving loops only call the initial gettimeofday() if
rate throttling is enabled (saves a system call per loop when not
throttling).

add trailing . to message

warn()->warnx() in a couple of places.

The port number might have changed in the proxy case. Re-evaluate it.

fix proxy code. strtol() does not work very well for parsing port names.

make ftp work again with the traditional gethostbyname/getservbyname
interfaces.

fetch_url() fixes:
- just display the hostname:port of the proxy url, rather than the full url.
this prevents someone `shoulder surfing' a proxy username/password
in $http_proxy. [suggested by perry]
- compact verbose notes for http fetchs; now displays
(via host:port, with authorization, with proxy authorization)
with each component being optional.
(and a couple introduced with the ipv6 mods...)
- don't override host with the canonical name; this prevented fetches from
http/1.1 virtual hosts from working if the virtual host was a CNAME.
[noted by bernd]
- call freeaddrinfo() if res was built with getaddrinfo()

fix problem with www authentication: len passed to snprintf() for
auth_url() strings was 1 too small. noted by veego@

* change fetch_ftp() to be fully rfc 1738 compliant; if the URL contains
the an empty directory (e.g, between `some' and `path' in
`ftp://host/some//path'), then execute `CWD ' (without a path).
This command will probably fail on rfc 959 compliant servers, so
issue a warning in this case and bail. [noted by cgd].
(i wonder if the people who wrote rfc 1738 actually realised that this
requirement appears to contravene the spec for `cwd' in rfc 959 ?)
* replace isurl() with isipv6addr(), and use appropriately. fixes
auto-login with `classic ftp URLs' (e.g, `ftp somehost:')
* cleanup and rework some of the ipv6 stuff in parse_url()
* prevent potential coredump in fetch_ftp() when parsing `;type=X'
* KNF a few lines
* fix a couple of comments
* cleanup the man page a bit

Make FTP downloads via HTTP proxy servers work again.

fix 'junk pointer free' issue.

add dual-stack (IPv4/v6) support. hope I broke no other part...

[fear this; more ftp hacking from lukem :-]

features:

don't search for trailing `;type=' in a NULL path...
fixes [bin/7800] by Mason Loring Bliss <mason@acheron.middleboro.ma.us>

include time.h explicitly, don't depend on other include side-effects

split the "SMALL" #ifdefs into ones for NO_ABOUT, NO_EDITCOMPLETE, and
NO_PROGRESS. -DSMALL still implies all of those. progress meter support
isn't necessary for the smallest possible ftp client, but it adds very
little space and makes users' lives much better. Therefore, it should
be enabled for installation media if at all possible.

* fix gate mode to login as `user@realhost' rather than using PASSERVE;
the latter only seemed to work for TIS Gauntlet and not TIS fwtk.
thanks to simonb@netbsd.org for testing this. fixes [bin/5556].
* if EOF (e.g, ^D) is entered at a username/password/account prompt which
happens to use fgets(), exit the login rather than treating EOF as CR.
* don't use the comma operator where separate statements are valid
* always use snprintf to copy stuff into malloced buffers, just in case
typos creep in and mean that the buffer ends up being overflowed

parse http:// urls (and the $http_proxy variable) for [user:[pass]@]
elements, which are used for the initial authentication attempt (if
requested by the server). in the case of $http_proxy, use the values
for proxy authentication.

* make parsing of ftp:// urls more RFC 1738 compliant;
- the path is split on `/', and each directory is CWD-ed into separately.
(from [standards/7484] by Alan Barrett <apb@iafrica.com>)
- support a trailing `;type=X' suffix, where X is a,i, or d. (d isn't
implemented, but it is recognised)
- the only non-compliant behaviour is that empty directories sections
(e.g `//') aren't run as `CWD ' - as a lot of ftpds don't like that.
Instead, treat this as a no-op.
* don't support globbing for ftp urls, since that's technically not
RFC compliant.
* fix a couple of man-page nits

branches: 1.52.2;
* implement -R; restart non-proxied command-line FTP xfers
* fix fetch_ftp() so that hcode parsing is not done for file:// urls
(a } in the wrong place, and code at the wrong indent level...)
* change outfile to being a global (so it gets correctly reset)
* change parse_url to not remove leading '/' for non ftp urls.
whilst this is not totally rfc1738 compliant, other code kinda
assumes this is the case, and it doesn't hurt

Add a few more variables that end up in registers in gcc-2.8.1

* add url_decode() - `in-place' decode %xx escapes in a given url component
* parse_url()
- only look for user[:pass] for an ftp url (per rfc1738)
- strip leading /'s in an ftp url. (almost per rfc 1738)
* fetch_url()
- decode a copy of the path and use that to build local filenames
- send port in http Host: header (suggested by cgd@netbsd.org)
* fetch_ftp()
- url_decode() the user, pass and path
- fix splitting of path into dir & file (partially from [bin/7073])
- don't bother caching the last host; it can cause problems when
using ftp:// transfers, or when the user changes between xfers
* improve documentation of auto-fetched url arguments (especially regarding
escape sequences in ftp:// urls)
* some whitespace & copyright updates

this should fix [bin/7073] William O Ferry <woferry@warp.wofme.com>,
as well as the metaissues raised in that PR.

only print "Requesting..." messages if (verbose)

* -v enables verbose & progress, -V disables both
* set setvbuf(ttyout, NULL, _IOLBF, 0) and remove a bunch of fflush(ttyout).
* use fwrite() instead of write() for progressmeter (don't intermix stdio
with non stdio ops)