MFH (r314778): use reallocarray(3) for extra bounds checks
MFH (r333306): fix typo in man page
MFH (r333571, r333572): preserve if-modified-since across redirects
MFH (r334317): simplify the DEBUG macro
MFH (r334319): style bug roundup
MFH (r334326): fix netrc file location logic, improve netrcfd handling
MFH (r338572): fix end-of-transfer statistics, improve no-tty display

PR: 202424, 224426, 228017

MFC r326276:

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Remove defunct SSLv2 support from fetch(1) and fetch(3).

Increase the default (and minimum) buffer size from 4 kB to 16 kB. Also,
propagate the buffer size to libc, which uses a 1 kB buffer by default,
negating any hypothetical benefit of increasing fetch(1)'s buffer size.

MFC after: 3 days

whitespace and bump copyright

Implement certificate verification, and many other SSL-related
imrovements; complete details in the PR.

PR: kern/175514
Submitted by: Michael Gmelin <freebsd@grem.de>
MFC after: 1 week

Correct check to see if file changes.

Reported by: swildner@DragonFlyBSD.org
Reviewed by: des

Fix bandwidth reporting when doing a restarted download with "-r".
The offset is already accounted for in xs->lastrcvd and doesn't
have to be subtracted again.

Reported by: Florian Smeets <flo@smeets.im>
Submitted by: Mateusz Guzik <mjguzik@gmail.com>
Tested by: Florian Smeets <flo@smeets.im>
MFC after: 1 week

Add check for failure of mkstemp and setenv.

Reviewed by: des
Approved by: cperciva (implicit)
Obtained from: DragonFlyBSD
MFC after: 1 week

Change fetch(1) to:

o Report the instantaneous bandwidth instead of an average since the
beginning of the download.

o At the finish of the download report the average bandwidth and also
the total time it took instead of 00m00s.

Reviewed by: des
MFC after: 1 week

More -Wmissing-variable-declarations fixes.

In addition to adding `static' where possible:

- bin/date: Move `retval' into extern.h to make it visible to date.c.
- bin/ed: Move globally used variables into ed.h.
- sbin/camcontrol: Move `verbose' into camcontrol.h and fix shadow warnings.
- usr.bin/calendar: Remove unneeded variables.
- usr.bin/chat: Make `line' local instead of global.
- usr.bin/elfdump: Comment out unneeded function.
- usr.bin/rlogin: Use _Noreturn instead of __dead2.
- usr.bin/tset: Pull `Ospeed' into extern.h.
- usr.sbin/mfiutil: Put global variables in mfiutil.h.
- usr.sbin/pkg: Remove unused `os_corres'.
- usr.sbin/quotaon, usr.sbin/repquota: Remove unused `qfname'.

Fix two issues related to the use of SIGINFO in fetch(1) to display
progress information. The first is that fetch_read() (used in the HTTP
code but not the FTP code) can enter an infinite loop if it has previously
been interrupted by a signal. The second is that when it is interrupted,
fetch_read() will discard any data it may have read up to that point.
Luckily, both bugs are extremely timing-sensitive and therefore difficult
to trigger.

PR: bin/153240
Submitted by: Mark <markjdb@gmail.com>
MFC after: 3 weeks

Document the fact that passive mode is now the default.
Update copyright dates and strip my middle name.

Use fseeko() instead of fseek(). The rest of the code is off_t-aware,
but the use of fseek() means fetch(1) can't correctly resume a transfer
that was interrupted past the 2 GB mark.

Pointed out by: ache@
MFC after: 3 weeks

Followup to r225599: the fseek() was a no-op since the file was opened
in append mode. Open it in read-write mode instead. Also move the
fseek up one level to cover the (unlikely but not impossible) case where
the server accepts ranges but does not send a Content-Size header.

PR: bin/117277
MFC after: 3 weeks

When resuming an HTTP download, we failed to verify that the range
returned by the server matched what we requested, and blindly appended
what we received to what we already had. This could go two ways: if the
delivered offset was higher than expected, the local file would contain
duplicate data, while if it was lower than expected, there would be data
missing from the middle of the file. Furthermore, if the transfer was
interrupted again, each subsequent attempt would compound the error.
Fix the first problem by restarting the transfer from scratch if there
is a gap, and the second by explicitly seeking to the correct location
in the local file so as to overwrite any duplicated data.

PR: bin/117277
Approved by: re (kib)
MFC after: 3 weeks

Treat an empty argument as an error, instead of fetching the
contents of the root directory.

MFC after: 1 week

Revert most part of 200420 as requested, as more review and polish is
needed.

Remove unneeded header includes from usr.bin/ except contributed code.

Tested with: make universe

The FTP_TIMEOUT and HTTP_TIMEOUT environment variables were ignored because
T_secs already had a non-zero default. Unbreak by moving the default to
ftp_timeout / http_timeout.

Submitted by: Christoph Mallon <christoph.mallon@gmx.de>
MFC after: 2 weeks

1. Update fetch to consistently return 1 on error, as the man page states,
rather than usually returning 1 but in a few instances using a sysexits(3)
return value.

2. Remove a few unused variables from libfetch.

PR: docs/122470 (1, only)
Reviewed by: des
> Description of fields to fill in above: 76 columns --|
> PR: If a GNATS PR is affected by the change.
> Submitted by: If someone else sent in the change.
> Reviewed by: If someone else reviewed your modification.
> Approved by: If you needed approval for this commit.
> Obtained from: If the change is from a third party.
> MFC after: N [day[s]|week[s]|month[s]]. Request a reminder email.
> Security: Vulnerability reference (one per line) or description.
> Empty fields above will be automatically removed.

M usr.bin/fetch/fetch.c
M lib/libfetch/fetch.c

Add support for HTTP 1.1 If-Modified-Since behavior.

fetch(1) accepts a new argument -i <file> that if specified will cause
the file to be downloaded only if it is more recent than the mtime of
<file>.

libfetch(3) accepts the mtime in the url structure and a flag to
indicate when this behavior is desired.

PR: bin/87841
Submitted by: Jukka A. Ukkonen <jau@iki.fi> (partially)
Reviewed by: des, ru
MFC after: 3 weeks

Improve usage and sync SYNOPSIS with usage.

Apply the same defaults to https as to http.

Submitted by: Mike Tancsa <mike@sentex.net>
MFC after: 1 week

Use the SCHEME_ knobs rather than knowing what they expand to.

Reviewed by: des

Try to preserve ownership and permissions when replacing an existing file.

PR: bin/104702
Submitted by: Kevin Day <toasty@dragondata.com>
MFC after: 1 week

Only clear sb.st_size if it is clearly wrong or meaningless. This fixes
mirror mode.

PR: bin/86940
MFC after: 2 weeks

Reflect progress in process title.

Submitted by: flz
MFC after: 2 weeks

Fix buffer overflow. This is FreeBSD-SA-04:16.fetch.

Approved by: des

Update copyright years.

The check for r_flag was accidentally removed in the previous commit.

Submitted by: SANETO Takanori <sanewo@ba2.so-net.ne.jp>
MFC after: 3 days

Fix a couple of edge cases in which sb.st_size may be incorrect or
meaningless. In particular, don't assume that it is left untouched if
stat(2) fails; that assumption happens to fail at high optimization
levels on some platforms.

MFC after: 1 week

Don't strip trailing linear whitespace from passwords.

MFC after: 2 weeks

Style nits.

Truncate long file names in stat_display(), as was originally intended.

MFC after: 1 week

The description of the -S option in the man page says we won't fail if the
remote size is unknown, but we do. Resolve this in the man page's favor.

Requested by: Andre Albsmeier <andre.albsmeier@siemens.com>
MFC after: 1 week

Fix integer overflow in the file size output when dealing with
large files (i.e. DVD images).

Reviewed by: des@

Workaround for servers that ignore byte ranges when using chunked
encoding.

MFC after: 3 days

Use %zu for size_t, like God intended.

Fix printf() format bug in previous commit (size_t != int).
Use %ju and an (uintmax_t) cast to print size_t values.
Use %jd and an (intmax_t) cast to print off_t values.
Use off_t variables to hold the difference between two off_t values.
Don't bother with unsigned char where char will do nicely.

Revamp the statistics code, and switch to a much more compact display
format. The old code tried to produce the exact same output as the
pre-libfetch implementation, but I no longer see any value in this.

Use floating point arithmetic to compute the ETA to avoid integer overflow
during slow transfers of large files.

Clean up the ETA logic a bit and make sure it works for restarted transfers.

Slight cleanup of the ETA / BPS code. Avoid a division by zero.

Add -N option to specify a netrc file.
Fix handling of -v option.
Don't treat negative offsets as valid positive ones.
Clean up the ETA and transfer rate code. Show transfer rate along with
ETA if the verbose level is higher than 1.

Use the URL instead of the output file name when reporting errors relating
to the remote document.

PR: 33856
MFC after: 1 week
Approved by: re

Don't set atime / mtime if we don't have an output file, which can happen
in certain failure scenarii (mostly DNS trouble).

Don't divide by zero to calculate the ETA, even if we haven't gotten
any data in 30 seconds.

Don't exit with success after timeout during authentication.

Don't complain about not knowing the remote file size when working in
quiet mode.

PR: bin/37079
Submitted by: Nicolas Rachinsky <list@rachinsky.de>

Whitespace and indentation cleanup.

Forgot to disable alarms after fetchXGet() in previous commit.

Add an ETA timer that kicks in after 30 seconds.

Re-add alarm(2) calls around the calls to fetchStat(3) and fetchXGet(3),
since these calls can still time out on DNS lookups or TCP connect(2).

Remove the alarm(2) calls in the main loop, since all methods properly
handle transfer timeouts (as opposed to connection timeouts).

Set the sigalrm flag if a timeout occurs in the main loop.

Move the signal: label up a little so we still set the atime and mtime
when the transfer times out or is interrupted, so that restarted transfers
will work as expected (as long as the file still exists).

MFC after: 2 weeks

s/optarg/s/ (cut'n'paste braino)

Spotted by: kris

Use strtol(3) instead of a home-grown version, and fix a "possibly
unitialized variable" warning.

Usage style sweep: spell "usage" with a small 'u'.
Also change one case of blatant __progname abuse (several more remain)
This commit does not touch anything in src/{contrib,crypto,gnu}/.

Restored the part of 1.45 that was clobbered in rev.1.46:
#include <sys/time.h> instead of depending on namespace pollution in
<sys/stat.h> for the declaration of struct timeval.

Add FBSDID. Remove unused include. Getopt returns -1 not EOF.

#include <sys/time.h> instead of depending on namespace pollution in
<sys/stat.h> for the declaration of struct timeval.

Removed unused includes (<time.h> doesn't declare anything of interest;
only <sys/time.h> does).

Sorted includes a bit.

If downloading to a temporary file, set the timestamp on the temp file, not
on the file we're replacing.

PR: bin/34992
Submitted by: Shunichiro Ariura <syun1rou@blackshell.org>
MFC after: 1 week

Set fetchDebug if v_level is 3 or more.

PR: bin/32615
MFC after: 1 week

Remove an extraneous argument to fprintf that -Wformat noticed.

Reviewed by: des

Remove a debugging warnx() that got committed by mistake.

Don't print running stats unless we're the foreground process.

PR: 30764

Add a cast to silence a warning (error because of WARNS=2) on Alpha.

Discussed with: des

If the local file does not exist, or is a regular file, and we're not trying
to resume a transfer, download the requested document into a temporary file
which we later rename. This avoids leaving half-completed files around in
case of a crash (it'll still leave a half-completed file, but with a hope-
fully non-conflicting name), and should reduce the need for human inter-
vention on ports-building machines.

The temporary file name for "foo/bar" is constructed by invoking mkstemps()
with the pattern "foo/.fetch.XXXXXX.bar"

Requested by: obrien

Remove a blank line that snuck in with the previous commit.
Set the default timeout to 120 seconds instead of 0 (no timeout).

Add some comments.

Include value of command line argument that causes an error message or
warning in said message, since fetch may be run from a makefile or script
which does not print the command line.

Approved by: des

If we're writing to stdout, we don't care about the size.

Submitted by: wollman

WARNS cleanup + remove an unused macro and an unused function argument.

Fix an embarassing if-test-reversal bug that broke scheme guessing.

Don't try to print a completion percentage for zero-size files.

PR: 27780
Submitted by: Craig Leres <leres@ee.lbl.gov>

Damn. Fix typo in previous commit.

When running on a tty, install an authentication callback.

Bring the usage message in synch with reality.

PR: bin/26160

Now that libfetch uses the high port range by default, add a -U option to
make it use the low (default) port range instead.

If SIGINFO is received during the transfer, print a status line similar
to the summary shown at the end of the transfer.

Some days, I really hate Unix...

Suggested by: cracauer

Warn if the size of the remote file isn't known.
If the -R option was specified, don't truncate the local file even if its
mtime is incorrect.

PR: bin/23719

If the URL did not specify a scheme, try to guess it from the host name.

If the transfer timed out, but we don't know how large the file is supposed
to be, assume it was truncated.

Format string auditing

Understand the difference between an empty file and a non-existent file.
This has been sitting in my tree for ages...

Don't try to set the mtime of the output file if it's not a regular file.

Pointed out by: cwt

Forced commit to note that the previous commit closed a PR.

PR: 19748

Don't unlink the target file if it's not a regular file.

Restore the old semantics of deleting the output file when interrupted.
Some people liked this and some didn't, so POLA won.

Don't print summary in quiet mode.

Don't display a running count in quiet mode.
Don't add the offset to the size; it's libfetch's job to report the correct
size of the requested file.

Fix two bugs related to resumed transfers:

- if the dates didn't match, fetch would append the received file to the
existing file instead of replacing it.

- if the local file was complete and up-to-date, fetch would miscalculate
the expected size and report a failure instead of a success, because it
had no way of knowing that the server was actually resending the entire
file since the requested offset was invalid.

Braino: don't use passive mode unless the user requests it.

PR: bin/20025
Submitted by: HIYAMA Takeshi <th@cis.ibaraki.ac.jp>

While I'm at it, break a line that was too long, remove a pointless diagnostic
and adjust the verbosity level of another.

Rearrange / rewrite large portions of fetch() to take advantage of new
libfetch features (fetchRestartCalls, fetchXGet()).

Since it doesn't make much sense to have m_flag and r_flag set at the same
time, and it can actually cause trouble in some cases, die if they're both
set.

Set the SA_RESETHAND flag for SIGINT so that when we've caught one, we can
kill ourselves with a second SIGINT (thus notifying our parent of our tragic
fate) instead of just exiting.

These changes fix several problems that would show up when fetching ports,
as well as speeding up HTTP transfers quite a bit (at least for relatively
small files).

Most of these changes were prompted by an interaction problem with an HTTP
server called SWS-1.0, which exhibited two bugs, the first of which prevented
fetch from working around the second (the first was not sending content-type
in reply to HEAD requests, the second was sending garbage after the end of
the requested file).

Better signal handling.

Submitted by: green

Fix a bug in the stats display that showed up when resuming aborted transfers.

Submitted by: green

Rework the stats code for the nth time. Much cleaner now.
Always display the completion percentage if stderr is a tty.
Drop the char-by-char transfer mode, it was based on an incorrect assumption
regarding the semantics of fread().
Finally (I hope) straighten out the business of setting the mtime, as well as
when to remove the output file and when not to.
Thanks are owed to the many who have provided nearly instantaneous and
highly constructive feedback and suggestions about these matters.

Now that the HTTP code supports timeouts, we don't need to use alarm()
around the call to fetchStat().
Catch SIGINT, and rework the signal handling so it doesn't skimp on the
cleanup after a timeout or interrupt. Also, don't just bail out after a
timeout; there may be more files to fetch.

Rework the stats code to avoid code duplication.
Fix a bug where the stats code would print the expected size instead of the
number of bytes received.
Fix the reading code so it'll support partial reads.

Don't segfault if given an invalid URL (doh!)

Submitted by: green

Straighten out the behvaiour of -m and -n (thanks to bde).
Check that stderr, not stdout, is a tty (thanks to green).

Implement the -c option

Don't forget to delete the output file if the request fails.
Don't delete the output file if -r was specified.

New libfetch-based fetch.