default is now 16K

Add -b <buflen> to specify the buffer size.

Ignore EOF from remote.

add timeout for ssl connect

Implement a timeout for SSL connection setup, using -q QUITTIME,
defaulting to 60 seconds.
SSL_connect(3) (unlike connect(2)) doesn't timeout by default.

Adapt ssl error messages destination: if unexpected error
from local API, use warn()/warnx() to stderr;
if expected error from a network operation (e.g., timeouts),
use fprintf to ttyout (which might be stdout).

Consistently use ftp_poll() instead of select();
ssl.c (using select()) was added 7 years after the
previous uses of select() were converted to poll().

Check EAGAIN as well as existing EINTR error from ftp_poll(),
for portability.

Simplify includes

Include "ftp_var.h" instead of various system headers and "extern.h".

Add option sslnoverify to control validation of SSL certificates.
Add netrc processing to fetch-mode (URL on command line) to enable options and autologin
via netrc.
Fix SSL cleanup in some error paths.

Certificate validation is now enabled by default. Set FTPSSLNOVERIFY=1 in environment
or configure a corresponding init macro via netrc to not validate certs (required if
you haven't installed a required CA certificate for OpenSSL).

Discussed with lukem@ on icb.

branches: 1.12.2;
Disable verification for now until we implement installation of trust anchors.

Add cert verification, together with an environment variable "NO_CERT_VERIFY",
to turn it off.

use fetch_*() for I/O with SMALLPROG / !WITH_SSL builds

Adapt the SMALLPROG / -UWITH_SSL build to also use the fetch_*()
methods from ssl.c, instead of using stdio, as stdio isn't robust
when using interruptable signals.

Disable ssl-specific support in the fetch_*() methods if WITH_SSL
isn't defined, so SMALLPROG still doesn't have ssl support (as expected).

The resulting SMALLPROG binary is slightly larger than before
(e.g., 157KiB vs 153KiB on amd64).

Set version to 20210603 for this fix and the SO_KEEPALIVE fix for PR 56129.

PR install/56219

branches: 1.9.4;
ftp: don't use restartable signals

Refactor to not rely upon restartable signals (SA_RESTART),
possibly fixing intermittent failures with -q QUITTIME.

ftp transfers: handle EINTR/EAGAIN in copy_bytes(),
instead of relying upon restartable signals.

http/https transfers: Explicitly print an error similar to
progressmeter() when timing-out for -Q QUITTIME in fetch_wait(),
and set errno to ETIMEDOUT so that the warn() in fetch_url()
prints a more accurate error message.

PR/55857

branches: 1.8.2;
redo the connection waiting handling to make it more clear.

Make fetch_read() return size_t like fread() does. It is bogus to
have one backing implementation that returns different values and
types than the other. Handle error setting properly; i.e. bail
out if the internal read returned an error. Now we get a proper
error message when the the server resets our connection instead of
a warning that the right failed with an invalid argument.

The server used for testing was:
http://capeweather.dyndns.org:8080/graphs/3474.png
Which seems to be unreliable :-)

branches: 1.6.4;
explicitly include <string.h> since OpenSSL-1.1 does not do it for us.

branches: 1.5.8;
Workaround const issues of SSL_set_tlsext_host_name.

servername cannot be NULL here.

Noted by joerg@.

Add Server Name Indication (SNI) support for https.

Needed for e.g. some github URLs.

branches: 1.2.2; 1.2.6; 1.2.10; 1.2.14;
backwards compatible with netbsd-6.

PR/47276: Add https support

Ignore EOF from remote.

add timeout for ssl connect

Implement a timeout for SSL connection setup, using -q QUITTIME,
defaulting to 60 seconds.
SSL_connect(3) (unlike connect(2)) doesn't timeout by default.

Adapt ssl error messages destination: if unexpected error
from local API, use warn()/warnx() to stderr;
if expected error from a network operation (e.g., timeouts),
use fprintf to ttyout (which might be stdout).

Consistently use ftp_poll() instead of select();
ssl.c (using select()) was added 7 years after the
previous uses of select() were converted to poll().

Check EAGAIN as well as existing EINTR error from ftp_poll(),
for portability.

Simplify includes

Include "ftp_var.h" instead of various system headers and "extern.h".

Add option sslnoverify to control validation of SSL certificates.
Add netrc processing to fetch-mode (URL on command line) to enable options and autologin
via netrc.
Fix SSL cleanup in some error paths.

Certificate validation is now enabled by default. Set FTPSSLNOVERIFY=1 in environment
or configure a corresponding init macro via netrc to not validate certs (required if
you haven't installed a required CA certificate for OpenSSL).

Discussed with lukem@ on icb.

branches: 1.12.2;
Disable verification for now until we implement installation of trust anchors.

Add cert verification, together with an environment variable "NO_CERT_VERIFY",
to turn it off.

use fetch_*() for I/O with SMALLPROG / !WITH_SSL builds

Adapt the SMALLPROG / -UWITH_SSL build to also use the fetch_*()
methods from ssl.c, instead of using stdio, as stdio isn't robust
when using interruptable signals.

Disable ssl-specific support in the fetch_*() methods if WITH_SSL
isn't defined, so SMALLPROG still doesn't have ssl support (as expected).

The resulting SMALLPROG binary is slightly larger than before
(e.g., 157KiB vs 153KiB on amd64).

Set version to 20210603 for this fix and the SO_KEEPALIVE fix for PR 56129.

PR install/56219

branches: 1.9.4;
ftp: don't use restartable signals

Refactor to not rely upon restartable signals (SA_RESTART),
possibly fixing intermittent failures with -q QUITTIME.

ftp transfers: handle EINTR/EAGAIN in copy_bytes(),
instead of relying upon restartable signals.

http/https transfers: Explicitly print an error similar to
progressmeter() when timing-out for -Q QUITTIME in fetch_wait(),
and set errno to ETIMEDOUT so that the warn() in fetch_url()
prints a more accurate error message.

PR/55857

branches: 1.8.2;
redo the connection waiting handling to make it more clear.

Make fetch_read() return size_t like fread() does. It is bogus to
have one backing implementation that returns different values and
types than the other. Handle error setting properly; i.e. bail
out if the internal read returned an error. Now we get a proper
error message when the the server resets our connection instead of
a warning that the right failed with an invalid argument.

The server used for testing was:
http://capeweather.dyndns.org:8080/graphs/3474.png
Which seems to be unreliable :-)

branches: 1.6.4;
explicitly include <string.h> since OpenSSL-1.1 does not do it for us.

branches: 1.5.8;
Workaround const issues of SSL_set_tlsext_host_name.

servername cannot be NULL here.

Noted by joerg@.

Add Server Name Indication (SNI) support for https.

Needed for e.g. some github URLs.

branches: 1.2.2; 1.2.6; 1.2.10; 1.2.14;
backwards compatible with netbsd-6.

PR/47276: Add https support

Simplify includes

Include "ftp_var.h" instead of various system headers and "extern.h".

Add option sslnoverify to control validation of SSL certificates.
Add netrc processing to fetch-mode (URL on command line) to enable options and autologin
via netrc.
Fix SSL cleanup in some error paths.

Certificate validation is now enabled by default. Set FTPSSLNOVERIFY=1 in environment
or configure a corresponding init macro via netrc to not validate certs (required if
you haven't installed a required CA certificate for OpenSSL).

Discussed with lukem@ on icb.

Disable verification for now until we implement installation of trust anchors.

Add cert verification, together with an environment variable "NO_CERT_VERIFY",
to turn it off.

use fetch_*() for I/O with SMALLPROG / !WITH_SSL builds

Adapt the SMALLPROG / -UWITH_SSL build to also use the fetch_*()
methods from ssl.c, instead of using stdio, as stdio isn't robust
when using interruptable signals.

Disable ssl-specific support in the fetch_*() methods if WITH_SSL
isn't defined, so SMALLPROG still doesn't have ssl support (as expected).

The resulting SMALLPROG binary is slightly larger than before
(e.g., 157KiB vs 153KiB on amd64).

Set version to 20210603 for this fix and the SO_KEEPALIVE fix for PR 56129.

PR install/56219

branches: 1.9.4;
ftp: don't use restartable signals

Refactor to not rely upon restartable signals (SA_RESTART),
possibly fixing intermittent failures with -q QUITTIME.

ftp transfers: handle EINTR/EAGAIN in copy_bytes(),
instead of relying upon restartable signals.

http/https transfers: Explicitly print an error similar to
progressmeter() when timing-out for -Q QUITTIME in fetch_wait(),
and set errno to ETIMEDOUT so that the warn() in fetch_url()
prints a more accurate error message.

PR/55857

branches: 1.8.2;
redo the connection waiting handling to make it more clear.

Make fetch_read() return size_t like fread() does. It is bogus to
have one backing implementation that returns different values and
types than the other. Handle error setting properly; i.e. bail
out if the internal read returned an error. Now we get a proper
error message when the the server resets our connection instead of
a warning that the right failed with an invalid argument.

The server used for testing was:
http://capeweather.dyndns.org:8080/graphs/3474.png
Which seems to be unreliable :-)

branches: 1.6.4;
explicitly include <string.h> since OpenSSL-1.1 does not do it for us.

branches: 1.5.8;
Workaround const issues of SSL_set_tlsext_host_name.

servername cannot be NULL here.

Noted by joerg@.

Add Server Name Indication (SNI) support for https.

Needed for e.g. some github URLs.

branches: 1.2.2; 1.2.6; 1.2.10; 1.2.14;
backwards compatible with netbsd-6.

PR/47276: Add https support

Add option sslnoverify to control validation of SSL certificates.
Add netrc processing to fetch-mode (URL on command line) to enable options and autologin
via netrc.
Fix SSL cleanup in some error paths.

Certificate validation is now enabled by default. Set FTPSSLNOVERIFY=1 in environment
or configure a corresponding init macro via netrc to not validate certs (required if
you haven't installed a required CA certificate for OpenSSL).

Discussed with lukem@ on icb.

Disable verification for now until we implement installation of trust anchors.

Add cert verification, together with an environment variable "NO_CERT_VERIFY",
to turn it off.

use fetch_*() for I/O with SMALLPROG / !WITH_SSL builds

Adapt the SMALLPROG / -UWITH_SSL build to also use the fetch_*()
methods from ssl.c, instead of using stdio, as stdio isn't robust
when using interruptable signals.

Disable ssl-specific support in the fetch_*() methods if WITH_SSL
isn't defined, so SMALLPROG still doesn't have ssl support (as expected).

The resulting SMALLPROG binary is slightly larger than before
(e.g., 157KiB vs 153KiB on amd64).

Set version to 20210603 for this fix and the SO_KEEPALIVE fix for PR 56129.

PR install/56219

branches: 1.9.4;
ftp: don't use restartable signals

Refactor to not rely upon restartable signals (SA_RESTART),
possibly fixing intermittent failures with -q QUITTIME.

ftp transfers: handle EINTR/EAGAIN in copy_bytes(),
instead of relying upon restartable signals.

http/https transfers: Explicitly print an error similar to
progressmeter() when timing-out for -Q QUITTIME in fetch_wait(),
and set errno to ETIMEDOUT so that the warn() in fetch_url()
prints a more accurate error message.

PR/55857

branches: 1.8.2;
redo the connection waiting handling to make it more clear.

Make fetch_read() return size_t like fread() does. It is bogus to
have one backing implementation that returns different values and
types than the other. Handle error setting properly; i.e. bail
out if the internal read returned an error. Now we get a proper
error message when the the server resets our connection instead of
a warning that the right failed with an invalid argument.

The server used for testing was:
http://capeweather.dyndns.org:8080/graphs/3474.png
Which seems to be unreliable :-)

branches: 1.6.4;
explicitly include <string.h> since OpenSSL-1.1 does not do it for us.

branches: 1.5.8;
Workaround const issues of SSL_set_tlsext_host_name.

servername cannot be NULL here.

Noted by joerg@.

Add Server Name Indication (SNI) support for https.

Needed for e.g. some github URLs.

branches: 1.2.2; 1.2.6; 1.2.10; 1.2.14;
backwards compatible with netbsd-6.

PR/47276: Add https support

Disable verification for now until we implement installation of trust anchors.

Add cert verification, together with an environment variable "NO_CERT_VERIFY",
to turn it off.

use fetch_*() for I/O with SMALLPROG / !WITH_SSL builds

Adapt the SMALLPROG / -UWITH_SSL build to also use the fetch_*()
methods from ssl.c, instead of using stdio, as stdio isn't robust
when using interruptable signals.

Disable ssl-specific support in the fetch_*() methods if WITH_SSL
isn't defined, so SMALLPROG still doesn't have ssl support (as expected).

The resulting SMALLPROG binary is slightly larger than before
(e.g., 157KiB vs 153KiB on amd64).

Set version to 20210603 for this fix and the SO_KEEPALIVE fix for PR 56129.

PR install/56219

branches: 1.9.4;
ftp: don't use restartable signals

Refactor to not rely upon restartable signals (SA_RESTART),
possibly fixing intermittent failures with -q QUITTIME.

ftp transfers: handle EINTR/EAGAIN in copy_bytes(),
instead of relying upon restartable signals.

http/https transfers: Explicitly print an error similar to
progressmeter() when timing-out for -Q QUITTIME in fetch_wait(),
and set errno to ETIMEDOUT so that the warn() in fetch_url()
prints a more accurate error message.

PR/55857

branches: 1.8.2;
redo the connection waiting handling to make it more clear.

Make fetch_read() return size_t like fread() does. It is bogus to
have one backing implementation that returns different values and
types than the other. Handle error setting properly; i.e. bail
out if the internal read returned an error. Now we get a proper
error message when the the server resets our connection instead of
a warning that the right failed with an invalid argument.

The server used for testing was:
http://capeweather.dyndns.org:8080/graphs/3474.png
Which seems to be unreliable :-)

branches: 1.6.4;
explicitly include <string.h> since OpenSSL-1.1 does not do it for us.

branches: 1.5.8;
Workaround const issues of SSL_set_tlsext_host_name.

servername cannot be NULL here.

Noted by joerg@.

Add Server Name Indication (SNI) support for https.

Needed for e.g. some github URLs.

branches: 1.2.2; 1.2.6; 1.2.10; 1.2.14;
backwards compatible with netbsd-6.

PR/47276: Add https support

Add cert verification, together with an environment variable "NO_CERT_VERIFY",
to turn it off.

use fetch_*() for I/O with SMALLPROG / !WITH_SSL builds

Adapt the SMALLPROG / -UWITH_SSL build to also use the fetch_*()
methods from ssl.c, instead of using stdio, as stdio isn't robust
when using interruptable signals.

Disable ssl-specific support in the fetch_*() methods if WITH_SSL
isn't defined, so SMALLPROG still doesn't have ssl support (as expected).

The resulting SMALLPROG binary is slightly larger than before
(e.g., 157KiB vs 153KiB on amd64).

Set version to 20210603 for this fix and the SO_KEEPALIVE fix for PR 56129.

PR install/56219

branches: 1.9.4;
ftp: don't use restartable signals

Refactor to not rely upon restartable signals (SA_RESTART),
possibly fixing intermittent failures with -q QUITTIME.

ftp transfers: handle EINTR/EAGAIN in copy_bytes(),
instead of relying upon restartable signals.

http/https transfers: Explicitly print an error similar to
progressmeter() when timing-out for -Q QUITTIME in fetch_wait(),
and set errno to ETIMEDOUT so that the warn() in fetch_url()
prints a more accurate error message.

PR/55857

branches: 1.8.2;
redo the connection waiting handling to make it more clear.

Make fetch_read() return size_t like fread() does. It is bogus to
have one backing implementation that returns different values and
types than the other. Handle error setting properly; i.e. bail
out if the internal read returned an error. Now we get a proper
error message when the the server resets our connection instead of
a warning that the right failed with an invalid argument.

The server used for testing was:
http://capeweather.dyndns.org:8080/graphs/3474.png
Which seems to be unreliable :-)

branches: 1.6.4;
explicitly include <string.h> since OpenSSL-1.1 does not do it for us.

Workaround const issues of SSL_set_tlsext_host_name.

servername cannot be NULL here.

Noted by joerg@.

Add Server Name Indication (SNI) support for https.

Needed for e.g. some github URLs.

branches: 1.2.2; 1.2.6; 1.2.10; 1.2.14;
backwards compatible with netbsd-6.

PR/47276: Add https support

use fetch_*() for I/O with SMALLPROG / !WITH_SSL builds

Adapt the SMALLPROG / -UWITH_SSL build to also use the fetch_*()
methods from ssl.c, instead of using stdio, as stdio isn't robust
when using interruptable signals.

Disable ssl-specific support in the fetch_*() methods if WITH_SSL
isn't defined, so SMALLPROG still doesn't have ssl support (as expected).

The resulting SMALLPROG binary is slightly larger than before
(e.g., 157KiB vs 153KiB on amd64).

Set version to 20210603 for this fix and the SO_KEEPALIVE fix for PR 56129.

PR install/56219

ftp: don't use restartable signals

Refactor to not rely upon restartable signals (SA_RESTART),
possibly fixing intermittent failures with -q QUITTIME.

ftp transfers: handle EINTR/EAGAIN in copy_bytes(),
instead of relying upon restartable signals.

http/https transfers: Explicitly print an error similar to
progressmeter() when timing-out for -Q QUITTIME in fetch_wait(),
and set errno to ETIMEDOUT so that the warn() in fetch_url()
prints a more accurate error message.

PR/55857

branches: 1.8.2;
redo the connection waiting handling to make it more clear.

Make fetch_read() return size_t like fread() does. It is bogus to
have one backing implementation that returns different values and
types than the other. Handle error setting properly; i.e. bail
out if the internal read returned an error. Now we get a proper
error message when the the server resets our connection instead of
a warning that the right failed with an invalid argument.

The server used for testing was:
http://capeweather.dyndns.org:8080/graphs/3474.png
Which seems to be unreliable :-)

branches: 1.6.4;
explicitly include <string.h> since OpenSSL-1.1 does not do it for us.

Workaround const issues of SSL_set_tlsext_host_name.

servername cannot be NULL here.

Noted by joerg@.

Add Server Name Indication (SNI) support for https.

Needed for e.g. some github URLs.

branches: 1.2.2; 1.2.6; 1.2.10; 1.2.14;
backwards compatible with netbsd-6.

PR/47276: Add https support

ftp: don't use restartable signals

Refactor to not rely upon restartable signals (SA_RESTART),
possibly fixing intermittent failures with -q QUITTIME.

ftp transfers: handle EINTR/EAGAIN in copy_bytes(),
instead of relying upon restartable signals.

http/https transfers: Explicitly print an error similar to
progressmeter() when timing-out for -Q QUITTIME in fetch_wait(),
and set errno to ETIMEDOUT so that the warn() in fetch_url()
prints a more accurate error message.

PR/55857

redo the connection waiting handling to make it more clear.

Make fetch_read() return size_t like fread() does. It is bogus to
have one backing implementation that returns different values and
types than the other. Handle error setting properly; i.e. bail
out if the internal read returned an error. Now we get a proper
error message when the the server resets our connection instead of
a warning that the right failed with an invalid argument.

The server used for testing was:
http://capeweather.dyndns.org:8080/graphs/3474.png
Which seems to be unreliable :-)

branches: 1.6.4;
explicitly include <string.h> since OpenSSL-1.1 does not do it for us.

Workaround const issues of SSL_set_tlsext_host_name.

servername cannot be NULL here.

Noted by joerg@.

Add Server Name Indication (SNI) support for https.

Needed for e.g. some github URLs.

branches: 1.2.2; 1.2.6; 1.2.10; 1.2.14;
backwards compatible with netbsd-6.

PR/47276: Add https support

redo the connection waiting handling to make it more clear.

Make fetch_read() return size_t like fread() does. It is bogus to
have one backing implementation that returns different values and
types than the other. Handle error setting properly; i.e. bail
out if the internal read returned an error. Now we get a proper
error message when the the server resets our connection instead of
a warning that the right failed with an invalid argument.

The server used for testing was:
http://capeweather.dyndns.org:8080/graphs/3474.png
Which seems to be unreliable :-)

explicitly include <string.h> since OpenSSL-1.1 does not do it for us.

Workaround const issues of SSL_set_tlsext_host_name.

servername cannot be NULL here.

Noted by joerg@.

Add Server Name Indication (SNI) support for https.

Needed for e.g. some github URLs.

branches: 1.2.2; 1.2.6; 1.2.10; 1.2.14;
backwards compatible with netbsd-6.

PR/47276: Add https support

explicitly include <string.h> since OpenSSL-1.1 does not do it for us.

Workaround const issues of SSL_set_tlsext_host_name.

servername cannot be NULL here.

Noted by joerg@.

Add Server Name Indication (SNI) support for https.

Needed for e.g. some github URLs.

branches: 1.2.2; 1.2.6; 1.2.10; 1.2.14;
backwards compatible with netbsd-6.

PR/47276: Add https support