usr.sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

pkg(7): now that we do use libmd, use it completly

Use SHA256_Fd and SHA256_Data instead of home made equivalent.
wrap those functions into hash.c to avoid header collition between
openssl and libmd

Suggested by: kevans

pkg(7): use libmd for sha256 instead of openssl

OpenSSL 3.0 has deprecated the sha256 api, let's use libmd which has the
same API instead.

In order to avoid the collision in definitions (sha256.h cannot be
included in the same file as a file where openssl headers has been
included) let's move the sha256 related code in its own file

PR: 270023
Reported by: ngie

Revert "grep: remove tautological condition"

This reverts commit f6d6c66889001208aee7b0c46efe1c8ddffda57c.

Gremlins snuck into my tree and injected some WIP.

grep: remove tautological condition

st_size is an off_t, it cannot hold values larger than OFF_MAX.

CID: 1008931

pkg(8): Remove a double word in a source code comment

- s/that that/that/

MFC after: 3 days

pkg: Add limited --debug/-d support

Add an internal debug level global:
- Level 1 (-d) currently does nothing.

- Level 2 (-d -d) enables libfetch debugging (quite verbose) so it's
possible to see what pkg is attempting to download without having
to sniff traffic.

Reviewed by: debdrup, bapt
Differential Revision: https://reviews.freebsd.org/D35756

/usr/sbin/pkg: don't sleep on fetch failure

PR: 263458

pkg: fix regression that breaks bootstrap from not a current directory.

Fixes: 40b9f924b18
Reviewed by: kevans

pkg: allow multiple add arguments again

While pkg(7) add only handles a single 'add' argument, pkg-add(8) fully
handles multiple arguments.

Stop rejecting it, just turn off local-bootstrap mode and proceed to
remote bootstrap if we need it.

While we're here, check if the first argument to pkg add is even a pkg
package. If it's not, also do remote bootstrap instead. Future work
could improve this altogether by picking out a pkg package out of many
and local bootstrap then pass the rest through to the newly installed
pkg.

Reviewed by: bapt, manu (earlier version)
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D28766

pkg(7): replace usage of sbuf(9) with open_memstream(3)

open_memstream(3) is a standard way to obtain the same feature we do get
by using sbuf(9) (aka dynamic size buffer), switching to using it makes
pkg(7) more portable, and reduces its number of dependencies.

Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D30005

pkg: settle the uniq extension to .pkg instead of .bsd

Requested by: many

pkg(7): when bootstrapping first search for pkg.bsd file then pkg.txz

The package extension is going to be changed to .bsd to be among other
things resilient to the change of compression format used and reduce
the impact of all third party tool of that change.

Ensure the bootstrap knows about it

Reviewed by: manu
Differential revision: https://reviews.freebsd.org/D29232

pkg(7): address minor nits (mostly clang-analyze complaints)

- One (1) spurious whitespace.
- One (1) occurrence of "random(3) bad, arc4random(3)" good.
- Three (3) writes that will never be seen.

The latter two points are complaints from clang-analyze. Switching to
arc4random(3) is decidedly a good idea because we weren't doing any kind
of PRNG seeding anyways. The discarded assignments are arguably good
for future-proofing, but it's better to improve the S/N ratio from
clang-analyze.

Reviewed by: bapt, manu
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D28525

pkg(7): add an -r reponame option for bootstrap and add

This is limited to bootstrap/add because some real pkg(8) commands
have -r flags with an incompatible meaning/usage, e.g., pkg-audit.
pkg(7) will still commence the search as it has, but it will ignore any
repo objects without the given name so that overrides and whatnot still
work as expected.

The use of it for add is noted in the manpage; notably, that the
signature config for that repository will be used over global config if
it's specified. i.e., pkg(7) should assume that the given pkg did come
from that repository and treat it appropriately.

Reviewed by: bapt, manu
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D28524

pkg(7): rework the arguments handling

Rework the arguments handling around using getopt_long:
* add long option support
* add -4 and -6 support to enforce ipv4 or ipv6

While here fix a regression which occured between FreeBSD 12.1 and
FreeBSD 12.2 where pkg bootstrap -y stopped working

PR: 252270
MFC after: 2 weeks
Submitted by: evilham <contact@evilham.com>
Differential Revision: https://reviews.freebsd.org/D27860

Make use of the getlocalbase() function for run-time adjustment of the
local software base directory, as committed in SVN rev. 367813.

The pkg and mailwrapper programs used the LOCALBASE environment variable
for this purpose and this functionality is preserved by getlocalbase().

After this change, the value of the user.localbase sysctl variable is used
if present (and not overridden in the environment).

The nvmecontrol program gains support of a dynamic path to its plugin
directory with this update.

Differential Revision: https://reviews.freebsd.org/D27237

Revert the whole getlocalbase() set of changes while a different design is
hashed out.

Fix the previous revision, it suffered from an incomplete change to the
getlocalbase API. Also don't erroneously subtract the lenth from the
buffer a second time.

Replace hardcoded references to _PATH_LOCALBASE with calls to getlocalbase.3

Reviewed by: imp, se

Replace literal uses of /usr/local in C sources with _PATH_LOCALBASE

Literal references to /usr/local exist in a large number of files in
the FreeBSD base system. Many are in contributed software, in configuration
files, or in the documentation, but 19 uses have been identified in C
source files or headers outside the contrib and sys/contrib directories.

This commit makes it possible to set _PATH_LOCALBASE in paths.h to use
a different prefix for locally installed software.

In order to avoid changes to openssh source files, LOCALBASE is passed to
the build via Makefiles under src/secure. While _PATH_LOCALBASE could have
been used here, there is precedent in the construction of the path used to
a xauth program which depends on the LOCALBASE value passed on the compiler
command line to select a non-default directory.

This could be changed in a later commit to make the openssh build
consistently use _PATH_LOCALBASE. It is considered out-of-scope for this
commit.

Reviewed by: imp
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D26942

pkg-bootstrap: complain on improper `pkg bootstrap` usage

Right now, the bootstrap will gloss over things like pkg bootstrap -x or
pkg bootstrap -f pkg. Make it more clear that this is incorrect, and hint
at the correct formatting.

Reported by: jhb (IIRC via IRC)
Approved by: bapt, jhb, manu
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D24750

pkg: Provide a friendlier message when bootstrap fails due to address resolution

The current message when bootstapping pkg fails for any reason implies that pkg
is not available. We have the error code from fetch so if bootstrap failed due
to address resolution say so.

Reviewed by: bapt, bz
Approved by: bz (co-mentor)
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D25323

pkg: accept -y and --yes from arguments

By popular demand, pkg now walks thought the arguments passed and
if it finds -y or --yes it does accept those as equivalent of
ASSUME_ALWAYS_YES env var.

Requested by: many
MFC after: 1 week

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

Flush stdout before yes/no confirmations, to force question
through pipes/tee(1)/whatever

Remove usage of _WITH_GETLINE from usr.sbin

Remove extra debug that crept in

Implement pubkey support for the bootstrap

Note that to not interfer with finger print it expects a signature on pkg itself
which is named pkg.txz.pubkeysign

To genrate it:
echo -n "$(sha256 -q pkg.txz)" | openssl dgst -sha256 -sign /thekey \
-binary -out ./pkg.txz.pubkeysig

Note the "echo -n" which prevent signing the '\n' one would get otherwise

PR: 202622
MFC after: 1 week

Fix indentation, no functional changes

Issue warning and refuse to proceed further if the configured
repository signature_type is unsupported by bootstrap pkg(7).

Previously, when signature_type specified an unsupported method,
the bootstrap pkg(7) would proceed like when signature_type is
"none". MITM attackers may be able to use this vulnerability and
bypass validation and install their own versions of pkg(8).

At this time, only fingerprint and none are supported by the
bootstrap pkg(7).

FreeBSD's official pkg(8) repository uses the fingerprint method
and is therefore unaffected.

Errata candidate.

Discussed with: bapt@
Submitted by: Fabian Keil
Obtained from: ElectroBSD

Allow fetching pkg(8) even if servers/proxies are not passing Content-length

Test the return of fetchParseURL(3)

CID: 1125811
MFC after: 1 week

Plug resources leak

CID: 1125813
CID: 1125807
CID: 1125808
MFC after: 1 week

When we fail to extract the pkg binaries (for example, / is read-only),
give a more helpful error message.

MFC after: 1 week

Remove a useless newline, warnx already appends a newline

Update my copyright

Switch pkg(7) from libyaml to libucl

Fix yet another typo

Reported by: kib

Fix typos

Reported by: bryanv

Do not print a question in non-interractive mode but explain why boostrapping
has failed and how to allow it to happen

Reported by: jmmv

Do not create pkg.conf, as it is not necessary anymore and packagesite is deprecated

Reported by: glebius
MFC after: 3 days

argc is now used

MFC after: 2 days

Remove useless checks for NULL pointer before free(3)

Reported by: eadler
Approved by: eadler

Do not use just freed memory.

Sponsored by: Nginx, Inc.

Remove condition which can never be true as the previous loop
is never exited while remote == NULL.

Reported by: Coverity
Approved by: bapt
MFC after: 2 days

Add -f support to 'pkg bootstrap' and 'pkg add' to force installation
of pkg(8) even if already installed.

This is useful if you somehow messup pkg(8) and need to reinstall from
remote with it already being registered in the pkg(8) /var/db/pkg database.

Also add some sanity checks to 'pkg add'.

Approved by: bapt
MFC after: 2 days

Add a 'pkg bootstrap' command which will bootstrap pkg(8) without
forwarding any command to it after installation.

This is useful if the only goal is to install pkg(8) without any extra
output.

Requested by: cperciva
Approved by: bapt
MFC after: 2 days

Fix build with GCC.

BIO_new_mem_buf takes a void* buf, but internally it never modifies the
buf. It assigns the buffer to another pointer and then marks it as
read-only. So deconsting it should be safe here.

Also fix warning about 'buf' possibly being unused in parse_cert()

Approved by: bapt
MFC after: 2 days
X-MFC-With: r257147

Add support to check the signature of a local pkg.txz file being
added with "pkg add". If the pkg.conf is configured to check for
signature, then the pkg.txz.sig file will be expected and validated
per r257147

Approved by: bapt
MFC after: 2 days

Tell which fingerprint pkg is being validated against.

Approved by: bapt
MFC after: 2 days

Support checking signature for pkg bootstrap.

If the pkg.conf is configured with SIGNATURE_TYPE: FINGERPRINTS,
and FINGERPRINTS: /etc/keys/pkg then a pkg.sig file is fetched along
with pkg.txz. The signature contains the signature provided by the
signing server, and the public key. The .sig is the exact output
from the signing server in the following format:

SIGNATURE
<openssl signed>
CERT
<rsa public key>
END

The signature is verified with the following logic:

- If the .sig file is missing, it fails.
- If the .sig doesn't validate, it fails.
- If the public key in the .sig is not in the known trusted fingerprints,
it fails.
- If the public key is in the revoked key list, it fails.

Approved by: bapt
MFC after: 2 days
Discussed by: bapt with des, jonathan, gavin

Be verbose and tell where pkg(8) is being bootstrapped from.

Approved by: bapt
MFC after: 2 days

Wrap long lines

Approved by: bapt
MFC after: 2 days

Add support for using "pkg+http://" for the PACKAGESITE.

pkg 1.2 is adding this support as well. This should help
lessen the confusion on why the default SRV PACKAGESITE
does not load in a browser.

Adapated from: matthew's upstream pkg change
Approved by: bapt
MFC after: 2 days

Allow to bootstrap by doing pkg add ./a/path/to/a/pkg_package.txz

Requested by: many
MFC after: 3 days

Improve SRV records support for the pkg(8) bootstrap:
- order srv records by priorities
- for all entries of the same priority, order randomly respect the weight
- select the port where to fetch from respect the port provided in the SRV record

Obtained from: pkg git repo
MFC after: 3 days

Improve bootstrapping message

MFC after: 3 days

Fix bootstrapping pkgng by not appending "/latest/" to PACKAGESITE.

Reporter by : Alexander Yerenkow (via irc)

Add the ability to correctly read pkg.conf is exists.

Only look for boostrap useful options:
- PACKAGESITE
- ABI
- MIRROR_TYPE
- ASSUME_ALWAYS_YES

While here makes PACKAGESITE expand the ${ABI} variable.
Allow to deactivate any SRV record look up (MIRROR_TYPE=none)
Use the same mechanism as for pkgng itself: first get configuration out of
environment variable and fallback on pkg.conf if exists.

Reviewed by: bdrewery

Do not use deprecated functions from libarchive

In case of failure of the pkg boostrap advice the user to either change the
PACKAGESITE they use or install from ports directly indicating where the port
is localted in the port collection

Submitted by: kientzle

Remove extraneous space and new-line.

Submitted by: pjd, gcooper
Approved by: bapt
MFC after: 2 weeks

Third time's the charm. pkg -N output needs to go to stderr.

Approved by: bapt
MFC after: 2 weeks

Switch from 'pkg -n' to 'pkg -N' as the test for pkgn activation
status. '-n' is already used extensively elsewhere in pkgng (to mean
'dry-run') and this reduces the potential confusion

Submitted by: eadler
Approved by: bapt
MFC after: 2 weeks

In preparation for making 'pkg -n' the one true method of determining
whether a system has been configured to use pkgng, cause /usr/sbin/pkg
recognise a -n option and exit with a failure code when the pkg port
is not installed.

Approved by: bapt
MFC after: 2 weeks

if PACKAGESITE url scheme is not 'file://', always try to revolv SRV records and
use them if any. It allows the bootstrap to use directly pkg.FreeBSD.org instead
of pkgbeta.FreeBSD.org

MFC after: 1 month

Fix confirmation logic when detecting a tty

Reported by: mjg

- change ALWAYS_ASSUME_YES to ASSUME_ALWAYS_YES for consistency with pkg(8)
- if not on a tty prompt about the missing pkg(8) but default on 'no' except if
ASSUME_ALWAYS_YES is set

MFC after: 2 days

Make pkg bootstrap program ask for confirmation before proceeding.

The previous behaviour was to silently download and install the pkg
package, without ever telling user about what it was doing and why.

Discussed with: bapt
Reviewed by: kib

Add missing \n when generating pkg.conf

Reported by: beat
Approved by: des (mentor)
MFC after: 1 day

- close the open file after fetching
- create a default /usr/local/etc/pkg.conf

Approved by: des (mentor)

- Use _PATH_TMP instead of hardcoded /tmp
- more style(9) fixes
- some refactoring
- better error detection
- Add the DPADD to Makefile

Submitted by: Garrett Cooper <yanegomi@gmail.com>
Approved by: des (mentor)

- Fix style(9) bugs + inconsistencies

Submitted by: marius
Approved by: des (mentor)

Do not do double initialisation
style(9) says for (;;) not while (1)

Reported by: culot
Approved by: des

add usr.sbin/pkg which is a bootstrap tool for pkgng.
it respects PACKAGESITE, PACKAGEROOT, and a new environment variable ABI (if a user want to use a different API from the base one for its packages)
it has no man page on purpose to avoid hidding the pkg(8) man page from the pkgng package.
for now uses pkgbeta.FreeBSD.org as default mirror to find its package
it respects MK_PKGTOOLS

Approved by: des (mentor)