MFC r326276:

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

MFC r314659,r314676:

r314659:

usr.sbin: normalize paths using SRCTOP-relative paths or :H when possible

This simplifies make logic/output

r314676:

Fix build after r314656

Some of the changes I introduced to use .ALLSRC were correct in spirit,
but incorrect in reality -- in particular, ../Makefile.inc hadn't been
pulled in via bsd.init.mk (via bsd.lib.mk, bsd.prog.mk), so the value
of .ALLSRC (evaluated immediately) was empty. .include bsd.init.mk
explicitly so we can be certain that the values used as dependencies in
the targets are defined when the target recipe has been evaluated.

Reminder: thou shalt separate out separate functional changes before
committing them.

(YUGE) Pointyhat to: ngie
In collaboration with: bdrewery

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Remove some unneeded headers

Remove extra debug that crept in

Implement pubkey support for the bootstrap

Note that to not interfer with finger print it expects a signature on pkg itself
which is named pkg.txz.pubkeysign

To genrate it:
echo -n "$(sha256 -q pkg.txz)" | openssl dgst -sha256 -sign /thekey \
-binary -out ./pkg.txz.pubkeysig

Note the "echo -n" which prevent signing the '\n' one would get otherwise

PR: 202622
MFC after: 1 week

Fix indentation, no functional changes

Plug a possible memory leak.

MFC after: 2 weeks

Issue warning and refuse to proceed further if the configured
repository signature_type is unsupported by bootstrap pkg(7).

Previously, when signature_type specified an unsupported method,
the bootstrap pkg(7) would proceed like when signature_type is
"none". MITM attackers may be able to use this vulnerability and
bypass validation and install their own versions of pkg(8).

At this time, only fingerprint and none are supported by the
bootstrap pkg(7).

FreeBSD's official pkg(8) repository uses the fingerprint method
and is therefore unaffected.

Errata candidate.

Discussed with: bapt@
Submitted by: Fabian Keil
Obtained from: ElectroBSD

Add META_MODE support.

Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.

Still need to add real targets under targets/ to build packages.

Differential Revision: D2796
Reviewed by: brooks imp

Allow fetching pkg(8) even if servers/proxies are not passing Content-length

Test the return of fetchParseURL(3)

CID: 1125811
MFC after: 1 week

Plug resources leak

CID: 1125813
CID: 1125807
CID: 1125808
MFC after: 1 week

Convert usr.sbin to LIBADD
Reduce overlinking

Remove spurious libelf dependency.

Submitted by: pluknet

Use pkg-1.4-style platform identifiers based on MACHINE_ARCH (e.g.
FreeBSD:11:amd64 instead of freebsd:11:x86:64) when bootstrapping pkg.
Thanks to portmgr for providing symlinks so both styles work.

Reviewed by: bapt
MFC after: 3 weeks

Rework privatelib/internallib

Make sure everything linking to a privatelib and/or an internallib does it directly
from the OBJDIR rather than DESTDIR.
Add src.libnames.mk so bsd.libnames.mk is not polluted by libraries not existsing
in final installation
Introduce the LD* variable which is what ld(1) is expecting (via LDADD) to link to
internal/privatelib
Directly link to the .so in case of private library to avoid having to complexify
LDFLAGS.

Phabric: https://phabric.freebsd.org/D553
Reviewed by: imp, emaste

When we fail to extract the pkg binaries (for example, / is read-only),
give a more helpful error message.

MFC after: 1 week

Remove ia64.

This includes:
o All directories named *ia64*
o All files named *ia64*
o All ia64-specific code guarded by __ia64__
o All ia64-specific makefile logic
o Mention of ia64 in comments and documentation

This excludes:
o Everything under contrib/
o Everything under crypto/
o sys/xen/interface
o sys/sys/elf_common.h

Discussed at: BSDcan

Import libucl 0.4.0
Adapt pkg(7) to the new libucl API

Add missing LIBM to DPADD from r263650

Reported by: jilles

Fix build with libucl 20140321

Fix ASSUME_ALWAYS_YES not being parsed properly from config after UCL conversion.

Sponsored by: EMC / Isilon Storage Division
MFC after: 1 week

Fix ABI from /usr/local/etc/pkg.conf not being respected.

Regression from r259266.

Sponsored by: EMC / Isilon Storage Division
MFC after: 1 week

Remove a useless newline, warnx already appends a newline

Update my copyright

Switch pkg(7) from libyaml to libucl

Implicit include of sys/queue.h instead of relying on gelf.h/libelf.h to bring it
Newer version of gelf.h and libelf.h does not include sys/queue.h anymore

Submitted by: kaiw
MFC after: 3 days

Fix yet another typo

Reported by: kib

Fix typos

Reported by: bryanv

Do not print a question in non-interractive mode but explain why boostrapping
has failed and how to allow it to happen

Reported by: jmmv

Fix multi-repository support by properly respecting 'enabled' flag.

This will read the REPOS_DIR env/config setting (default is /etc/pkg
and /usr/local/etc/pkg/repos) and use the last enabled repository.

This can be changed in the environment using a comma-separated list,
or in /usr/local/etc/pkg.conf with JSON array syntax of:
REPOS_DIR: ["/etc/pkg", "/usr/local/etc/pkg/repos"]

Approved by: bapt
MFC after: 1 week

Do not create pkg.conf, as it is not necessary anymore and packagesite is deprecated

Reported by: glebius
MFC after: 3 days

Follow-up to r258227 and document 'enabled' as a boolean instead of
a string.

Approved by: bapt
MFC after: 2 days
X-MFC-With: r258227

argc is now used

MFC after: 2 days

Remove useless checks for NULL pointer before free(3)

Reported by: eadler
Approved by: eadler

Do not use just freed memory.

Sponsored by: Nginx, Inc.

Remove condition which can never be true as the previous loop
is never exited while remote == NULL.

Reported by: Coverity
Approved by: bapt
MFC after: 2 days

Use proper capitalization for FreeBSD.org

Approved by: bapt
MFC after: 2 days

Add -f support to 'pkg bootstrap' and 'pkg add' to force installation
of pkg(8) even if already installed.

This is useful if you somehow messup pkg(8) and need to reinstall from
remote with it already being registered in the pkg(8) /var/db/pkg database.

Also add some sanity checks to 'pkg add'.

Approved by: bapt
MFC after: 2 days

A few minor mdoc improvements.

Add a pkg(7) manpage for bootstrap

Approved by: bapt
MFC after: 2 days

Add a 'pkg bootstrap' command which will bootstrap pkg(8) without
forwarding any command to it after installation.

This is useful if the only goal is to install pkg(8) without any extra
output.

Requested by: cperciva
Approved by: bapt
MFC after: 2 days

Fix build with GCC.

BIO_new_mem_buf takes a void* buf, but internally it never modifies the
buf. It assigns the buffer to another pointer and then marks it as
read-only. So deconsting it should be safe here.

Also fix warning about 'buf' possibly being unused in parse_cert()

Approved by: bapt
MFC after: 2 days
X-MFC-With: r257147

Link in libcrypto as well to fix build in some cases.

Approved by: bapt
MFC after: 2 days
Reported by: many

Add support to check the signature of a local pkg.txz file being
added with "pkg add". If the pkg.conf is configured to check for
signature, then the pkg.txz.sig file will be expected and validated
per r257147

Approved by: bapt
MFC after: 2 days

Tell which fingerprint pkg is being validated against.

Approved by: bapt
MFC after: 2 days

Support checking signature for pkg bootstrap.

If the pkg.conf is configured with SIGNATURE_TYPE: FINGERPRINTS,
and FINGERPRINTS: /etc/keys/pkg then a pkg.sig file is fetched along
with pkg.txz. The signature contains the signature provided by the
signing server, and the public key. The .sig is the exact output
from the signing server in the following format:

SIGNATURE
<openssl signed>
CERT
<rsa public key>
END

The signature is verified with the following logic:

- If the .sig file is missing, it fails.
- If the .sig doesn't validate, it fails.
- If the public key in the .sig is not in the known trusted fingerprints,
it fails.
- If the public key is in the revoked key list, it fails.

Approved by: bapt
MFC after: 2 days
Discussed by: bapt with des, jonathan, gavin

Be verbose and tell where pkg(8) is being bootstrapped from.

Approved by: bapt
MFC after: 2 days

Add support for reading configuration files from /etc/pkg.
For now only /etc/pkg/FreeBSD.conf is supported. Its style is:

Repo: {
URL: "...",
MIRROR_TYPE: "...",
...
}

The configuration will be read from /usr/local/etc/pkg.conf if exists,
otherwise /etc/pkg/FreeBSD.conf

Approved by: bapt
MFC after: 2 days

Wrap long lines

Approved by: bapt
MFC after: 2 days

Add support for using "pkg+http://" for the PACKAGESITE.

pkg 1.2 is adding this support as well. This should help
lessen the confusion on why the default SRV PACKAGESITE
does not load in a browser.

Adapated from: matthew's upstream pkg change
Approved by: bapt
MFC after: 2 days

Fix build with gcc

MFC after: 3 days

Allow to bootstrap by doing pkg add ./a/path/to/a/pkg_package.txz

Requested by: many
MFC after: 3 days

Improve SRV records support for the pkg(8) bootstrap:
- order srv records by priorities
- for all entries of the same priority, order randomly respect the weight
- select the port where to fetch from respect the port provided in the SRV record

Obtained from: pkg git repo
MFC after: 3 days

Improve bootstrapping message

MFC after: 3 days

Rename libbsdyml to libyaml, make private, and bump
SHLIB_MAJOR to 1.0

Suggested by: des
Approved by: bapt
MFC after: 1 week

Cleanup elf macros

Only define EF_MIPS_ABI when not already supplied
Remove old now unused ARM macros

Reported by: imp
Approved by: re (kib)

Add support to detect arm vs armv6

There are two different versions of the ARM ABI depending on the
TARGET_ARCH. As these are sligntly different a package built for
one may not work on another. We need to detect which one we are on
by parsing the .ARM.attributes section.

This will only work on the ARM EABI as this section is part of the
ABI definition. As armv6 only supports the ARM EABI this is not a
problem for the oabi.

Older versions of libelf in FreeBSD fail to read the
.ARM.attributes section needed. As armv6 is unsupported on these
versions we can assume we are running on arm.

Submitted by: andrew
Approved by: re (delphij)
Obtained from: pkgng git

Fix detection of arm ABIs

Submitted by: andrew
Obtained from: pkg git

Fix a bad calloc(3) call

MFC after: 2 weeks

Fix a typo in DPADD.

Fix bootstrapping pkgng by not appending "/latest/" to PACKAGESITE.

Reporter by : Alexander Yerenkow (via irc)

choose in priority the allocated ABI if any to do the ${ABI} substitution in PACKAGESITE

Reviewed by: bdrewery

Add the ability to correctly read pkg.conf is exists.

Only look for boostrap useful options:
- PACKAGESITE
- ABI
- MIRROR_TYPE
- ASSUME_ALWAYS_YES

While here makes PACKAGESITE expand the ${ABI} variable.
Allow to deactivate any SRV record look up (MIRROR_TYPE=none)
Use the same mechanism as for pkgng itself: first get configuration out of
environment variable and fallback on pkg.conf if exists.

Reviewed by: bdrewery

Do not use deprecated functions from libarchive

In case of failure of the pkg boostrap advice the user to either change the
PACKAGESITE they use or install from ports directly indicating where the port
is localted in the port collection

Submitted by: kientzle

Directly uses calloc(3) instread of malloc(3) + memset(3)

Reported by: Jeremy Chadwick <jdc@koitsu.org>
MFC after: 3 days

Remove extraneous space and new-line.

Submitted by: pjd, gcooper
Approved by: bapt
MFC after: 2 weeks

Third time's the charm. pkg -N output needs to go to stderr.

Approved by: bapt
MFC after: 2 weeks

Switch from 'pkg -n' to 'pkg -N' as the test for pkgn activation
status. '-n' is already used extensively elsewhere in pkgng (to mean
'dry-run') and this reduces the potential confusion

Submitted by: eadler
Approved by: bapt
MFC after: 2 weeks

In preparation for making 'pkg -n' the one true method of determining
whether a system has been configured to use pkgng, cause /usr/sbin/pkg
recognise a -n option and exit with a failure code when the pkg port
is not installed.

Approved by: bapt
MFC after: 2 weeks

if PACKAGESITE url scheme is not 'file://', always try to revolv SRV records and
use them if any. It allows the bootstrap to use directly pkg.FreeBSD.org instead
of pkgbeta.FreeBSD.org

MFC after: 1 month

More -Wmissing-variable-declarations fixes.

In addition to adding `static' where possible:

- bin/date: Move `retval' into extern.h to make it visible to date.c.
- bin/ed: Move globally used variables into ed.h.
- sbin/camcontrol: Move `verbose' into camcontrol.h and fix shadow warnings.
- usr.bin/calendar: Remove unneeded variables.
- usr.bin/chat: Make `line' local instead of global.
- usr.bin/elfdump: Comment out unneeded function.
- usr.bin/rlogin: Use _Noreturn instead of __dead2.
- usr.bin/tset: Pull `Ospeed' into extern.h.
- usr.sbin/mfiutil: Put global variables in mfiutil.h.
- usr.sbin/pkg: Remove unused `os_corres'.
- usr.sbin/quotaon, usr.sbin/repquota: Remove unused `qfname'.

Fix confirmation logic when detecting a tty

Reported by: mjg

- change ALWAYS_ASSUME_YES to ASSUME_ALWAYS_YES for consistency with pkg(8)
- if not on a tty prompt about the missing pkg(8) but default on 'no' except if
ASSUME_ALWAYS_YES is set

MFC after: 2 days

Make pkg bootstrap program ask for confirmation before proceeding.

The previous behaviour was to silently download and install the pkg
package, without ever telling user about what it was doing and why.

Discussed with: bapt
Reviewed by: kib

Add missing \n when generating pkg.conf

Reported by: beat
Approved by: des (mentor)
MFC after: 1 day

- close the open file after fetching
- create a default /usr/local/etc/pkg.conf

Approved by: des (mentor)

- Use _PATH_TMP instead of hardcoded /tmp
- more style(9) fixes
- some refactoring
- better error detection
- Add the DPADD to Makefile

Submitted by: Garrett Cooper <yanegomi@gmail.com>
Approved by: des (mentor)

- Fix style(9) bugs + inconsistencies

Submitted by: marius
Approved by: des (mentor)

Do not do double initialisation
style(9) says for (;;) not while (1)

Reported by: culot
Approved by: des

add usr.sbin/pkg which is a bootstrap tool for pkgng.
it respects PACKAGESITE, PACKAGEROOT, and a new environment variable ABI (if a user want to use a different API from the base one for its packages)
it has no man page on purpose to avoid hidding the pkg(8) man page from the pkgng package.
for now uses pkgbeta.FreeBSD.org as default mirror to find its package
it respects MK_PKGTOOLS

Approved by: des (mentor)