move to use v5.36;
tested by me over the last few weeks, and tb@
also fixed a "manual install" bug properly reported by tb@

aside that there should be *no functional change*.
If you see any message like "hey, the number of params is wrong"
it is a fringe case I didn't run into and should be easy to fix.

Refactor locator code so it's a proper object.
allow fw_update to short-circuit entirely, thus the specific
treatment of paths is actually less of a special case,
and the code that reaches into /etc/installurl is no longer used
at all in that case.

problem noticed by mlarkin@

introduce new url "installpath" to mean "the contents of installurl"
use it to simplify logic.

have ./ always be silent, wherever it may come from.

zap NOINSTALLPATH since it's more intuitive to just define PKG_PATH=
(nothing) now.

tweak installurl code to return a single string that gets parsed
so that we don't have to write the same thing twice later.

scrape most of the config framework
okay aja@, rpe@

add TRUSTED_PKG_PATH after discussing with aja@
use case is testing binary packages among developers, -Dunsigned means
everything is unsigned, which is bad for dependencies

redo sthen's idea deeper in pkg_add, so that *every* url can be abbreviated
the same way (e.g., PKG_PATH=ftp.openbsd.org will work)

okay sthen@

Allow pkg.conf's installpath to be set to a bare hostname (no /), which
implies http://hostname/pub/OpenBSD/[snapshots-or-version]/packages/[arch].
This could be handled at a different location in the code to support PKG_PATH
but this approach is pretty simple and I have something else to make use of it.
ok espie@ for now.

remove $_ usage that warns under perl 5.18

I had misgivings about that one, I was right. Missed an obvious testcases,
and my crash bunnies ARE a bunch of unreliable nitwits.

thanks kili@

allow installpath to use +=, requested by quite a few people,
but did require testing.

temporary option until I come up with a nicer way: if -DNOINSTALLPATH
don't even look in pkg.conf for the installpath.

zap fatpackages. nobody actually uses them.
accordingly, locations do not need to track architecture.

handle failures from ports tree with more care.
this does avoid infinite loops in case of wrong plists.
necessary since print-plist-with-depends CAN fail...

initial scaffolding for @rcscript: for now, acts like normal files,
but allow absolute pathnames (treat that as an implicit @cwd).

nasty inversion of parameters, so that state did not get propagated correctly.
ended up calling './'->errsay in ArcCheck, which of course doesn't work, so
a die() in the middle of a package install... compounded with file !existing,
so that adjust checksum would fail, and so not even a partial install...

parse pkg.conf and use it for default_path if PKG_PATH is not defined.

fix params so that state is state and not PackageLocator...

pass (and save) state around in Repository-related stuff.

allow say and errsay to work without parameters, as it's ways common.
create verbose_system up in state, because it makes sense without verbose.

whitespace cleanup

extra parameter to UpdateSet: use the factory to create extra object.
don't create updatesets directly, go through AddDelete::State.
Get all Locator/Repo access through state.

allow creation of a unique PackageLocator object, that stores state,
and thus can eventually be used to create repositories, print messages,
etc.

whitespace fixes

bump copyright

zap unused

streamline interface

move signal catching up, so we can clean repos and temp files at once.
Fix main program try/catch: signal message should use standard form so
it gets caught.

kill cache, we don't really need it

fixed version, somehow I forgot that I use parse to do weird things with
PKG_PATH...

temporary revert until I figure this out

move the code that recognizes /var/db/pkg as inst: into Repo::Local

tweak path parsing to handle inst:pkgname and such as well...

move pipe handling into RepositoryList

repositories no longer need match

Say goodbye to local $_
lexical scoping is so much neater (courtesy of perl 5.10)

better standards for inst: paths.

special-case parser so that /var/db/pkg becomes inst: during path_parse.

I am on drugs, I don't know where I found split can take strings as
arguments. Repair...

some minor systematic changes.
- mark all regexps I can with /o if they can be compiled once.
- turn $o->method() into $o->method
- remove unneeded prototypes
- reduce split /re/ into split "string" where possible.

add match_locations/filter_locations methods to search stuff that returns
full pkg locations instead of package names.

don't pass srcpath around, since we don't use it.

kill "available" routine, which is no longer used.

remove some specialness of the Filtering stuff.

kill unused stuff

also create PartialStem objects, and create a generic
PackageLocator->match($o), where $o can be a spec, a stem, a partialstem.

have PackageLocator defer to PackageRepositoryList, so that package
availability uses normal linker rules.

allow filtering while calling the Locator.

kill old PkgSpec interface

use match_ref consistently

also allow Spec objects in Repository matches.

spec match works better with a list

remove dependency on available_stems(), make it go thru PackageLocator

we want to no longer expose PackageLocator::available(), lift interfaces
up there, starting with match_spec.

$o->method() becomes $o->method

replace $p->visit('method') calls with $p->method
now that it works.

update copyright years, standardize licence.
reword the introduction to the stuff originally from FreeBSD to clarify
slightly.

get vfs size snapshotted and restored so that pkg_add -u has a harder
time overflowing the disk.

let stem work at the end of urls.

test sturm@ and bernd@,

okay deraadt@

propagate stuff to be able to explicitly clean-up repositories.
(we can't use a normal destructor, since this will be called for
each fork()ed subprocess).

cut down the Locator code into maintainable chunks.

zap more success messages. Keep pid of distant location pipe, just in case.

detect temporary connection errors, and retry with exponential timeout.

Add some nice interfaces instead of the hint() argument to close().
This allows the cache to finish properly even in some weird cases like
fat packages (calling finish_and_close).

kludge: avoid creating non-empty files.

I don't like the fix, but it works...

In case we are pretending, make sure we skip to the next archive member.
Otherwise, we might miss a truncated archive that's missing its last file,
and PKG_CACHE won't work, since it hits a SIGPIPE too soon.

Nice testcase from sturm@

zap 229 (frantisek holop)

add a hint to close that says how the fd got closed (real error, premature,
normal).

This will get used by parse_problems() to figure out what to report exactly,
e.g., short writes from prematurely closed fd are not real errors...
stuff that can't connect IPv6 and retries as v4 is not either, unless
there's a genuine `can't connect error'.

more non-errors from ftp.

I haven't received any feedback at all about this patch, so everyone
gets to test it.

Display an error message if no packages are available in the PKG_PATH.
Quote nlist command, if we invoke it accidentally in a directory with *.tgz
present.

Allows FETCH_CMD to replace ftp.
Note that, for simplicity, FETCH_CMD must be a single command with
a full path. Use a script if you must.

fix PackageRepository::Installed().
let grabPlist finish closing what it opened.

make package opening more efficient (and maybe more intuitive) by
delaying reading the full info directory to a temporary directory.

When we open a package, we just scan for the CONTENTS. If it's the
first file, we stash it into memory and STOP there, delaying grabbing
the rest of the special files to the first call to info().

Make sure the next() proxy shows only real file, by grabbing info files
if necessary.

Introduce handle->plist() method that can retrieve the plist from either
the version stored in memory, or in a file in the temp directory.

allows grabbing the packing-list directly from a PackageLocation,
in order to make further code more symetric and efficient.

beginning to a source path repository infrastructure, very likely to change
a lot yet.

minor nit

more clean-up:
- kill multiple inheritance;
- replace pipename() with open_pipe, and use explicit exec.
- add intermediate PackageRepository::Distant class.

- if PKG_CACHE is set, use an intermediate wedge that stores the
retrieved package in that directory.

- handle signals correctly in that case.

oops

move .tgz addition down to PackageRepositoryList.
re-add list, which returns a \@, whereas available returns a @.

filter names out first, as it's slightly faster

more Package/FatPackage common code: separate the proxy 'next' into two
parts: the one that gets (and renames) files from the archive (getNext)
that depends on Package/FatPackage, and the one that handles opening/
reopening/caching archives (true next);

typo, fix fat packages reading.

grabPlist shall behave like find.

Repository::Installed class, to provide uniform access to installed
packages compared to other `repositories'.

yet more uniform names.

don't actively avoid the Composite Pattern.
PackageRepositoryList can handle very close to PackageRepository

new class: PackageRepositoryList, of which pkgpath is one instance.
new method: wipe(), that removes the info directory, redirected to
the PackageRepository object which `knows' how to wipe stuff out.

(this allows us to eliminate special cases for installed/not installed
packages in the near future).

make grabPlist slightly smarter so that we can still open the package
afterwards.

allows grabbing a packing-list directly, without needing to go through
a temporary file, by using perl's new-fangled
open my $fh, '<', \$scalar

some long-delayed clean-up of this code.
- rename PackageLocation to PackageRepository.
- new class `PackageLocation' which symbolizes an actually opened package.
- rename quite a few things and reorg stuff to make lots more sense.
- put fat packages into a derived FatPackageLocation class.
- fix a bug that made sure fat packages could not be opened on stdin.

ways cleaner and easier to extend...

* New -i interactive option, that can ask obnoxious questions.
* move the signature code to a packing-list method, prepend the pkgname.
* depend on distant listings, cache these systematically.
* cut the PKG_PATH at :/ systematically, much simpler than parsing URLs.
People should get used to trailing /s.
* let pkg_add -u actually run the update.
* check for signature in more places, for instance, let pkg_add installed_stuff
be a void operation.
* grab enough stuff during updates to check signatures, so that pkg_add -u
can report updates that are not needed right away, instead of waiting
for the more confusing `trying to update'
* beginning of @module support.
* @pkgpath support, including updates.
* systematically expand stems when applicable.
* use visitors systematically in pkg_info.
* pkg_info -S reports signature.
* remove hardcoding of categories in PackingList, so that future modules
will be able to expand them.
* add @updateset and @incompatibility keywords.

while scanning a potential package, process issues anyways:
if create() fails, we want to remove any stub file (and report the
problem), so that tools won't get confused in believing the packing-list
got created correctly.

remove tempdir right away if it turns out we're not dealing with a
package at all.

don't mix nlist with ls

support code for pkg_add -u:
- distant_available(), using the distant listing functions.
- stem compiler, to match lots of stems

make the code that enforces the connection limit a separate function, so
that the distant listing code can use it too. Since the distant listing code
creates transient connections, it does not actually need to be tracked,
just to ensure that it has a slot !

fix a few minor details: _list() needs to be called as a method.
use nlist *.tgz instead of ls to gain speed. http does not need
any command, it just retrieves the directory index.

(code not used yet, but so blatantly incorrect)

Handle hard links correctly: the names need to be aliased correctly for
pkg_merging, and the names need to lose the prefix path when getting
extracted.

select fat package based on name too.

handle fat packages based on architecture.

block-scoped require is way simpler than weird AUTOLOAD hack.

unified headers, switch to smaller copyright notice.

repair pkg_add -, by making sure `-' is a special object.

A few useful features:
- pkg_add -A arch, to make believe we are extracting on a given arch
(matches pkg_create).
- pkg_add -P cdrom/-P ftp, to enforce extracting only cdrom'able packages
or ftp'able packages. Helps a lot for checking cdrom lists.
- dependency look-up will look through local directory listings to solve
dependencies, so that non-default dependencies will work more seemlessly.
(doing the same thing for distant repository will happen after 3.5, but
this requires more apparatus: caching package lists, and killing/restoring
existing connections, plus more error handling).

okay pval

some code refactoring.
minor optimization: don't try to open pipe to local file that doesn't
exist anyways.

track connections per-host for ftp/http.
Don't allow more than maxcount() connections per-host (for now, one
connection).

tests by sturm@, fries@

Clean up PackageLocator so it looks somewhat more object-oriented.
Most importantly, put all the state information into the created object,
so that the actual archive can be closed, later reopened, and scanned
until the correct file is found.

This will be used to allow retrieving packages through ftp without keeping
loads of connections opened because of dependency resolving.

Approved by fries and naddy.

Handle pkg_add -
let pkg_add postpone conflict handling until it has the real name of
the package if necessary, and let PackageLocator deal correctly with a '-'
filename.

put pkgpath back in correct order.
automatically append package location specified on the command
line to pkgpath (reported by numerous people).

okay sturm@

Allows close() on already closed locator.
Fix second problem seen by krw@

pkgpath is . by default.

branches: 1.1.1;
Initial revision

Refactor locator code so it's a proper object.
allow fw_update to short-circuit entirely, thus the specific
treatment of paths is actually less of a special case,
and the code that reaches into /etc/installurl is no longer used
at all in that case.

problem noticed by mlarkin@

introduce new url "installpath" to mean "the contents of installurl"
use it to simplify logic.

have ./ always be silent, wherever it may come from.

zap NOINSTALLPATH since it's more intuitive to just define PKG_PATH=
(nothing) now.

tweak installurl code to return a single string that gets parsed
so that we don't have to write the same thing twice later.

scrape most of the config framework
okay aja@, rpe@

add TRUSTED_PKG_PATH after discussing with aja@
use case is testing binary packages among developers, -Dunsigned means
everything is unsigned, which is bad for dependencies

redo sthen's idea deeper in pkg_add, so that *every* url can be abbreviated
the same way (e.g., PKG_PATH=ftp.openbsd.org will work)

okay sthen@

Allow pkg.conf's installpath to be set to a bare hostname (no /), which
implies http://hostname/pub/OpenBSD/[snapshots-or-version]/packages/[arch].
This could be handled at a different location in the code to support PKG_PATH
but this approach is pretty simple and I have something else to make use of it.
ok espie@ for now.

remove $_ usage that warns under perl 5.18

I had misgivings about that one, I was right. Missed an obvious testcases,
and my crash bunnies ARE a bunch of unreliable nitwits.

thanks kili@

allow installpath to use +=, requested by quite a few people,
but did require testing.

temporary option until I come up with a nicer way: if -DNOINSTALLPATH
don't even look in pkg.conf for the installpath.

zap fatpackages. nobody actually uses them.
accordingly, locations do not need to track architecture.

handle failures from ports tree with more care.
this does avoid infinite loops in case of wrong plists.
necessary since print-plist-with-depends CAN fail...

initial scaffolding for @rcscript: for now, acts like normal files,
but allow absolute pathnames (treat that as an implicit @cwd).

nasty inversion of parameters, so that state did not get propagated correctly.
ended up calling './'->errsay in ArcCheck, which of course doesn't work, so
a die() in the middle of a package install... compounded with file !existing,
so that adjust checksum would fail, and so not even a partial install...

parse pkg.conf and use it for default_path if PKG_PATH is not defined.

fix params so that state is state and not PackageLocator...

pass (and save) state around in Repository-related stuff.

allow say and errsay to work without parameters, as it's ways common.
create verbose_system up in state, because it makes sense without verbose.

whitespace cleanup

extra parameter to UpdateSet: use the factory to create extra object.
don't create updatesets directly, go through AddDelete::State.
Get all Locator/Repo access through state.

allow creation of a unique PackageLocator object, that stores state,
and thus can eventually be used to create repositories, print messages,
etc.

whitespace fixes

bump copyright

zap unused

streamline interface

move signal catching up, so we can clean repos and temp files at once.
Fix main program try/catch: signal message should use standard form so
it gets caught.

kill cache, we don't really need it

fixed version, somehow I forgot that I use parse to do weird things with
PKG_PATH...

temporary revert until I figure this out

move the code that recognizes /var/db/pkg as inst: into Repo::Local

tweak path parsing to handle inst:pkgname and such as well...

move pipe handling into RepositoryList

repositories no longer need match

Say goodbye to local $_
lexical scoping is so much neater (courtesy of perl 5.10)

better standards for inst: paths.

special-case parser so that /var/db/pkg becomes inst: during path_parse.

I am on drugs, I don't know where I found split can take strings as
arguments. Repair...

some minor systematic changes.
- mark all regexps I can with /o if they can be compiled once.
- turn $o->method() into $o->method
- remove unneeded prototypes
- reduce split /re/ into split "string" where possible.

add match_locations/filter_locations methods to search stuff that returns
full pkg locations instead of package names.

don't pass srcpath around, since we don't use it.

kill "available" routine, which is no longer used.

remove some specialness of the Filtering stuff.

kill unused stuff

also create PartialStem objects, and create a generic
PackageLocator->match($o), where $o can be a spec, a stem, a partialstem.

have PackageLocator defer to PackageRepositoryList, so that package
availability uses normal linker rules.

allow filtering while calling the Locator.

kill old PkgSpec interface

use match_ref consistently

also allow Spec objects in Repository matches.

spec match works better with a list

remove dependency on available_stems(), make it go thru PackageLocator

we want to no longer expose PackageLocator::available(), lift interfaces
up there, starting with match_spec.

$o->method() becomes $o->method

replace $p->visit('method') calls with $p->method
now that it works.

update copyright years, standardize licence.
reword the introduction to the stuff originally from FreeBSD to clarify
slightly.

get vfs size snapshotted and restored so that pkg_add -u has a harder
time overflowing the disk.

let stem work at the end of urls.

test sturm@ and bernd@,

okay deraadt@

propagate stuff to be able to explicitly clean-up repositories.
(we can't use a normal destructor, since this will be called for
each fork()ed subprocess).

cut down the Locator code into maintainable chunks.

zap more success messages. Keep pid of distant location pipe, just in case.

detect temporary connection errors, and retry with exponential timeout.

Add some nice interfaces instead of the hint() argument to close().
This allows the cache to finish properly even in some weird cases like
fat packages (calling finish_and_close).

kludge: avoid creating non-empty files.

I don't like the fix, but it works...

In case we are pretending, make sure we skip to the next archive member.
Otherwise, we might miss a truncated archive that's missing its last file,
and PKG_CACHE won't work, since it hits a SIGPIPE too soon.

Nice testcase from sturm@

zap 229 (frantisek holop)

add a hint to close that says how the fd got closed (real error, premature,
normal).

This will get used by parse_problems() to figure out what to report exactly,
e.g., short writes from prematurely closed fd are not real errors...
stuff that can't connect IPv6 and retries as v4 is not either, unless
there's a genuine `can't connect error'.

more non-errors from ftp.

I haven't received any feedback at all about this patch, so everyone
gets to test it.

Display an error message if no packages are available in the PKG_PATH.
Quote nlist command, if we invoke it accidentally in a directory with *.tgz
present.

Allows FETCH_CMD to replace ftp.
Note that, for simplicity, FETCH_CMD must be a single command with
a full path. Use a script if you must.

fix PackageRepository::Installed().
let grabPlist finish closing what it opened.

make package opening more efficient (and maybe more intuitive) by
delaying reading the full info directory to a temporary directory.

When we open a package, we just scan for the CONTENTS. If it's the
first file, we stash it into memory and STOP there, delaying grabbing
the rest of the special files to the first call to info().

Make sure the next() proxy shows only real file, by grabbing info files
if necessary.

Introduce handle->plist() method that can retrieve the plist from either
the version stored in memory, or in a file in the temp directory.

allows grabbing the packing-list directly from a PackageLocation,
in order to make further code more symetric and efficient.

beginning to a source path repository infrastructure, very likely to change
a lot yet.

minor nit

more clean-up:
- kill multiple inheritance;
- replace pipename() with open_pipe, and use explicit exec.
- add intermediate PackageRepository::Distant class.

- if PKG_CACHE is set, use an intermediate wedge that stores the
retrieved package in that directory.

- handle signals correctly in that case.

oops

move .tgz addition down to PackageRepositoryList.
re-add list, which returns a \@, whereas available returns a @.

filter names out first, as it's slightly faster

more Package/FatPackage common code: separate the proxy 'next' into two
parts: the one that gets (and renames) files from the archive (getNext)
that depends on Package/FatPackage, and the one that handles opening/
reopening/caching archives (true next);

typo, fix fat packages reading.

grabPlist shall behave like find.

Repository::Installed class, to provide uniform access to installed
packages compared to other `repositories'.

yet more uniform names.

don't actively avoid the Composite Pattern.
PackageRepositoryList can handle very close to PackageRepository

new class: PackageRepositoryList, of which pkgpath is one instance.
new method: wipe(), that removes the info directory, redirected to
the PackageRepository object which `knows' how to wipe stuff out.

(this allows us to eliminate special cases for installed/not installed
packages in the near future).

make grabPlist slightly smarter so that we can still open the package
afterwards.

allows grabbing a packing-list directly, without needing to go through
a temporary file, by using perl's new-fangled
open my $fh, '<', \$scalar

some long-delayed clean-up of this code.
- rename PackageLocation to PackageRepository.
- new class `PackageLocation' which symbolizes an actually opened package.
- rename quite a few things and reorg stuff to make lots more sense.
- put fat packages into a derived FatPackageLocation class.
- fix a bug that made sure fat packages could not be opened on stdin.

ways cleaner and easier to extend...

* New -i interactive option, that can ask obnoxious questions.
* move the signature code to a packing-list method, prepend the pkgname.
* depend on distant listings, cache these systematically.
* cut the PKG_PATH at :/ systematically, much simpler than parsing URLs.
People should get used to trailing /s.
* let pkg_add -u actually run the update.
* check for signature in more places, for instance, let pkg_add installed_stuff
be a void operation.
* grab enough stuff during updates to check signatures, so that pkg_add -u
can report updates that are not needed right away, instead of waiting
for the more confusing `trying to update'
* beginning of @module support.
* @pkgpath support, including updates.
* systematically expand stems when applicable.
* use visitors systematically in pkg_info.
* pkg_info -S reports signature.
* remove hardcoding of categories in PackingList, so that future modules
will be able to expand them.
* add @updateset and @incompatibility keywords.

while scanning a potential package, process issues anyways:
if create() fails, we want to remove any stub file (and report the
problem), so that tools won't get confused in believing the packing-list
got created correctly.

remove tempdir right away if it turns out we're not dealing with a
package at all.

don't mix nlist with ls

support code for pkg_add -u:
- distant_available(), using the distant listing functions.
- stem compiler, to match lots of stems

make the code that enforces the connection limit a separate function, so
that the distant listing code can use it too. Since the distant listing code
creates transient connections, it does not actually need to be tracked,
just to ensure that it has a slot !

fix a few minor details: _list() needs to be called as a method.
use nlist *.tgz instead of ls to gain speed. http does not need
any command, it just retrieves the directory index.

(code not used yet, but so blatantly incorrect)

Handle hard links correctly: the names need to be aliased correctly for
pkg_merging, and the names need to lose the prefix path when getting
extracted.

select fat package based on name too.

handle fat packages based on architecture.

block-scoped require is way simpler than weird AUTOLOAD hack.

unified headers, switch to smaller copyright notice.

repair pkg_add -, by making sure `-' is a special object.

A few useful features:
- pkg_add -A arch, to make believe we are extracting on a given arch
(matches pkg_create).
- pkg_add -P cdrom/-P ftp, to enforce extracting only cdrom'able packages
or ftp'able packages. Helps a lot for checking cdrom lists.
- dependency look-up will look through local directory listings to solve
dependencies, so that non-default dependencies will work more seemlessly.
(doing the same thing for distant repository will happen after 3.5, but
this requires more apparatus: caching package lists, and killing/restoring
existing connections, plus more error handling).

okay pval

some code refactoring.
minor optimization: don't try to open pipe to local file that doesn't
exist anyways.

track connections per-host for ftp/http.
Don't allow more than maxcount() connections per-host (for now, one
connection).

tests by sturm@, fries@

Clean up PackageLocator so it looks somewhat more object-oriented.
Most importantly, put all the state information into the created object,
so that the actual archive can be closed, later reopened, and scanned
until the correct file is found.

This will be used to allow retrieving packages through ftp without keeping
loads of connections opened because of dependency resolving.

Approved by fries and naddy.

Handle pkg_add -
let pkg_add postpone conflict handling until it has the real name of
the package if necessary, and let PackageLocator deal correctly with a '-'
filename.

put pkgpath back in correct order.
automatically append package location specified on the command
line to pkgpath (reported by numerous people).

okay sturm@

Allows close() on already closed locator.
Fix second problem seen by krw@

pkgpath is . by default.

branches: 1.1.1;
Initial revision