Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r264481:
Add dwarf annotations to the amd64 _rtld_bind_start to allow debuggers
to unwind around the calls from PLT to binder.

MFC r234840:
Split the symlook_obj1 into a loop iterating over the ELF object symbol
hash elements, and a helper matched_symbol() which match the given hash
entry and request, performing needed type and version checks.

MFC r234841:
Add GNU hash support for rtld.

MFC r235054:
Work around a situation where symlook_obj() could be called for the
object for which digest_dynamic1() was not done yet. Just return
EINVAL and do not try to dereference NULL buckets hash array.

MFC r233231:
Fix several problems with our ELF filters implementation.

Do not relocate twice an object which happens to be needed by loaded
binary (or dso) and some filtee opened due to symbol resolution when
relocating need objects. Record the state of the relocation
processing in Obj_Entry and short-circuit relocate_objects() if
current object already processed.

Do not call constructors for filtees loaded during the early
relocation processing before image is initialized enough to run
user-provided code. Filtees are loaded using dlopen_object(), which
normally performs relocation and initialization. If filtee is
lazy-loaded during the relocation of dso needed by the main object,
dlopen_object() runs too earlier, when most runtime services are not
yet ready.

Postpone the constructors call to the time when main binary and
depended libraries constructors are run, passing the new flag
RTLD_LO_EARLY to dlopen_object(). Symbol lookups callers inform
symlook_* functions about early stage of initialization with
SYMLOOK_EARLY. Pass flags through all functions participating in
object relocation.

Use the opportunity and fix flags argument to find_symdef() in
arch-specific reloc.c to use proper name SYMLOOK_IN_PLT instead of
true, which happen to have the same numeric value.

MFC r233777 (by kan):
Do not try to adjust stacks if dlopen_object is called too early.

MFC r233778 (by kan):
Remove extra blank line from revious commit.

MFC note: the ARM and MIPS TLS support is not merged back, so the chunks
from r233231 which fix misuse of flags in calls to find_symdef() in
the corresponding relocation type handlers were not applied. When TLS
support is merged, the rest of r233231 should be applied too.

MFC r232831:
Add support for preinit, init and fini arrays to rtld.
Only binaries marked with proper ABI note gets array ctr/dtrs called.

MFC r232856:
When iterating over the dso program headers, the object is not initialized
yet, and object segments are not yet mapped. Only parse the notes that
appear in the first page of the dso (as it should be anyway), and use
the preloaded page content.

MFC r232857 (by dim):
Fix a warning/error with clang.

MFC r232859 (by dim):
Amend r232857, now dropping the casts entirely, as they were not
necessary at all.

MFC r228435:
Add support for STT_GNU_IFUNC and R_MACHINE_IRELATIVE GNU extensions to
rtld on 386 and amd64.

MFC r228503:
Postpone the resolution for irelative/ifunc right before initializers
are called, and drop bind lock around calls to dispatcher. Use
initlist to iterate over the objects instead of the ->next, due to
drop of the bind lock in iteration.

For i386/reloc.c:reloc_iresolve(), fix calculation of the dispatch
function address for dso, by taking into account possible non-zero
relocbase.

MFC r228635 (by nwhitehorn):
Fix RTLD on PowerPC after r228435. Changing the order of init_pltgot()
caused the icache to be invalidated at the wrong time, resulting in
an icache full of nonsense in the PLT section.

MFC r227458, r226436:

- change "is is" to "is" or "it is"
- change "the the" to "the"
- other typo fixes

Approved by: lstewart

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

When loading dso without PT_GNU_STACK phdr, only call
__pthread_map_stacks_exec() on architectures that allow executable
stacks.

Reported and tested by: marcel (ia64)

Add section .note.GNU-stack for assembly files used by 386 and amd64.

Sort -mno-(mmx|3dnow|sse|sse2|sse3) options consistently throughout the
tree.

Submitted by: arundel

On amd64 and i386, tell the compiler to refrain from generating SSE,
3DNow, MMX and floating point instructions in rtld-elf.

Otherwise, _rtld_bind() (and whatever it calls) could possibly clobber
function arguments that are passed in SSE/3DNow/MMX/FP registers,
usually floating point values. This can happen, for example, when clang
generates SSE code for memset() or memcpy() calls.

One symptom of this is sshd dying early on amd64 with "PRNG not seeded",
which is ultimately caused by libcrypto.so.6 calling RAND_add() with a
double parameter. That parameter is passed via %xmm0, which gets wiped
out by an SSE memset() in _rtld_bind().

Reviewed by: kib, kan

Remove '-elf' from build flags for libexec/rtld-elf for amd64 and i386.
ELF has been the default format for almost 12 years now.

Implement support for ELF filters in rtld. Both normal and auxillary
filters are implemented.

Filtees are loaded on demand, unless LD_LOADFLTR environment variable
is set or -z loadfltr was specified during the linking. This forces
rtld to upgrade read-locked rtld_bind_lock to write lock when it
encounters an object with filter during symbol lookup.

Consolidate common arguments of the symbol lookup functions in the
SymLook structure. Track the state of the rtld locks in the
RtldLockState structure. Pass local RtldLockState through the rtld
symbol lookup calls to allow lock upgrades.

Reviewed by: kan
Tested by: Mykola Dzham <i levsha me>, nwhitehorn (powerpc)

MFtbemd:

Prefer MACHNE_CPUARCH to MACHINE_ARCH in most contexts where you want
to test of all the CPUs of a given family conform.

Only use the cache after the early stage of loading. This is
because calling mmap() etc. may use GOT which is not set up
yet. Use calloc() instead of mmap() in cases where this
was the case before (sparc64, powerpc, arm).

Submitted by: Dimitry Andric (dimitry andric com)
Reviewed by: kan
Approved by: ed (mentor)

Now that the kernel defines CACHE_LINE_SIZE in machine/param.h, use
that definition in the custom locking code for the run-time linker
rather than local definitions.

Pointed out by: tinderbox
MFC after: 2 weeks

*thwack*! all the world's not i386.

Pointy hat to: des

Allocate space for thread pointer, this allows thread library to access
its pointer from begin, and simplifies _get_curthread() in libthr.

Implement ELF symbol versioning using GNU semantics. This code aims
to be compatible with symbol versioning support as implemented by
GNU libc and documented by http://people.redhat.com/~drepper/symbol-versioning
and LSB 3.0.

Implement dlvsym() function to allow lookups for a specific version of
a given symbol.

Explicitly cast ELF_R_TYPE() to the right type.

Remove these unused files before any other archs include the same bogus
file.

Add support for Thread Local Storage.

More stack alignment fixes. Arrange so we call _rtld() in ld-elf.so.1
with the correct alignment. This is important because this calls to
library static constructors are made from here. The bug in the old crt*.s
files hid this because in this case, two wrongs do indeed make a right.
Also, call _rtld_bind() with the correct alignment, because it calls back
into the pthread library locking functions. If things happen just
the wrong way, we get a SIG10 due to the broken stack alignment.

Fix dynamic linking a bit more.. enough that mozilla-firebird works if you
dig up the patches for amd64 support for it.

Note to self: do not put a 64 bit value in a 32 bit space.

Revert last change. ../rtld.c uses CACHE_LINE_SIZE too.
Change it to 64 while here.

Reported by: ps

Only define CACHE_LINE_SIZE in one place..

CACHE_LINE_SIZE is 64 on athlon and amd64 chips, not 32. This should
probably be 128 since that is what the hardware prefetch fill size is
on both the p3, p4 and athlon* cpus.

Allow threading libraries to register their own locking
implementation in case default one provided by rtld is
not suitable.

Consolidate various identical MD lock implementation into
a single file using appropriate machine/atomic.h.

Approved by: re (scottl)

Initial pass at supporting shared libraries on amd64. There are still
a few missing relocation types in amd64/reloc.c, but I have not found
any of them in use yet. :-)

Approved by: re (amd64/* blanket)

Remove 80386 bandaids from code repocopied from i386. rtld_start.S still
todo.

No need to zero fill memory, mmapped anonymously. Kernel will
return pre-zeroed pages itself.

Noticed by: jake

Fix the handling of high PLT entries (> 32764) on sparc64. This requires
additional arguments to reloc_jmpslot(), which is why MI code and MD code
of other platforms had to be changed.

Reviewed by: jake
Approved by: re

Remove the nanosleep calls from the spin loops in the locking code.
They provided little benefit (if any) and they caused some problems
in OpenOffice, at least in post-KSE -current and perhaps in other
environments too. The nanosleep calls prevented the profiling timer
from advancing during the spinloops, thereby preventing the thread
scheduler from ever pre-empting the spinning thread. Alexander
Kabaev diagnosed this problem, Martin Blapp helped with testing,
and Matt Dillon provided some helpful suggestions.

This is a short-term fix for a larger problem. The use of spinlocking
isn't guaranteed to work in all cases. For example, if the spinning
thread has higher priority than all other threads, it may never be
pre-empted, and the thread holding the lock may never progress far
enough to release the lock. On the other hand, spinlocking is the
only locking that can work with an arbitrary unknown threads package.

I have some ideas for a much better fix in the longer term. It
would eliminate all locking inside the dynamic linker by making it
safe for symbol lookups and lazy binding to proceed in parallel
with a call to dlopen or dlclose. This means that the only mutual
exclusion needed would be to prevent multiple simultaneous calls
to dlopen and/or dlclose. That mutual exclusion could be put into
the native pthreads library. Applications using foreign threads
packages would have to make their own arrangements to ensure that
they did not have multiple threads in dlopen and/or dlclose -- a
reasonable requirement in my opinion.

MFC after: 3 days

Update the asm statements to use the "+" modifier instead of
matching constraints where appropriate. This makes the dynamic
linker buildable at -O0 again.

Thanks to Bruce Evans for identifying the cause of the build
problem.

MFC after: 1 week

Correct a bug in the last commit. The whole point of creating a 'done:'
goto target was so the cache could be freed. So free the cache after
done: rather then before done: (!)

Submitted by: Gavin Atkinson <gavin@ury.york.ac.uk>

In tracking down an installation seg fault with then openoffice port
Martin Blapp determined that the elf dynamic loader was at fault. In
particular, the loader uses alloca() to allocate a symbol cache on the
stack. Normally this would work just fine, but if the loader is called
from a threaded program and the object being loaded is fairly large the
alloca() can blow away the thread stack and effect other nearby thread
stacks as well. My testing showed that the symbol cache can be as large
as 250KBytes during the openoffice port build and install sequence. Martin
was able to work around the problem by disabling the symbol cache
(cache = NULL;). However, this solution is not adequate for commit because
it can cause an enormous cpu burden for applications which do a lot of
dynamic loading (e.g. like konqueror).

The solution is to use anonymous mmap() to temporarily allocate space to
hold the symbol cache. In testing I found that replacing the alloca()
with mmap() has no observable degredation in performance.

It should be noted that this bug does not necessarily cause an immediate
crash but can instead result in long term corruption and instability in
applications that load modules from threads. The bug is almost certainly
responsible for some of the instabilities found in konqueror, for example,
and possibly netscape too.

Sleuthing work by: Martin Blapp <mb@imp.ch>
X-MFC after: Before or after the 4.6 release depending on the release engineers

Update rtld for the "new" ia64 ABI. In the old toolchain, the
DT_INIT and DT_FINI tags pointed to fptr records. In 2.11.2, it points
to the actuall address of the function. On IA64 you cannot just take
an address of a function, store it in a function pointer variable and
call it.. the function pointers point to a fptr data block that has the
target gp and address in it. This is absolutely necessary for using
the in-tree binutils toolchain, but (unfortunately) will not work with
old shared libraries. Save your old ld-elf.so.1 if you want to use
old ones still. Do not mix-and-match.

This is a no-op change for i386 and alpha.

Reviewed by: dfr

Add ia64 support. Various adjustments were made to existing targets to
cope with a few interface changes required by the ia64. In particular,
function pointers on ia64 need special treatment in rtld.

Performance improvements for the ELF dynamic linker. These
particularly help programs which load many shared libraries with
a lot of relocations. Large C++ programs such as are found in KDE
are a prime example.

While relocating a shared object, maintain a vector of symbols
which have already been looked up, directly indexed by symbol
number. Typically, symbols which are referenced by a relocation
entry are referenced by many of them. This is the same optimization
I made to the a.out dynamic linker in 1995 (rtld.c revision 1.30).

Also, compare the first character of a sought-after symbol with its
symbol table entry before calling strcmp().

On a PII/400 these changes reduce the start-up time of a typical
KDE program from 833 msec (elapsed) to 370 msec.

MFC after: 5 days

Fix a bug which could cause programs with user threads packages to
lock against themselves, causing infinite spinning. Brian Feldman
found this problem when testing with Mozilla and supplied the fix,
which I have revised slightly.

Here is the failure scenario. A thread calls dlopen() and acquires
the writer lock. While the thread still holds the lock, a signal
is delivered and caught. The signal handler tries to call a function
which hasn't been bound yet. It thus enters the dynamic linker
and tries to acquire the reader lock. Since the writer lock is
already held, it will spin forever in the signal handler. The
thread holding the lock won't be able to progress and release the
lock.

The solution is to block almost all signals while holding the
exclusive lock.

A similar problem could conceivably occur in the opposite order.
Namely, a thread is holding the reader lock and then a signal
handler calls dlopen() or dlclose() and spins waiting for the writer
lock. We deal with this administratively by proclaiming that signal
handlers aren't allowed to call dlopen() or dlclose(). Actually
we don't have to proclaim a thing, since signal handlers aren't
allowed to call any system functions except those which are explicitly
permitted.

Submitted by: Brian Fundakowski Feldman <green>

Solve the dynamic linker's problems with multithreaded programs once
and for all (I hope). Packages such as wine, JDK, and linuxthreads
should no longer have any problems with re-entering the dynamic
linker.

This commit replaces the locking used in the dynamic linker with a
new spinlock-based reader/writer lock implementation. Brian
Fundakowski Feldman <green> argued for this from the very beginning,
but it took me a long time to come around to his point of view.
Spinlocks are the only kinds of locks that work with all thread
packages. But on uniprocessor systems they can be inefficient,
because while a contender for the lock is spinning the holder of the
lock cannot make any progress toward releasing it. To alleviate
this disadvantage I have borrowed a trick from Sleepycat's Berkeley
DB implementation. When spinning for a lock, the requester does a
nanosleep() call for 1 usec. each time around the loop. This will
generally yield the CPU to other threads, allowing the lock holder
to finish its business and release the lock. I chose 1 usec. as the
minimum sleep which would with reasonable certainty not be rounded
down to 0.

The formerly machine-independent file "lockdflt.c" has been moved
into the architecture-specific subdirectories by repository copy.
It now contains the machine-dependent spinlocking code. For the
spinlocks I used the very nifty "simple, non-scalable reader-preference
lock" which I found at

<http://www.cs.rochester.edu/u/scott/synchronization/pseudocode/rw.html>

on all CPUs except the 80386 (the specific CPU model, not the
architecture). The 80386 CPU doesn't support the necessary "cmpxchg"
instruction, so on that CPU a simple exclusive test-and-set lock
is used instead. 80386 CPUs are detected at initialization time by
trying to execute "cmpxchg" and catching the resulting SIGILL
signal.

To reduce contention for the locks, I have revamped a couple of
key data structures, permitting all common operations to be done
under non-exclusive (reader) locking. The only operations that
require exclusive locking now are the rare intrusive operations
such as dlopen() and dlclose().

The dllockinit() interface is now deprecated. It still exists,
but only as a do-nothing stub. I plan to remove it as soon as is
reasonably possible. (From the very beginning it was clearly
labeled as experimental and subject to change.) As far as I know,
only the linuxthreads port uses dllockinit(). This interface turned
out to have several problems. As one example, when the dynamic
linker called a client-supplied locking function, that function
sometimes needed lazy binding, causing re-entry into the dynamic
linker and a big looping mess. And in any case, it turned out to be
too burdensome to require threads packages to register themselves
with the dynamic linker.

When a threads package registers locking methods with dllockinit(),
figure out which shared object(s) contain the the locking methods
and fully bind those objects as if they had been loaded with
LD_BIND_NOW=1. The goal is to keep the locking methods from
requiring any lazy binding. Otherwise infinite recursion occurs
in _rtld_bind.

This fixes the infinite recursion problem in the linuxthreads port.

Block almost all signals in the default locking method instead of
just a few of them. This looks like it solves the recent

ld-elf.so.1: assert failed: /usr/src/libexec/rtld-elf/lockdflt.c:55

failures seen by some applications such as JDK.

Revamp the mechanism for enumerating and calling shared objects'
init and fini functions. Now the code is very careful to hold no
locks when calling these functions. Thus the dynamic linker cannot
be re-entered with a lock already held.

Remove the tolerance for recursive locking that I added in revision
1.2 of dllockinit.c. Recursive locking shouldn't happen any more.

Mozilla and JDK users: I'd appreciate confirmation that things still
work right (or at least the same) with these changes.

Work around an assert failure in the dynamic linker's default thread
locking functions. If an application loads a shared object with
dlopen() and the shared object has an init function which requires
lazy binding, then _rtld_bind is called when the thread is already
inside the dynamic linker. This leads to a recursive acquisition
of the lock, which I was not expecting -- hence the assert failure.

This work-around makes the default locking functions handle recursive
locking. It is NOT the correct fix -- that should be implemented
at the generic locking level rather than in the default locking
functions. I will implement the correct fix in a future commit.

Since the dllockinit() interface will likely need to change, warn
about that in both the man page and the header file.

Add a new function dllockinit() for registering thread locking
functions to be used by the dynamic linker. This can be called by
threads packages at start-up time. I will add the call to libc_r
soon.

Also add a default locking method that is used up until dllockinit()
is called. The default method works by blocking SIGVTALRM, SIGPROF,
and SIGALRM in critical sections. It is based on the observation
that most user-space threads packages implement thread preemption
with one of these signals (usually SIGVTALRM).

The dynamic linker has never been reentrant, but it became less
reentrant in revision 1.34 of "src/libexec/rtld-elf/rtld.c".
Starting with that revision, multiple threads each doing lazy
binding could interfere with each other. The usual symptom was
that a symbol was falsely reported as undefined at start-up time.
It was rare but not unseen. This commit fixes it.

$Id$ -> $FreeBSD$

Fix a serious performance bug for large programs on the Alpha,
discovered by Hidetoshi Shimokawa. Large programs need multiple
GOTs. The lazy binding stub in the PLT can be reached from any of
these GOTs, but the dynamic linker only has enough information to
fix up the first GOT entry. Thus calls through the other GOTs went
through the time-consuming lazy binding process on every call.

This fix rewrites the PLT entries themselves to bypass the lazy
binding.

Tested by Hidetoshi Shimokawa and Steve Price.

Reviewed by: Doug Rabson <dfr@freebsd.org>

Eliminate all machine-dependent code from the main source body and
the Makefile, and move it down into the architecture-specific
subdirectories.

Eliminate an asm() statement for the i386.

Make the dynamic linker work if it is built as an executable instead
of as a shared library. See i386/Makefile.inc to find out how to
do it. Note, this change is not enabled and it might never be
enabled. But it might be useful in the future. Building the
dynamic linker as an executable should make it start up faster,
because it won't have any relocations. But in practice I suspect
the difference is negligible.

Add alpha support.

Submitted by: John Birrell <jb@cimlogic.com.au> (with extra hacks by me)
Obtained from: Probably NetBSD

Import the ELF dynamic linker. This is the ElfKit version with
quite a few enhancements and bug fixes. There are still some known
deficiencies, but it should be adequate to get us started with ELF.

Submitted by: John Polstra <jdp@polstra.com>