sys/atomic.h: Fix atomic_store_* on sparcv7, sparcv8.

These did not cooperate with the hash-locked scheme of the other
atomic operations, with the effect that, for instance, a typical
naive spin lock based on atomic_*,

volatile unsigned locked = 0;
lock()
{
while (atomic_swap_uint(&locked, 1))
continue;
membar_acquire();
}
unlock()
{
membar_release();
atomic_store_relaxed(&locked, 0);
}

would fail to achieve mutual exclusion.

For this case, we need to use atomic_swap_* (or, for 8- or 16-bit
objects, atomic_cas_32 loops, since there is no atomic_swap_8 or
atomic_swap_16).

The new machine/types.h macro __HAVE_HASHLOCKED_ATOMICS says whether
these contortions are necessary.

Note that this _requires_ the use of atomic_store_*(p, v), not
regular stores *p = v, to work with the r/m/w atomic operations.

ksyms: Use pserialize(9) for kernel access to ksyms.

This makes it available in interrupt context, e.g. for printing
messages with kernel symbol names for return addresses as drm wants
to do.

Revert "ksyms: Use pserialize(9) for kernel access to ksyms."

ksyms: Use pserialize(9) for kernel access to ksyms.

This makes it available in interrupt context, e.g. for printing
messages with kernel symbol names for return addresses as drm wants
to do.

kcsan: fix the copyright notices

Make copystr() a MI C function, part of libkern and shared on all
architectures.

Notes:

- On alpha and ia64 the function is kept but gets renamed locally to avoid
symbol collision. This is because on these two arches, I am not sure
whether the ASM callers do not rely on fixed registers, so I prefer to
keep the ASM body for now.
- On Vax, only the symbol is removed, because the body is used from other
functions.
- On RISC-V, this change fixes a bug: copystr() was just a wrapper around
strlcpy(), but strlcpy() makes the operation less safe (strlen on the
source beyond its size).
- The kASan, kCSan and kMSan wrappers are removed, because now that
copystr() is in C, the compiler transformations are applied to it,
without the need for manual wrappers.

Could test on amd64 only, but should be fine.

Drop the todo and qualify the accesses.

branches: 1.7.2; 1.7.4;
Add a comment.

Add KCSAN instrumentation for atomic_{load,store}_*.

Instrument copyout() in kCSan, for parity with kMSan.

Don't include "opt_kcsan.h" since there's already <sys/csan.h> included.

Exclude the PTE space from KCSAN, since there the same VA can point to
different PAs.

Change kcsan_md_is_avail() to always return true; I was testing with
interrupts disabled as debugging. Change the delay/sample parameters
to have better fluidity.

Add Kernel Concurrency Sanitizer (kCSan) support. This sanitizer allows us
to detect race conditions at runtime. It is a variation of TSan that is
easy to implement and more suited to kernel internals, albeit theoretically
less precise than TSan's happens-before.

We do basically two things:

- On every KCSAN_NACCESSES (=2000) memory accesses, we create a cell
describing the access, and delay the calling CPU (10ms).

- On all memory accesses, we verify if the memory we're reading/writing
is referenced in a cell already.

The combination of the two means that, if for example cpu0 does a read that
is selected and cpu1 does a write at the same address, kCSan will fire,
because cpu1's write collides with cpu0's read cell.

The coverage of the instrumentation is the same as that of kASan. Also, the
code is organized in a way similar to kASan, so it is easy to add support
for more architectures than amd64. kCSan is compatible with KCOV.

Reviewed by Kamil.

ksyms: Use pserialize(9) for kernel access to ksyms.

This makes it available in interrupt context, e.g. for printing
messages with kernel symbol names for return addresses as drm wants
to do.

Revert "ksyms: Use pserialize(9) for kernel access to ksyms."

ksyms: Use pserialize(9) for kernel access to ksyms.

This makes it available in interrupt context, e.g. for printing
messages with kernel symbol names for return addresses as drm wants
to do.

kcsan: fix the copyright notices

Make copystr() a MI C function, part of libkern and shared on all
architectures.

Notes:

- On alpha and ia64 the function is kept but gets renamed locally to avoid
symbol collision. This is because on these two arches, I am not sure
whether the ASM callers do not rely on fixed registers, so I prefer to
keep the ASM body for now.
- On Vax, only the symbol is removed, because the body is used from other
functions.
- On RISC-V, this change fixes a bug: copystr() was just a wrapper around
strlcpy(), but strlcpy() makes the operation less safe (strlen on the
source beyond its size).
- The kASan, kCSan and kMSan wrappers are removed, because now that
copystr() is in C, the compiler transformations are applied to it,
without the need for manual wrappers.

Could test on amd64 only, but should be fine.

Drop the todo and qualify the accesses.

branches: 1.7.2; 1.7.4;
Add a comment.

Add KCSAN instrumentation for atomic_{load,store}_*.

Instrument copyout() in kCSan, for parity with kMSan.

Don't include "opt_kcsan.h" since there's already <sys/csan.h> included.

Exclude the PTE space from KCSAN, since there the same VA can point to
different PAs.

Change kcsan_md_is_avail() to always return true; I was testing with
interrupts disabled as debugging. Change the delay/sample parameters
to have better fluidity.

Add Kernel Concurrency Sanitizer (kCSan) support. This sanitizer allows us
to detect race conditions at runtime. It is a variation of TSan that is
easy to implement and more suited to kernel internals, albeit theoretically
less precise than TSan's happens-before.

We do basically two things:

- On every KCSAN_NACCESSES (=2000) memory accesses, we create a cell
describing the access, and delay the calling CPU (10ms).

- On all memory accesses, we verify if the memory we're reading/writing
is referenced in a cell already.

The combination of the two means that, if for example cpu0 does a read that
is selected and cpu1 does a write at the same address, kCSan will fire,
because cpu1's write collides with cpu0's read cell.

The coverage of the instrumentation is the same as that of kASan. Also, the
code is organized in a way similar to kASan, so it is easy to add support
for more architectures than amd64. kCSan is compatible with KCOV.

Reviewed by Kamil.

Revert "ksyms: Use pserialize(9) for kernel access to ksyms."

ksyms: Use pserialize(9) for kernel access to ksyms.

This makes it available in interrupt context, e.g. for printing
messages with kernel symbol names for return addresses as drm wants
to do.

kcsan: fix the copyright notices

Make copystr() a MI C function, part of libkern and shared on all
architectures.

Notes:

- On alpha and ia64 the function is kept but gets renamed locally to avoid
symbol collision. This is because on these two arches, I am not sure
whether the ASM callers do not rely on fixed registers, so I prefer to
keep the ASM body for now.
- On Vax, only the symbol is removed, because the body is used from other
functions.
- On RISC-V, this change fixes a bug: copystr() was just a wrapper around
strlcpy(), but strlcpy() makes the operation less safe (strlen on the
source beyond its size).
- The kASan, kCSan and kMSan wrappers are removed, because now that
copystr() is in C, the compiler transformations are applied to it,
without the need for manual wrappers.

Could test on amd64 only, but should be fine.

Drop the todo and qualify the accesses.

branches: 1.7.2; 1.7.4;
Add a comment.

Add KCSAN instrumentation for atomic_{load,store}_*.

Instrument copyout() in kCSan, for parity with kMSan.

Don't include "opt_kcsan.h" since there's already <sys/csan.h> included.

Exclude the PTE space from KCSAN, since there the same VA can point to
different PAs.

Change kcsan_md_is_avail() to always return true; I was testing with
interrupts disabled as debugging. Change the delay/sample parameters
to have better fluidity.

Add Kernel Concurrency Sanitizer (kCSan) support. This sanitizer allows us
to detect race conditions at runtime. It is a variation of TSan that is
easy to implement and more suited to kernel internals, albeit theoretically
less precise than TSan's happens-before.

We do basically two things:

- On every KCSAN_NACCESSES (=2000) memory accesses, we create a cell
describing the access, and delay the calling CPU (10ms).

- On all memory accesses, we verify if the memory we're reading/writing
is referenced in a cell already.

The combination of the two means that, if for example cpu0 does a read that
is selected and cpu1 does a write at the same address, kCSan will fire,
because cpu1's write collides with cpu0's read cell.

The coverage of the instrumentation is the same as that of kASan. Also, the
code is organized in a way similar to kASan, so it is easy to add support
for more architectures than amd64. kCSan is compatible with KCOV.

Reviewed by Kamil.

ksyms: Use pserialize(9) for kernel access to ksyms.

This makes it available in interrupt context, e.g. for printing
messages with kernel symbol names for return addresses as drm wants
to do.

kcsan: fix the copyright notices

Make copystr() a MI C function, part of libkern and shared on all
architectures.

Notes:

- On alpha and ia64 the function is kept but gets renamed locally to avoid
symbol collision. This is because on these two arches, I am not sure
whether the ASM callers do not rely on fixed registers, so I prefer to
keep the ASM body for now.
- On Vax, only the symbol is removed, because the body is used from other
functions.
- On RISC-V, this change fixes a bug: copystr() was just a wrapper around
strlcpy(), but strlcpy() makes the operation less safe (strlen on the
source beyond its size).
- The kASan, kCSan and kMSan wrappers are removed, because now that
copystr() is in C, the compiler transformations are applied to it,
without the need for manual wrappers.

Could test on amd64 only, but should be fine.

Drop the todo and qualify the accesses.

branches: 1.7.2; 1.7.4;
Add a comment.

Add KCSAN instrumentation for atomic_{load,store}_*.

Instrument copyout() in kCSan, for parity with kMSan.

Don't include "opt_kcsan.h" since there's already <sys/csan.h> included.

Exclude the PTE space from KCSAN, since there the same VA can point to
different PAs.

Change kcsan_md_is_avail() to always return true; I was testing with
interrupts disabled as debugging. Change the delay/sample parameters
to have better fluidity.

Add Kernel Concurrency Sanitizer (kCSan) support. This sanitizer allows us
to detect race conditions at runtime. It is a variation of TSan that is
easy to implement and more suited to kernel internals, albeit theoretically
less precise than TSan's happens-before.

We do basically two things:

- On every KCSAN_NACCESSES (=2000) memory accesses, we create a cell
describing the access, and delay the calling CPU (10ms).

- On all memory accesses, we verify if the memory we're reading/writing
is referenced in a cell already.

The combination of the two means that, if for example cpu0 does a read that
is selected and cpu1 does a write at the same address, kCSan will fire,
because cpu1's write collides with cpu0's read cell.

The coverage of the instrumentation is the same as that of kASan. Also, the
code is organized in a way similar to kASan, so it is easy to add support
for more architectures than amd64. kCSan is compatible with KCOV.

Reviewed by Kamil.

kcsan: fix the copyright notices

Make copystr() a MI C function, part of libkern and shared on all
architectures.

Notes:

- On alpha and ia64 the function is kept but gets renamed locally to avoid
symbol collision. This is because on these two arches, I am not sure
whether the ASM callers do not rely on fixed registers, so I prefer to
keep the ASM body for now.
- On Vax, only the symbol is removed, because the body is used from other
functions.
- On RISC-V, this change fixes a bug: copystr() was just a wrapper around
strlcpy(), but strlcpy() makes the operation less safe (strlen on the
source beyond its size).
- The kASan, kCSan and kMSan wrappers are removed, because now that
copystr() is in C, the compiler transformations are applied to it,
without the need for manual wrappers.

Could test on amd64 only, but should be fine.

Drop the todo and qualify the accesses.

branches: 1.7.2; 1.7.4;
Add a comment.

Add KCSAN instrumentation for atomic_{load,store}_*.

Instrument copyout() in kCSan, for parity with kMSan.

Don't include "opt_kcsan.h" since there's already <sys/csan.h> included.

Exclude the PTE space from KCSAN, since there the same VA can point to
different PAs.

Change kcsan_md_is_avail() to always return true; I was testing with
interrupts disabled as debugging. Change the delay/sample parameters
to have better fluidity.

Add Kernel Concurrency Sanitizer (kCSan) support. This sanitizer allows us
to detect race conditions at runtime. It is a variation of TSan that is
easy to implement and more suited to kernel internals, albeit theoretically
less precise than TSan's happens-before.

We do basically two things:

- On every KCSAN_NACCESSES (=2000) memory accesses, we create a cell
describing the access, and delay the calling CPU (10ms).

- On all memory accesses, we verify if the memory we're reading/writing
is referenced in a cell already.

The combination of the two means that, if for example cpu0 does a read that
is selected and cpu1 does a write at the same address, kCSan will fire,
because cpu1's write collides with cpu0's read cell.

The coverage of the instrumentation is the same as that of kASan. Also, the
code is organized in a way similar to kASan, so it is easy to add support
for more architectures than amd64. kCSan is compatible with KCOV.

Reviewed by Kamil.

Make copystr() a MI C function, part of libkern and shared on all
architectures.

Notes:

- On alpha and ia64 the function is kept but gets renamed locally to avoid
symbol collision. This is because on these two arches, I am not sure
whether the ASM callers do not rely on fixed registers, so I prefer to
keep the ASM body for now.
- On Vax, only the symbol is removed, because the body is used from other
functions.
- On RISC-V, this change fixes a bug: copystr() was just a wrapper around
strlcpy(), but strlcpy() makes the operation less safe (strlen on the
source beyond its size).
- The kASan, kCSan and kMSan wrappers are removed, because now that
copystr() is in C, the compiler transformations are applied to it,
without the need for manual wrappers.

Could test on amd64 only, but should be fine.

Drop the todo and qualify the accesses.

branches: 1.7.2; 1.7.4;
Add a comment.

Add KCSAN instrumentation for atomic_{load,store}_*.

Instrument copyout() in kCSan, for parity with kMSan.

Don't include "opt_kcsan.h" since there's already <sys/csan.h> included.

Exclude the PTE space from KCSAN, since there the same VA can point to
different PAs.

Change kcsan_md_is_avail() to always return true; I was testing with
interrupts disabled as debugging. Change the delay/sample parameters
to have better fluidity.

Add Kernel Concurrency Sanitizer (kCSan) support. This sanitizer allows us
to detect race conditions at runtime. It is a variation of TSan that is
easy to implement and more suited to kernel internals, albeit theoretically
less precise than TSan's happens-before.

We do basically two things:

- On every KCSAN_NACCESSES (=2000) memory accesses, we create a cell
describing the access, and delay the calling CPU (10ms).

- On all memory accesses, we verify if the memory we're reading/writing
is referenced in a cell already.

The combination of the two means that, if for example cpu0 does a read that
is selected and cpu1 does a write at the same address, kCSan will fire,
because cpu1's write collides with cpu0's read cell.

The coverage of the instrumentation is the same as that of kASan. Also, the
code is organized in a way similar to kASan, so it is easy to add support
for more architectures than amd64. kCSan is compatible with KCOV.

Reviewed by Kamil.

Drop the todo and qualify the accesses.

branches: 1.7.4;
Add a comment.

Add KCSAN instrumentation for atomic_{load,store}_*.

Instrument copyout() in kCSan, for parity with kMSan.

Don't include "opt_kcsan.h" since there's already <sys/csan.h> included.

Exclude the PTE space from KCSAN, since there the same VA can point to
different PAs.

Change kcsan_md_is_avail() to always return true; I was testing with
interrupts disabled as debugging. Change the delay/sample parameters
to have better fluidity.

Add Kernel Concurrency Sanitizer (kCSan) support. This sanitizer allows us
to detect race conditions at runtime. It is a variation of TSan that is
easy to implement and more suited to kernel internals, albeit theoretically
less precise than TSan's happens-before.

We do basically two things:

- On every KCSAN_NACCESSES (=2000) memory accesses, we create a cell
describing the access, and delay the calling CPU (10ms).

- On all memory accesses, we verify if the memory we're reading/writing
is referenced in a cell already.

The combination of the two means that, if for example cpu0 does a read that
is selected and cpu1 does a write at the same address, kCSan will fire,
because cpu1's write collides with cpu0's read cell.

The coverage of the instrumentation is the same as that of kASan. Also, the
code is organized in a way similar to kASan, so it is easy to add support
for more architectures than amd64. kCSan is compatible with KCOV.

Reviewed by Kamil.

Add a comment.

Add KCSAN instrumentation for atomic_{load,store}_*.

Instrument copyout() in kCSan, for parity with kMSan.

Don't include "opt_kcsan.h" since there's already <sys/csan.h> included.

Exclude the PTE space from KCSAN, since there the same VA can point to
different PAs.

Change kcsan_md_is_avail() to always return true; I was testing with
interrupts disabled as debugging. Change the delay/sample parameters
to have better fluidity.

Add Kernel Concurrency Sanitizer (kCSan) support. This sanitizer allows us
to detect race conditions at runtime. It is a variation of TSan that is
easy to implement and more suited to kernel internals, albeit theoretically
less precise than TSan's happens-before.

We do basically two things:

- On every KCSAN_NACCESSES (=2000) memory accesses, we create a cell
describing the access, and delay the calling CPU (10ms).

- On all memory accesses, we verify if the memory we're reading/writing
is referenced in a cell already.

The combination of the two means that, if for example cpu0 does a read that
is selected and cpu1 does a write at the same address, kCSan will fire,
because cpu1's write collides with cpu0's read cell.

The coverage of the instrumentation is the same as that of kASan. Also, the
code is organized in a way similar to kASan, so it is easy to add support
for more architectures than amd64. kCSan is compatible with KCOV.

Reviewed by Kamil.

Add KCSAN instrumentation for atomic_{load,store}_*.

Instrument copyout() in kCSan, for parity with kMSan.

Don't include "opt_kcsan.h" since there's already <sys/csan.h> included.

Exclude the PTE space from KCSAN, since there the same VA can point to
different PAs.

Change kcsan_md_is_avail() to always return true; I was testing with
interrupts disabled as debugging. Change the delay/sample parameters
to have better fluidity.

Add Kernel Concurrency Sanitizer (kCSan) support. This sanitizer allows us
to detect race conditions at runtime. It is a variation of TSan that is
easy to implement and more suited to kernel internals, albeit theoretically
less precise than TSan's happens-before.

We do basically two things:

- On every KCSAN_NACCESSES (=2000) memory accesses, we create a cell
describing the access, and delay the calling CPU (10ms).

- On all memory accesses, we verify if the memory we're reading/writing
is referenced in a cell already.

The combination of the two means that, if for example cpu0 does a read that
is selected and cpu1 does a write at the same address, kCSan will fire,
because cpu1's write collides with cpu0's read cell.

The coverage of the instrumentation is the same as that of kASan. Also, the
code is organized in a way similar to kASan, so it is easy to add support
for more architectures than amd64. kCSan is compatible with KCOV.

Reviewed by Kamil.

Instrument copyout() in kCSan, for parity with kMSan.

Don't include "opt_kcsan.h" since there's already <sys/csan.h> included.

Exclude the PTE space from KCSAN, since there the same VA can point to
different PAs.

Change kcsan_md_is_avail() to always return true; I was testing with
interrupts disabled as debugging. Change the delay/sample parameters
to have better fluidity.

Add Kernel Concurrency Sanitizer (kCSan) support. This sanitizer allows us
to detect race conditions at runtime. It is a variation of TSan that is
easy to implement and more suited to kernel internals, albeit theoretically
less precise than TSan's happens-before.

We do basically two things:

- On every KCSAN_NACCESSES (=2000) memory accesses, we create a cell
describing the access, and delay the calling CPU (10ms).

- On all memory accesses, we verify if the memory we're reading/writing
is referenced in a cell already.

The combination of the two means that, if for example cpu0 does a read that
is selected and cpu1 does a write at the same address, kCSan will fire,
because cpu1's write collides with cpu0's read cell.

The coverage of the instrumentation is the same as that of kASan. Also, the
code is organized in a way similar to kASan, so it is easy to add support
for more architectures than amd64. kCSan is compatible with KCOV.

Reviewed by Kamil.

Exclude the PTE space from KCSAN, since there the same VA can point to
different PAs.

Change kcsan_md_is_avail() to always return true; I was testing with
interrupts disabled as debugging. Change the delay/sample parameters
to have better fluidity.

Add Kernel Concurrency Sanitizer (kCSan) support. This sanitizer allows us
to detect race conditions at runtime. It is a variation of TSan that is
easy to implement and more suited to kernel internals, albeit theoretically
less precise than TSan's happens-before.

We do basically two things:

- On every KCSAN_NACCESSES (=2000) memory accesses, we create a cell
describing the access, and delay the calling CPU (10ms).

- On all memory accesses, we verify if the memory we're reading/writing
is referenced in a cell already.

The combination of the two means that, if for example cpu0 does a read that
is selected and cpu1 does a write at the same address, kCSan will fire,
because cpu1's write collides with cpu0's read cell.

The coverage of the instrumentation is the same as that of kASan. Also, the
code is organized in a way similar to kASan, so it is easy to add support
for more architectures than amd64. kCSan is compatible with KCOV.

Reviewed by Kamil.