We should block non-wakeup interrupts until we're in the resume path and
have disabled interrupts at the CPU level again. So instead of
cpu_suspended use a new intr_suspended variable that is set and cleared in
intr_enable_wakeup() and intr_disable_wakeup().

ok mglocker@, mlarkin@

remove unused CONCAT define

Implement the guts for "suspend-to-idle" on amd64. This enables suspend
on machines that don't support S3. In its current state it doesn't save
a lot of power, but this should improve over time. Implementation of
wakeup methods is incomplete which means that some machine can't resume
at the moment.

ok mglocker@, mlarkin@, stsp@, deraadt@

Implement wakeup interrupts on amd64. Provide a dummy implementation for
i386 such that we can call the necessary hooks in the suspend/resume code
without adding #ifdefs. Tweak the arm64 implementation such that we can
call the hooks earlier as this is necessary to mask MSI and MSI-X
interrupts on arm64.

ok deraadt@, mlarkin@

Implement Multiple Message MSI support on amd64. This is experimental code
to assist qwx(4) development. We may remove this code again at some point
in the future.

Multiple Message MSI has some serious design flaws, especially when
combined with the APIC interrupt controller architecture. It was
superseded by MSI-X. Unfortunately qwx(4) does not implement MSI-X.

ok stsp@, deraadt@

Sync with i386 by asserting that IPL values should be at least IPL_NONE.

pci_intr_establish_cpu() for establishing an interrupt no a specific cpu.

the cpu is specified by a struct cpu_info *, which should generally
come from an intrmap.

this is adapted from a diff that patrick@ sent round a few years
ago for a pci_intr_map_msix_cpuid, where you asked for an msi vector
on a specific cpu, and then called pci_intr_establish with the
handle you get. kettenis pointed out that it's hard on some archs
to carry cpu on a pci interrupt handle, so i tweaked it to turn it
into a pci_intr_establish_cpu instead.

jmatthew@ and i (but mostly jmatthew@ to be honest) have been
experimenting with this api on multiple archs and it is working out
well. i'm putting this diff in now on amd64 so people can kick the
tyres a bit.

tested with hacked up vmx(4), ix(4), and mcx(4)

make intr_barrier run sched_barrier on the cpu the interrupt pinned to.

intr_barrier passed NULL to sched_barrier before this, which ends
up being the primary cpu. that's been mostly right until this point,
but is set to change.

X86_IPI_NAMES's only use was #if 0'ed out; delete both

ok kettenis@ deraadt@

Always grab the KERNEL_LOCK() for !IPL_MPSAFE handlers.

Suggested by kettenis@, ok visa@

reduce the amount of includes in arch/amd64
ok mpi@ deraadt@

Get rid of read_psl() and write_psl() by replacing
read_psl() + disable_intr() with intr_disable(),
and write_psl() with intr_restore().

OK kettenis@

Setup Hyper-V hypercall page and an IDT vector.

ok mlarkin, kettenis, deraadt

Set up an IDT vector for Xen callbacks

This adds support for delivering the combined Xen interrupt
that later fans out into event port specific (device specific)
interrupts via an IDT of a guest system.

The Xen IDT vector is set to be the first of the IPL_NET group
and is implemented the same way LAPIC timer and IPIs are done.
The additional machinery is there to be able to mask it via
standard mechanisms (e.g. splnet).

Discussed with kettenis@, OK mlarkin, reyk

Introduce intr_barrier(4), an interface that guarantees that an interrupt
handler that was running has finished.

ok miod@, guenther@ (both for the equivalent sparc64 diff)

a few more free sizes. ok deraadt

fairly simple sizes for free(); ok tedu

Move grab/release of the kernel_lock for softintrs from the ASM stubs to
softintr_dispatch(). Delete traces of long superseded stats code.

ok beck@ mpi@ uebayasi@

Use "softintr_pic0" instead of "softintr_fakepic" when faking a
struct device so there is enough space in the buffer for a NUL and
the unit is included in the string.

discussed with deraadt and millert

Remove some includes include-what-you-use claims don't
have any direct symbols used. Tested for indirect use by compiling
amd64/i386/sparc64 kernels.

ok tedu@ deraadt@

when we're entering an interrupt handler, record its ipl so splassert
can check if we're entering code we think should only be used from
lower ipls.

modelled a bit on how sparc64 does things.

with help from and ok kettenis@

delete all the simplelocks. ok deraadt

remove uneeded proc.h includes
ok mpi@ kspillner@

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

It's been a quarter century: we can assume volatile is present with that name.

ok dlg@ mpi@ deraadt@

Since interrupts at IPL_AUDIO no longer grab the kernel lock, and nobody
is supposed to establish interrupts at IPL_VM, we only need to "flatten"
mutex ipls up until IPL_TTY. Add an assertion to enforce that IPL_AUDIO
interrupts are establish with the IPL_MPSAFE flag set. Should improve
latency of audio interrupts a tiny bit.

ok matthew@

Implement a mechanism to establish interrupt handlers that don't grab the
kernel lock upon entry through a new IPL_MPSAFE flag/level.

ok mikeb@, mpi@

Take the kernel lock and call the actual interrupt handler from a
single c function. This will hopefully make easier to stop taking
the kernel lock when running "mp safe" interrupt handlers.

help from ok kettenis

Remove excessive sys/cdefs.h inclusion
ok guenther millert kettenis

Make sure we don't accidentally use ISA or EISA interrupt mappings on PCI
busses.

tested by krw@

Add bounds checks for access to mp_busses.

Raise the number of interrupt sources per CPU from 32 to 64. This effectively
triples the number of interrupt vectors that can be handled by the primary
CPU. Important for MSI, but could also fix some issues with large machines
loaded with a lot of devices.

tested by many; ok deraadt@, marco@

Remove out of data XXX comment.

More than a decade ago, interrupt handlers on sparc started returning 0
(interrupt was not for me), 1 (positive interrupt was for me), or -1
(i am not sure...). We have continued with this practice in as many
drivers as possible, throughout the tree.

This makes some of the architectures use that information in their
interrupt handler calling code -- if 1 is returned (and we know
this specific machine does not have edge-shared interrupts), we
finish servicing other possible handlers on the same pin. If the
interrupt pin remains asserted (from a different device), we will
end up back in the interrupt servicing code of course... but this is
cheaper than calling all the chained interrupts on a pin.

This does of course count on shared level interrupts being properly
sorted by IPL.

There have been some concerns about starvation of drivers which
incorrectly return 1. Those drivers should be hunted down so that
they return -1.

ok and help from various people. In snaps for about a week now.

Correct x86_atomic_*_ul() to operate on 64bit integers, eliminate
x86_atomic_*_l(), and update the callers of all of those to use the
underlying x86_atomic_*_u32() functions that they were expecting anyway.
Whether the x86_atomic_*_ul() macros should be eliminated has been deferred.

ok kettenis@, tedu@

Get rid of evcount's support for arranging counters in a tree
hierarchy. Everything attached to a single root node anyway, so at
best we had a bush.

"i think it is good" deraadt@

Set the user data of interrupt counters to the interrupt vector number used
for the interrupt. Makes the vmstat -i output similar to what we see on
i386.

ok deraadt@, krw@

Constify the what/name parameter of pci_intr_establish().

Tested by myself, sthen, oga, kettenis, and jasper.
Input from sthen and jasper.

ok kettenis

(Manpage follows shortly.)

revert all changes related to the mpsafe intr handler. i screwed up the commit
and even then it didn't work. we have higher standards than this.

the not MP version didn't compile, mglocker found

mpsafe intr_establish that doesn't get biglock, so that we may dream of the day when this is useful.
mostly macro magic that does nothing. only actually useful on amd64 for now, compliments of art.

Only call pic->pic_delroute() when disestablishing the last interrupt handler
for a pin. Resolves issues with azalia(4) devices that have no usable codecs
when they share an interrupt pin with other devices.

ok weingart@, krw@

replace M_WAITOK with M_NOWAIT

ok krw@ kettenis@

Rename IPL_SOFTSERIAL to IPL_SOFTTY and rename the associated symbols.
Remove IPL_SERIAL since it is unused.

ok krw@, weingart@, deraadt@

When allocating an interrupt 'slot', always check all slots to see if we
already allocated a slot for a particular apic pin.

ok weingart@

MALLOC/FREE -> malloc/free and M_ZERO changes

ok krw@

evcnt & friends were replaced by a proper interrupt counting mechanism 3 years ago; ok miod

ansi/deregister

Fix for ioapic irq routing.

From NetBSD, NetBSD PR 22728 for details

ok beck@ krw@
tested by beck@ krw@ marc@ and a few others on a number of amd64 systems.

Turn CPU_INFO_FOREACH into a real construct, like all queue(3) iterators,
instead of the contents of a for() loop. No functional change.
From the m88k SMP tree; ok art@ deraadt@

[complete diff this time]

In splraise, change an:
if (x > foo->bar)
foo->bar = x;
to:
foo->bar = MAX(x, foo->bar);

This forces gcc to generate much better code even though both
experessions are equivalent. Normally I wouldn't bother with
microoptimizations like this, but I needed some generated assembler
that uses cmov and splraise used so often..

ok toby@ (well, he ok:ed a diff that didn't use the MAX macro, but it's
the same code)

remove trailing newline in panic(9); ok millert@ and deraadt@

If irq-sharing happens over different ioapics, we must not overwrite setup info for one of them

Check for memory allocation failure before zeroing memory, not after;
from mpech@

From NetBSD:
date: 2004/06/28 09:13:11; author: fvdl; state: Exp; lines: +6 -5
Updaing ci_ilevel and testing ci_ipending must be done with all interrupts
off, or priority inversion can occur, which can lead to IPI deadlocks.
Leaves interrupts off for a bit longer, sadly, but with no noticeable
effects on the systems I tested on.

From YAMAMOTO Takashi.

Fixes the IPI rendezvous panics for me.

pass pin instead of level out to userland

Use new event counter API for interrupt counting on amd64. Based in part
on some changes in the i386 codebase.

deinline splraise, spllower and setsoftint.
Makes the kernel smaller and faster.
deraadt@ ok

SMP support. Big parts from NetBSD, but with some really serious debugging
done by me, niklas and others. Especially wrt. NXE support.

Still needs some polishing, especially in dmesg messages, but we're now
building kernel faster than ever.

an amd64 arch support.
hacked by art@ from netbsd sources and then later debugged
by me into the shape where it can host itself.
no bootloader yet as needs redoing from the
recent advanced i386 sources (anyone? ;)

remove unused CONCAT define

Implement the guts for "suspend-to-idle" on amd64. This enables suspend
on machines that don't support S3. In its current state it doesn't save
a lot of power, but this should improve over time. Implementation of
wakeup methods is incomplete which means that some machine can't resume
at the moment.

ok mglocker@, mlarkin@, stsp@, deraadt@

Implement wakeup interrupts on amd64. Provide a dummy implementation for
i386 such that we can call the necessary hooks in the suspend/resume code
without adding #ifdefs. Tweak the arm64 implementation such that we can
call the hooks earlier as this is necessary to mask MSI and MSI-X
interrupts on arm64.

ok deraadt@, mlarkin@

Implement Multiple Message MSI support on amd64. This is experimental code
to assist qwx(4) development. We may remove this code again at some point
in the future.

Multiple Message MSI has some serious design flaws, especially when
combined with the APIC interrupt controller architecture. It was
superseded by MSI-X. Unfortunately qwx(4) does not implement MSI-X.

ok stsp@, deraadt@

Sync with i386 by asserting that IPL values should be at least IPL_NONE.

pci_intr_establish_cpu() for establishing an interrupt no a specific cpu.

the cpu is specified by a struct cpu_info *, which should generally
come from an intrmap.

this is adapted from a diff that patrick@ sent round a few years
ago for a pci_intr_map_msix_cpuid, where you asked for an msi vector
on a specific cpu, and then called pci_intr_establish with the
handle you get. kettenis pointed out that it's hard on some archs
to carry cpu on a pci interrupt handle, so i tweaked it to turn it
into a pci_intr_establish_cpu instead.

jmatthew@ and i (but mostly jmatthew@ to be honest) have been
experimenting with this api on multiple archs and it is working out
well. i'm putting this diff in now on amd64 so people can kick the
tyres a bit.

tested with hacked up vmx(4), ix(4), and mcx(4)

make intr_barrier run sched_barrier on the cpu the interrupt pinned to.

intr_barrier passed NULL to sched_barrier before this, which ends
up being the primary cpu. that's been mostly right until this point,
but is set to change.

X86_IPI_NAMES's only use was #if 0'ed out; delete both

ok kettenis@ deraadt@

Always grab the KERNEL_LOCK() for !IPL_MPSAFE handlers.

Suggested by kettenis@, ok visa@

reduce the amount of includes in arch/amd64
ok mpi@ deraadt@

Get rid of read_psl() and write_psl() by replacing
read_psl() + disable_intr() with intr_disable(),
and write_psl() with intr_restore().

OK kettenis@

Setup Hyper-V hypercall page and an IDT vector.

ok mlarkin, kettenis, deraadt

Set up an IDT vector for Xen callbacks

This adds support for delivering the combined Xen interrupt
that later fans out into event port specific (device specific)
interrupts via an IDT of a guest system.

The Xen IDT vector is set to be the first of the IPL_NET group
and is implemented the same way LAPIC timer and IPIs are done.
The additional machinery is there to be able to mask it via
standard mechanisms (e.g. splnet).

Discussed with kettenis@, OK mlarkin, reyk

Introduce intr_barrier(4), an interface that guarantees that an interrupt
handler that was running has finished.

ok miod@, guenther@ (both for the equivalent sparc64 diff)

a few more free sizes. ok deraadt

fairly simple sizes for free(); ok tedu

Move grab/release of the kernel_lock for softintrs from the ASM stubs to
softintr_dispatch(). Delete traces of long superseded stats code.

ok beck@ mpi@ uebayasi@

Use "softintr_pic0" instead of "softintr_fakepic" when faking a
struct device so there is enough space in the buffer for a NUL and
the unit is included in the string.

discussed with deraadt and millert

Remove some includes include-what-you-use claims don't
have any direct symbols used. Tested for indirect use by compiling
amd64/i386/sparc64 kernels.

ok tedu@ deraadt@

when we're entering an interrupt handler, record its ipl so splassert
can check if we're entering code we think should only be used from
lower ipls.

modelled a bit on how sparc64 does things.

with help from and ok kettenis@

delete all the simplelocks. ok deraadt

remove uneeded proc.h includes
ok mpi@ kspillner@

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

It's been a quarter century: we can assume volatile is present with that name.

ok dlg@ mpi@ deraadt@

Since interrupts at IPL_AUDIO no longer grab the kernel lock, and nobody
is supposed to establish interrupts at IPL_VM, we only need to "flatten"
mutex ipls up until IPL_TTY. Add an assertion to enforce that IPL_AUDIO
interrupts are establish with the IPL_MPSAFE flag set. Should improve
latency of audio interrupts a tiny bit.

ok matthew@

Implement a mechanism to establish interrupt handlers that don't grab the
kernel lock upon entry through a new IPL_MPSAFE flag/level.

ok mikeb@, mpi@

Take the kernel lock and call the actual interrupt handler from a
single c function. This will hopefully make easier to stop taking
the kernel lock when running "mp safe" interrupt handlers.

help from ok kettenis

Remove excessive sys/cdefs.h inclusion
ok guenther millert kettenis

Make sure we don't accidentally use ISA or EISA interrupt mappings on PCI
busses.

tested by krw@

Add bounds checks for access to mp_busses.

Raise the number of interrupt sources per CPU from 32 to 64. This effectively
triples the number of interrupt vectors that can be handled by the primary
CPU. Important for MSI, but could also fix some issues with large machines
loaded with a lot of devices.

tested by many; ok deraadt@, marco@

Remove out of data XXX comment.

More than a decade ago, interrupt handlers on sparc started returning 0
(interrupt was not for me), 1 (positive interrupt was for me), or -1
(i am not sure...). We have continued with this practice in as many
drivers as possible, throughout the tree.

This makes some of the architectures use that information in their
interrupt handler calling code -- if 1 is returned (and we know
this specific machine does not have edge-shared interrupts), we
finish servicing other possible handlers on the same pin. If the
interrupt pin remains asserted (from a different device), we will
end up back in the interrupt servicing code of course... but this is
cheaper than calling all the chained interrupts on a pin.

This does of course count on shared level interrupts being properly
sorted by IPL.

There have been some concerns about starvation of drivers which
incorrectly return 1. Those drivers should be hunted down so that
they return -1.

ok and help from various people. In snaps for about a week now.

Correct x86_atomic_*_ul() to operate on 64bit integers, eliminate
x86_atomic_*_l(), and update the callers of all of those to use the
underlying x86_atomic_*_u32() functions that they were expecting anyway.
Whether the x86_atomic_*_ul() macros should be eliminated has been deferred.

ok kettenis@, tedu@

Get rid of evcount's support for arranging counters in a tree
hierarchy. Everything attached to a single root node anyway, so at
best we had a bush.

"i think it is good" deraadt@

Set the user data of interrupt counters to the interrupt vector number used
for the interrupt. Makes the vmstat -i output similar to what we see on
i386.

ok deraadt@, krw@

Constify the what/name parameter of pci_intr_establish().

Tested by myself, sthen, oga, kettenis, and jasper.
Input from sthen and jasper.

ok kettenis

(Manpage follows shortly.)

revert all changes related to the mpsafe intr handler. i screwed up the commit
and even then it didn't work. we have higher standards than this.

the not MP version didn't compile, mglocker found

mpsafe intr_establish that doesn't get biglock, so that we may dream of the day when this is useful.
mostly macro magic that does nothing. only actually useful on amd64 for now, compliments of art.

Only call pic->pic_delroute() when disestablishing the last interrupt handler
for a pin. Resolves issues with azalia(4) devices that have no usable codecs
when they share an interrupt pin with other devices.

ok weingart@, krw@

replace M_WAITOK with M_NOWAIT

ok krw@ kettenis@

Rename IPL_SOFTSERIAL to IPL_SOFTTY and rename the associated symbols.
Remove IPL_SERIAL since it is unused.

ok krw@, weingart@, deraadt@

When allocating an interrupt 'slot', always check all slots to see if we
already allocated a slot for a particular apic pin.

ok weingart@

MALLOC/FREE -> malloc/free and M_ZERO changes

ok krw@

evcnt & friends were replaced by a proper interrupt counting mechanism 3 years ago; ok miod

ansi/deregister

Fix for ioapic irq routing.

From NetBSD, NetBSD PR 22728 for details

ok beck@ krw@
tested by beck@ krw@ marc@ and a few others on a number of amd64 systems.

Turn CPU_INFO_FOREACH into a real construct, like all queue(3) iterators,
instead of the contents of a for() loop. No functional change.
From the m88k SMP tree; ok art@ deraadt@

[complete diff this time]

In splraise, change an:
if (x > foo->bar)
foo->bar = x;
to:
foo->bar = MAX(x, foo->bar);

This forces gcc to generate much better code even though both
experessions are equivalent. Normally I wouldn't bother with
microoptimizations like this, but I needed some generated assembler
that uses cmov and splraise used so often..

ok toby@ (well, he ok:ed a diff that didn't use the MAX macro, but it's
the same code)

remove trailing newline in panic(9); ok millert@ and deraadt@

If irq-sharing happens over different ioapics, we must not overwrite setup info for one of them

Check for memory allocation failure before zeroing memory, not after;
from mpech@

From NetBSD:
date: 2004/06/28 09:13:11; author: fvdl; state: Exp; lines: +6 -5
Updaing ci_ilevel and testing ci_ipending must be done with all interrupts
off, or priority inversion can occur, which can lead to IPI deadlocks.
Leaves interrupts off for a bit longer, sadly, but with no noticeable
effects on the systems I tested on.

From YAMAMOTO Takashi.

Fixes the IPI rendezvous panics for me.

pass pin instead of level out to userland

Use new event counter API for interrupt counting on amd64. Based in part
on some changes in the i386 codebase.

deinline splraise, spllower and setsoftint.
Makes the kernel smaller and faster.
deraadt@ ok

SMP support. Big parts from NetBSD, but with some really serious debugging
done by me, niklas and others. Especially wrt. NXE support.

Still needs some polishing, especially in dmesg messages, but we're now
building kernel faster than ever.

an amd64 arch support.
hacked by art@ from netbsd sources and then later debugged
by me into the shape where it can host itself.
no bootloader yet as needs redoing from the
recent advanced i386 sources (anyone? ;)

Implement the guts for "suspend-to-idle" on amd64. This enables suspend
on machines that don't support S3. In its current state it doesn't save
a lot of power, but this should improve over time. Implementation of
wakeup methods is incomplete which means that some machine can't resume
at the moment.

ok mglocker@, mlarkin@, stsp@, deraadt@

Implement wakeup interrupts on amd64. Provide a dummy implementation for
i386 such that we can call the necessary hooks in the suspend/resume code
without adding #ifdefs. Tweak the arm64 implementation such that we can
call the hooks earlier as this is necessary to mask MSI and MSI-X
interrupts on arm64.

ok deraadt@, mlarkin@

Implement Multiple Message MSI support on amd64. This is experimental code
to assist qwx(4) development. We may remove this code again at some point
in the future.

Multiple Message MSI has some serious design flaws, especially when
combined with the APIC interrupt controller architecture. It was
superseded by MSI-X. Unfortunately qwx(4) does not implement MSI-X.

ok stsp@, deraadt@

Sync with i386 by asserting that IPL values should be at least IPL_NONE.

pci_intr_establish_cpu() for establishing an interrupt no a specific cpu.

the cpu is specified by a struct cpu_info *, which should generally
come from an intrmap.

this is adapted from a diff that patrick@ sent round a few years
ago for a pci_intr_map_msix_cpuid, where you asked for an msi vector
on a specific cpu, and then called pci_intr_establish with the
handle you get. kettenis pointed out that it's hard on some archs
to carry cpu on a pci interrupt handle, so i tweaked it to turn it
into a pci_intr_establish_cpu instead.

jmatthew@ and i (but mostly jmatthew@ to be honest) have been
experimenting with this api on multiple archs and it is working out
well. i'm putting this diff in now on amd64 so people can kick the
tyres a bit.

tested with hacked up vmx(4), ix(4), and mcx(4)

make intr_barrier run sched_barrier on the cpu the interrupt pinned to.

intr_barrier passed NULL to sched_barrier before this, which ends
up being the primary cpu. that's been mostly right until this point,
but is set to change.

X86_IPI_NAMES's only use was #if 0'ed out; delete both

ok kettenis@ deraadt@

Always grab the KERNEL_LOCK() for !IPL_MPSAFE handlers.

Suggested by kettenis@, ok visa@

reduce the amount of includes in arch/amd64
ok mpi@ deraadt@

Get rid of read_psl() and write_psl() by replacing
read_psl() + disable_intr() with intr_disable(),
and write_psl() with intr_restore().

OK kettenis@

Setup Hyper-V hypercall page and an IDT vector.

ok mlarkin, kettenis, deraadt

Set up an IDT vector for Xen callbacks

This adds support for delivering the combined Xen interrupt
that later fans out into event port specific (device specific)
interrupts via an IDT of a guest system.

The Xen IDT vector is set to be the first of the IPL_NET group
and is implemented the same way LAPIC timer and IPIs are done.
The additional machinery is there to be able to mask it via
standard mechanisms (e.g. splnet).

Discussed with kettenis@, OK mlarkin, reyk

Introduce intr_barrier(4), an interface that guarantees that an interrupt
handler that was running has finished.

ok miod@, guenther@ (both for the equivalent sparc64 diff)

a few more free sizes. ok deraadt

fairly simple sizes for free(); ok tedu

Move grab/release of the kernel_lock for softintrs from the ASM stubs to
softintr_dispatch(). Delete traces of long superseded stats code.

ok beck@ mpi@ uebayasi@

Use "softintr_pic0" instead of "softintr_fakepic" when faking a
struct device so there is enough space in the buffer for a NUL and
the unit is included in the string.

discussed with deraadt and millert

Remove some includes include-what-you-use claims don't
have any direct symbols used. Tested for indirect use by compiling
amd64/i386/sparc64 kernels.

ok tedu@ deraadt@

when we're entering an interrupt handler, record its ipl so splassert
can check if we're entering code we think should only be used from
lower ipls.

modelled a bit on how sparc64 does things.

with help from and ok kettenis@

delete all the simplelocks. ok deraadt

remove uneeded proc.h includes
ok mpi@ kspillner@

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

It's been a quarter century: we can assume volatile is present with that name.

ok dlg@ mpi@ deraadt@

Since interrupts at IPL_AUDIO no longer grab the kernel lock, and nobody
is supposed to establish interrupts at IPL_VM, we only need to "flatten"
mutex ipls up until IPL_TTY. Add an assertion to enforce that IPL_AUDIO
interrupts are establish with the IPL_MPSAFE flag set. Should improve
latency of audio interrupts a tiny bit.

ok matthew@

Implement a mechanism to establish interrupt handlers that don't grab the
kernel lock upon entry through a new IPL_MPSAFE flag/level.

ok mikeb@, mpi@

Take the kernel lock and call the actual interrupt handler from a
single c function. This will hopefully make easier to stop taking
the kernel lock when running "mp safe" interrupt handlers.

help from ok kettenis

Remove excessive sys/cdefs.h inclusion
ok guenther millert kettenis

Make sure we don't accidentally use ISA or EISA interrupt mappings on PCI
busses.

tested by krw@

Add bounds checks for access to mp_busses.

Raise the number of interrupt sources per CPU from 32 to 64. This effectively
triples the number of interrupt vectors that can be handled by the primary
CPU. Important for MSI, but could also fix some issues with large machines
loaded with a lot of devices.

tested by many; ok deraadt@, marco@

Remove out of data XXX comment.

More than a decade ago, interrupt handlers on sparc started returning 0
(interrupt was not for me), 1 (positive interrupt was for me), or -1
(i am not sure...). We have continued with this practice in as many
drivers as possible, throughout the tree.

This makes some of the architectures use that information in their
interrupt handler calling code -- if 1 is returned (and we know
this specific machine does not have edge-shared interrupts), we
finish servicing other possible handlers on the same pin. If the
interrupt pin remains asserted (from a different device), we will
end up back in the interrupt servicing code of course... but this is
cheaper than calling all the chained interrupts on a pin.

This does of course count on shared level interrupts being properly
sorted by IPL.

There have been some concerns about starvation of drivers which
incorrectly return 1. Those drivers should be hunted down so that
they return -1.

ok and help from various people. In snaps for about a week now.

Correct x86_atomic_*_ul() to operate on 64bit integers, eliminate
x86_atomic_*_l(), and update the callers of all of those to use the
underlying x86_atomic_*_u32() functions that they were expecting anyway.
Whether the x86_atomic_*_ul() macros should be eliminated has been deferred.

ok kettenis@, tedu@

Get rid of evcount's support for arranging counters in a tree
hierarchy. Everything attached to a single root node anyway, so at
best we had a bush.

"i think it is good" deraadt@

Set the user data of interrupt counters to the interrupt vector number used
for the interrupt. Makes the vmstat -i output similar to what we see on
i386.

ok deraadt@, krw@

Constify the what/name parameter of pci_intr_establish().

Tested by myself, sthen, oga, kettenis, and jasper.
Input from sthen and jasper.

ok kettenis

(Manpage follows shortly.)

revert all changes related to the mpsafe intr handler. i screwed up the commit
and even then it didn't work. we have higher standards than this.

the not MP version didn't compile, mglocker found

mpsafe intr_establish that doesn't get biglock, so that we may dream of the day when this is useful.
mostly macro magic that does nothing. only actually useful on amd64 for now, compliments of art.

Only call pic->pic_delroute() when disestablishing the last interrupt handler
for a pin. Resolves issues with azalia(4) devices that have no usable codecs
when they share an interrupt pin with other devices.

ok weingart@, krw@

replace M_WAITOK with M_NOWAIT

ok krw@ kettenis@

Rename IPL_SOFTSERIAL to IPL_SOFTTY and rename the associated symbols.
Remove IPL_SERIAL since it is unused.

ok krw@, weingart@, deraadt@

When allocating an interrupt 'slot', always check all slots to see if we
already allocated a slot for a particular apic pin.

ok weingart@

MALLOC/FREE -> malloc/free and M_ZERO changes

ok krw@

evcnt & friends were replaced by a proper interrupt counting mechanism 3 years ago; ok miod

ansi/deregister

Fix for ioapic irq routing.

From NetBSD, NetBSD PR 22728 for details

ok beck@ krw@
tested by beck@ krw@ marc@ and a few others on a number of amd64 systems.

Turn CPU_INFO_FOREACH into a real construct, like all queue(3) iterators,
instead of the contents of a for() loop. No functional change.
From the m88k SMP tree; ok art@ deraadt@

[complete diff this time]

In splraise, change an:
if (x > foo->bar)
foo->bar = x;
to:
foo->bar = MAX(x, foo->bar);

This forces gcc to generate much better code even though both
experessions are equivalent. Normally I wouldn't bother with
microoptimizations like this, but I needed some generated assembler
that uses cmov and splraise used so often..

ok toby@ (well, he ok:ed a diff that didn't use the MAX macro, but it's
the same code)

remove trailing newline in panic(9); ok millert@ and deraadt@

If irq-sharing happens over different ioapics, we must not overwrite setup info for one of them

Check for memory allocation failure before zeroing memory, not after;
from mpech@

From NetBSD:
date: 2004/06/28 09:13:11; author: fvdl; state: Exp; lines: +6 -5
Updaing ci_ilevel and testing ci_ipending must be done with all interrupts
off, or priority inversion can occur, which can lead to IPI deadlocks.
Leaves interrupts off for a bit longer, sadly, but with no noticeable
effects on the systems I tested on.

From YAMAMOTO Takashi.

Fixes the IPI rendezvous panics for me.

pass pin instead of level out to userland

Use new event counter API for interrupt counting on amd64. Based in part
on some changes in the i386 codebase.

deinline splraise, spllower and setsoftint.
Makes the kernel smaller and faster.
deraadt@ ok

SMP support. Big parts from NetBSD, but with some really serious debugging
done by me, niklas and others. Especially wrt. NXE support.

Still needs some polishing, especially in dmesg messages, but we're now
building kernel faster than ever.

an amd64 arch support.
hacked by art@ from netbsd sources and then later debugged
by me into the shape where it can host itself.
no bootloader yet as needs redoing from the
recent advanced i386 sources (anyone? ;)

Implement Multiple Message MSI support on amd64. This is experimental code
to assist qwx(4) development. We may remove this code again at some point
in the future.

Multiple Message MSI has some serious design flaws, especially when
combined with the APIC interrupt controller architecture. It was
superseded by MSI-X. Unfortunately qwx(4) does not implement MSI-X.

ok stsp@, deraadt@

Sync with i386 by asserting that IPL values should be at least IPL_NONE.

pci_intr_establish_cpu() for establishing an interrupt no a specific cpu.

the cpu is specified by a struct cpu_info *, which should generally
come from an intrmap.

this is adapted from a diff that patrick@ sent round a few years
ago for a pci_intr_map_msix_cpuid, where you asked for an msi vector
on a specific cpu, and then called pci_intr_establish with the
handle you get. kettenis pointed out that it's hard on some archs
to carry cpu on a pci interrupt handle, so i tweaked it to turn it
into a pci_intr_establish_cpu instead.

jmatthew@ and i (but mostly jmatthew@ to be honest) have been
experimenting with this api on multiple archs and it is working out
well. i'm putting this diff in now on amd64 so people can kick the
tyres a bit.

tested with hacked up vmx(4), ix(4), and mcx(4)

make intr_barrier run sched_barrier on the cpu the interrupt pinned to.

intr_barrier passed NULL to sched_barrier before this, which ends
up being the primary cpu. that's been mostly right until this point,
but is set to change.

X86_IPI_NAMES's only use was #if 0'ed out; delete both

ok kettenis@ deraadt@

Always grab the KERNEL_LOCK() for !IPL_MPSAFE handlers.

Suggested by kettenis@, ok visa@

reduce the amount of includes in arch/amd64
ok mpi@ deraadt@

Get rid of read_psl() and write_psl() by replacing
read_psl() + disable_intr() with intr_disable(),
and write_psl() with intr_restore().

OK kettenis@

Setup Hyper-V hypercall page and an IDT vector.

ok mlarkin, kettenis, deraadt

Set up an IDT vector for Xen callbacks

This adds support for delivering the combined Xen interrupt
that later fans out into event port specific (device specific)
interrupts via an IDT of a guest system.

The Xen IDT vector is set to be the first of the IPL_NET group
and is implemented the same way LAPIC timer and IPIs are done.
The additional machinery is there to be able to mask it via
standard mechanisms (e.g. splnet).

Discussed with kettenis@, OK mlarkin, reyk

Introduce intr_barrier(4), an interface that guarantees that an interrupt
handler that was running has finished.

ok miod@, guenther@ (both for the equivalent sparc64 diff)

a few more free sizes. ok deraadt

fairly simple sizes for free(); ok tedu

Move grab/release of the kernel_lock for softintrs from the ASM stubs to
softintr_dispatch(). Delete traces of long superseded stats code.

ok beck@ mpi@ uebayasi@

Use "softintr_pic0" instead of "softintr_fakepic" when faking a
struct device so there is enough space in the buffer for a NUL and
the unit is included in the string.

discussed with deraadt and millert

Remove some includes include-what-you-use claims don't
have any direct symbols used. Tested for indirect use by compiling
amd64/i386/sparc64 kernels.

ok tedu@ deraadt@

when we're entering an interrupt handler, record its ipl so splassert
can check if we're entering code we think should only be used from
lower ipls.

modelled a bit on how sparc64 does things.

with help from and ok kettenis@

delete all the simplelocks. ok deraadt

remove uneeded proc.h includes
ok mpi@ kspillner@

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

It's been a quarter century: we can assume volatile is present with that name.

ok dlg@ mpi@ deraadt@

Since interrupts at IPL_AUDIO no longer grab the kernel lock, and nobody
is supposed to establish interrupts at IPL_VM, we only need to "flatten"
mutex ipls up until IPL_TTY. Add an assertion to enforce that IPL_AUDIO
interrupts are establish with the IPL_MPSAFE flag set. Should improve
latency of audio interrupts a tiny bit.

ok matthew@

Implement a mechanism to establish interrupt handlers that don't grab the
kernel lock upon entry through a new IPL_MPSAFE flag/level.

ok mikeb@, mpi@

Take the kernel lock and call the actual interrupt handler from a
single c function. This will hopefully make easier to stop taking
the kernel lock when running "mp safe" interrupt handlers.

help from ok kettenis

Remove excessive sys/cdefs.h inclusion
ok guenther millert kettenis

Make sure we don't accidentally use ISA or EISA interrupt mappings on PCI
busses.

tested by krw@

Add bounds checks for access to mp_busses.

Raise the number of interrupt sources per CPU from 32 to 64. This effectively
triples the number of interrupt vectors that can be handled by the primary
CPU. Important for MSI, but could also fix some issues with large machines
loaded with a lot of devices.

tested by many; ok deraadt@, marco@

Remove out of data XXX comment.

More than a decade ago, interrupt handlers on sparc started returning 0
(interrupt was not for me), 1 (positive interrupt was for me), or -1
(i am not sure...). We have continued with this practice in as many
drivers as possible, throughout the tree.

This makes some of the architectures use that information in their
interrupt handler calling code -- if 1 is returned (and we know
this specific machine does not have edge-shared interrupts), we
finish servicing other possible handlers on the same pin. If the
interrupt pin remains asserted (from a different device), we will
end up back in the interrupt servicing code of course... but this is
cheaper than calling all the chained interrupts on a pin.

This does of course count on shared level interrupts being properly
sorted by IPL.

There have been some concerns about starvation of drivers which
incorrectly return 1. Those drivers should be hunted down so that
they return -1.

ok and help from various people. In snaps for about a week now.

Correct x86_atomic_*_ul() to operate on 64bit integers, eliminate
x86_atomic_*_l(), and update the callers of all of those to use the
underlying x86_atomic_*_u32() functions that they were expecting anyway.
Whether the x86_atomic_*_ul() macros should be eliminated has been deferred.

ok kettenis@, tedu@

Get rid of evcount's support for arranging counters in a tree
hierarchy. Everything attached to a single root node anyway, so at
best we had a bush.

"i think it is good" deraadt@

Set the user data of interrupt counters to the interrupt vector number used
for the interrupt. Makes the vmstat -i output similar to what we see on
i386.

ok deraadt@, krw@

Constify the what/name parameter of pci_intr_establish().

Tested by myself, sthen, oga, kettenis, and jasper.
Input from sthen and jasper.

ok kettenis

(Manpage follows shortly.)

revert all changes related to the mpsafe intr handler. i screwed up the commit
and even then it didn't work. we have higher standards than this.

the not MP version didn't compile, mglocker found

mpsafe intr_establish that doesn't get biglock, so that we may dream of the day when this is useful.
mostly macro magic that does nothing. only actually useful on amd64 for now, compliments of art.

Only call pic->pic_delroute() when disestablishing the last interrupt handler
for a pin. Resolves issues with azalia(4) devices that have no usable codecs
when they share an interrupt pin with other devices.

ok weingart@, krw@

replace M_WAITOK with M_NOWAIT

ok krw@ kettenis@

Rename IPL_SOFTSERIAL to IPL_SOFTTY and rename the associated symbols.
Remove IPL_SERIAL since it is unused.

ok krw@, weingart@, deraadt@

When allocating an interrupt 'slot', always check all slots to see if we
already allocated a slot for a particular apic pin.

ok weingart@

MALLOC/FREE -> malloc/free and M_ZERO changes

ok krw@

evcnt & friends were replaced by a proper interrupt counting mechanism 3 years ago; ok miod

ansi/deregister

Fix for ioapic irq routing.

From NetBSD, NetBSD PR 22728 for details

ok beck@ krw@
tested by beck@ krw@ marc@ and a few others on a number of amd64 systems.

Turn CPU_INFO_FOREACH into a real construct, like all queue(3) iterators,
instead of the contents of a for() loop. No functional change.
From the m88k SMP tree; ok art@ deraadt@

[complete diff this time]

In splraise, change an:
if (x > foo->bar)
foo->bar = x;
to:
foo->bar = MAX(x, foo->bar);

This forces gcc to generate much better code even though both
experessions are equivalent. Normally I wouldn't bother with
microoptimizations like this, but I needed some generated assembler
that uses cmov and splraise used so often..

ok toby@ (well, he ok:ed a diff that didn't use the MAX macro, but it's
the same code)

remove trailing newline in panic(9); ok millert@ and deraadt@

If irq-sharing happens over different ioapics, we must not overwrite setup info for one of them

Check for memory allocation failure before zeroing memory, not after;
from mpech@

From NetBSD:
date: 2004/06/28 09:13:11; author: fvdl; state: Exp; lines: +6 -5
Updaing ci_ilevel and testing ci_ipending must be done with all interrupts
off, or priority inversion can occur, which can lead to IPI deadlocks.
Leaves interrupts off for a bit longer, sadly, but with no noticeable
effects on the systems I tested on.

From YAMAMOTO Takashi.

Fixes the IPI rendezvous panics for me.

pass pin instead of level out to userland

Use new event counter API for interrupt counting on amd64. Based in part
on some changes in the i386 codebase.

deinline splraise, spllower and setsoftint.
Makes the kernel smaller and faster.
deraadt@ ok

SMP support. Big parts from NetBSD, but with some really serious debugging
done by me, niklas and others. Especially wrt. NXE support.

Still needs some polishing, especially in dmesg messages, but we're now
building kernel faster than ever.

an amd64 arch support.
hacked by art@ from netbsd sources and then later debugged
by me into the shape where it can host itself.
no bootloader yet as needs redoing from the
recent advanced i386 sources (anyone? ;)

Sync with i386 by asserting that IPL values should be at least IPL_NONE.

pci_intr_establish_cpu() for establishing an interrupt no a specific cpu.

the cpu is specified by a struct cpu_info *, which should generally
come from an intrmap.

this is adapted from a diff that patrick@ sent round a few years
ago for a pci_intr_map_msix_cpuid, where you asked for an msi vector
on a specific cpu, and then called pci_intr_establish with the
handle you get. kettenis pointed out that it's hard on some archs
to carry cpu on a pci interrupt handle, so i tweaked it to turn it
into a pci_intr_establish_cpu instead.

jmatthew@ and i (but mostly jmatthew@ to be honest) have been
experimenting with this api on multiple archs and it is working out
well. i'm putting this diff in now on amd64 so people can kick the
tyres a bit.

tested with hacked up vmx(4), ix(4), and mcx(4)

make intr_barrier run sched_barrier on the cpu the interrupt pinned to.

intr_barrier passed NULL to sched_barrier before this, which ends
up being the primary cpu. that's been mostly right until this point,
but is set to change.

X86_IPI_NAMES's only use was #if 0'ed out; delete both

ok kettenis@ deraadt@

Always grab the KERNEL_LOCK() for !IPL_MPSAFE handlers.

Suggested by kettenis@, ok visa@

reduce the amount of includes in arch/amd64
ok mpi@ deraadt@

Get rid of read_psl() and write_psl() by replacing
read_psl() + disable_intr() with intr_disable(),
and write_psl() with intr_restore().

OK kettenis@

Setup Hyper-V hypercall page and an IDT vector.

ok mlarkin, kettenis, deraadt

Set up an IDT vector for Xen callbacks

This adds support for delivering the combined Xen interrupt
that later fans out into event port specific (device specific)
interrupts via an IDT of a guest system.

The Xen IDT vector is set to be the first of the IPL_NET group
and is implemented the same way LAPIC timer and IPIs are done.
The additional machinery is there to be able to mask it via
standard mechanisms (e.g. splnet).

Discussed with kettenis@, OK mlarkin, reyk

Introduce intr_barrier(4), an interface that guarantees that an interrupt
handler that was running has finished.

ok miod@, guenther@ (both for the equivalent sparc64 diff)

a few more free sizes. ok deraadt

fairly simple sizes for free(); ok tedu

Move grab/release of the kernel_lock for softintrs from the ASM stubs to
softintr_dispatch(). Delete traces of long superseded stats code.

ok beck@ mpi@ uebayasi@

Use "softintr_pic0" instead of "softintr_fakepic" when faking a
struct device so there is enough space in the buffer for a NUL and
the unit is included in the string.

discussed with deraadt and millert

Remove some includes include-what-you-use claims don't
have any direct symbols used. Tested for indirect use by compiling
amd64/i386/sparc64 kernels.

ok tedu@ deraadt@

when we're entering an interrupt handler, record its ipl so splassert
can check if we're entering code we think should only be used from
lower ipls.

modelled a bit on how sparc64 does things.

with help from and ok kettenis@

delete all the simplelocks. ok deraadt

remove uneeded proc.h includes
ok mpi@ kspillner@

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

It's been a quarter century: we can assume volatile is present with that name.

ok dlg@ mpi@ deraadt@

Since interrupts at IPL_AUDIO no longer grab the kernel lock, and nobody
is supposed to establish interrupts at IPL_VM, we only need to "flatten"
mutex ipls up until IPL_TTY. Add an assertion to enforce that IPL_AUDIO
interrupts are establish with the IPL_MPSAFE flag set. Should improve
latency of audio interrupts a tiny bit.

ok matthew@

Implement a mechanism to establish interrupt handlers that don't grab the
kernel lock upon entry through a new IPL_MPSAFE flag/level.

ok mikeb@, mpi@

Take the kernel lock and call the actual interrupt handler from a
single c function. This will hopefully make easier to stop taking
the kernel lock when running "mp safe" interrupt handlers.

help from ok kettenis

Remove excessive sys/cdefs.h inclusion
ok guenther millert kettenis

Make sure we don't accidentally use ISA or EISA interrupt mappings on PCI
busses.

tested by krw@

Add bounds checks for access to mp_busses.

Raise the number of interrupt sources per CPU from 32 to 64. This effectively
triples the number of interrupt vectors that can be handled by the primary
CPU. Important for MSI, but could also fix some issues with large machines
loaded with a lot of devices.

tested by many; ok deraadt@, marco@

Remove out of data XXX comment.

More than a decade ago, interrupt handlers on sparc started returning 0
(interrupt was not for me), 1 (positive interrupt was for me), or -1
(i am not sure...). We have continued with this practice in as many
drivers as possible, throughout the tree.

This makes some of the architectures use that information in their
interrupt handler calling code -- if 1 is returned (and we know
this specific machine does not have edge-shared interrupts), we
finish servicing other possible handlers on the same pin. If the
interrupt pin remains asserted (from a different device), we will
end up back in the interrupt servicing code of course... but this is
cheaper than calling all the chained interrupts on a pin.

This does of course count on shared level interrupts being properly
sorted by IPL.

There have been some concerns about starvation of drivers which
incorrectly return 1. Those drivers should be hunted down so that
they return -1.

ok and help from various people. In snaps for about a week now.

Correct x86_atomic_*_ul() to operate on 64bit integers, eliminate
x86_atomic_*_l(), and update the callers of all of those to use the
underlying x86_atomic_*_u32() functions that they were expecting anyway.
Whether the x86_atomic_*_ul() macros should be eliminated has been deferred.

ok kettenis@, tedu@

Get rid of evcount's support for arranging counters in a tree
hierarchy. Everything attached to a single root node anyway, so at
best we had a bush.

"i think it is good" deraadt@

Set the user data of interrupt counters to the interrupt vector number used
for the interrupt. Makes the vmstat -i output similar to what we see on
i386.

ok deraadt@, krw@

Constify the what/name parameter of pci_intr_establish().

Tested by myself, sthen, oga, kettenis, and jasper.
Input from sthen and jasper.

ok kettenis

(Manpage follows shortly.)

revert all changes related to the mpsafe intr handler. i screwed up the commit
and even then it didn't work. we have higher standards than this.

the not MP version didn't compile, mglocker found

mpsafe intr_establish that doesn't get biglock, so that we may dream of the day when this is useful.
mostly macro magic that does nothing. only actually useful on amd64 for now, compliments of art.

Only call pic->pic_delroute() when disestablishing the last interrupt handler
for a pin. Resolves issues with azalia(4) devices that have no usable codecs
when they share an interrupt pin with other devices.

ok weingart@, krw@

replace M_WAITOK with M_NOWAIT

ok krw@ kettenis@

Rename IPL_SOFTSERIAL to IPL_SOFTTY and rename the associated symbols.
Remove IPL_SERIAL since it is unused.

ok krw@, weingart@, deraadt@

When allocating an interrupt 'slot', always check all slots to see if we
already allocated a slot for a particular apic pin.

ok weingart@

MALLOC/FREE -> malloc/free and M_ZERO changes

ok krw@

evcnt & friends were replaced by a proper interrupt counting mechanism 3 years ago; ok miod

ansi/deregister

Fix for ioapic irq routing.

From NetBSD, NetBSD PR 22728 for details

ok beck@ krw@
tested by beck@ krw@ marc@ and a few others on a number of amd64 systems.

Turn CPU_INFO_FOREACH into a real construct, like all queue(3) iterators,
instead of the contents of a for() loop. No functional change.
From the m88k SMP tree; ok art@ deraadt@

[complete diff this time]

In splraise, change an:
if (x > foo->bar)
foo->bar = x;
to:
foo->bar = MAX(x, foo->bar);

This forces gcc to generate much better code even though both
experessions are equivalent. Normally I wouldn't bother with
microoptimizations like this, but I needed some generated assembler
that uses cmov and splraise used so often..

ok toby@ (well, he ok:ed a diff that didn't use the MAX macro, but it's
the same code)

remove trailing newline in panic(9); ok millert@ and deraadt@

If irq-sharing happens over different ioapics, we must not overwrite setup info for one of them

Check for memory allocation failure before zeroing memory, not after;
from mpech@

From NetBSD:
date: 2004/06/28 09:13:11; author: fvdl; state: Exp; lines: +6 -5
Updaing ci_ilevel and testing ci_ipending must be done with all interrupts
off, or priority inversion can occur, which can lead to IPI deadlocks.
Leaves interrupts off for a bit longer, sadly, but with no noticeable
effects on the systems I tested on.

From YAMAMOTO Takashi.

Fixes the IPI rendezvous panics for me.

pass pin instead of level out to userland

Use new event counter API for interrupt counting on amd64. Based in part
on some changes in the i386 codebase.

deinline splraise, spllower and setsoftint.
Makes the kernel smaller and faster.
deraadt@ ok

SMP support. Big parts from NetBSD, but with some really serious debugging
done by me, niklas and others. Especially wrt. NXE support.

Still needs some polishing, especially in dmesg messages, but we're now
building kernel faster than ever.

an amd64 arch support.
hacked by art@ from netbsd sources and then later debugged
by me into the shape where it can host itself.
no bootloader yet as needs redoing from the
recent advanced i386 sources (anyone? ;)

pci_intr_establish_cpu() for establishing an interrupt no a specific cpu.

the cpu is specified by a struct cpu_info *, which should generally
come from an intrmap.

this is adapted from a diff that patrick@ sent round a few years
ago for a pci_intr_map_msix_cpuid, where you asked for an msi vector
on a specific cpu, and then called pci_intr_establish with the
handle you get. kettenis pointed out that it's hard on some archs
to carry cpu on a pci interrupt handle, so i tweaked it to turn it
into a pci_intr_establish_cpu instead.

jmatthew@ and i (but mostly jmatthew@ to be honest) have been
experimenting with this api on multiple archs and it is working out
well. i'm putting this diff in now on amd64 so people can kick the
tyres a bit.

tested with hacked up vmx(4), ix(4), and mcx(4)

make intr_barrier run sched_barrier on the cpu the interrupt pinned to.

intr_barrier passed NULL to sched_barrier before this, which ends
up being the primary cpu. that's been mostly right until this point,
but is set to change.

X86_IPI_NAMES's only use was #if 0'ed out; delete both

ok kettenis@ deraadt@

Always grab the KERNEL_LOCK() for !IPL_MPSAFE handlers.

Suggested by kettenis@, ok visa@

reduce the amount of includes in arch/amd64
ok mpi@ deraadt@

Get rid of read_psl() and write_psl() by replacing
read_psl() + disable_intr() with intr_disable(),
and write_psl() with intr_restore().

OK kettenis@

Setup Hyper-V hypercall page and an IDT vector.

ok mlarkin, kettenis, deraadt

Set up an IDT vector for Xen callbacks

This adds support for delivering the combined Xen interrupt
that later fans out into event port specific (device specific)
interrupts via an IDT of a guest system.

The Xen IDT vector is set to be the first of the IPL_NET group
and is implemented the same way LAPIC timer and IPIs are done.
The additional machinery is there to be able to mask it via
standard mechanisms (e.g. splnet).

Discussed with kettenis@, OK mlarkin, reyk

Introduce intr_barrier(4), an interface that guarantees that an interrupt
handler that was running has finished.

ok miod@, guenther@ (both for the equivalent sparc64 diff)

a few more free sizes. ok deraadt

fairly simple sizes for free(); ok tedu

Move grab/release of the kernel_lock for softintrs from the ASM stubs to
softintr_dispatch(). Delete traces of long superseded stats code.

ok beck@ mpi@ uebayasi@

Use "softintr_pic0" instead of "softintr_fakepic" when faking a
struct device so there is enough space in the buffer for a NUL and
the unit is included in the string.

discussed with deraadt and millert

Remove some includes include-what-you-use claims don't
have any direct symbols used. Tested for indirect use by compiling
amd64/i386/sparc64 kernels.

ok tedu@ deraadt@

when we're entering an interrupt handler, record its ipl so splassert
can check if we're entering code we think should only be used from
lower ipls.

modelled a bit on how sparc64 does things.

with help from and ok kettenis@

delete all the simplelocks. ok deraadt

remove uneeded proc.h includes
ok mpi@ kspillner@

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

It's been a quarter century: we can assume volatile is present with that name.

ok dlg@ mpi@ deraadt@

Since interrupts at IPL_AUDIO no longer grab the kernel lock, and nobody
is supposed to establish interrupts at IPL_VM, we only need to "flatten"
mutex ipls up until IPL_TTY. Add an assertion to enforce that IPL_AUDIO
interrupts are establish with the IPL_MPSAFE flag set. Should improve
latency of audio interrupts a tiny bit.

ok matthew@

Implement a mechanism to establish interrupt handlers that don't grab the
kernel lock upon entry through a new IPL_MPSAFE flag/level.

ok mikeb@, mpi@

Take the kernel lock and call the actual interrupt handler from a
single c function. This will hopefully make easier to stop taking
the kernel lock when running "mp safe" interrupt handlers.

help from ok kettenis

Remove excessive sys/cdefs.h inclusion
ok guenther millert kettenis

Make sure we don't accidentally use ISA or EISA interrupt mappings on PCI
busses.

tested by krw@

Add bounds checks for access to mp_busses.

Raise the number of interrupt sources per CPU from 32 to 64. This effectively
triples the number of interrupt vectors that can be handled by the primary
CPU. Important for MSI, but could also fix some issues with large machines
loaded with a lot of devices.

tested by many; ok deraadt@, marco@

Remove out of data XXX comment.

More than a decade ago, interrupt handlers on sparc started returning 0
(interrupt was not for me), 1 (positive interrupt was for me), or -1
(i am not sure...). We have continued with this practice in as many
drivers as possible, throughout the tree.

This makes some of the architectures use that information in their
interrupt handler calling code -- if 1 is returned (and we know
this specific machine does not have edge-shared interrupts), we
finish servicing other possible handlers on the same pin. If the
interrupt pin remains asserted (from a different device), we will
end up back in the interrupt servicing code of course... but this is
cheaper than calling all the chained interrupts on a pin.

This does of course count on shared level interrupts being properly
sorted by IPL.

There have been some concerns about starvation of drivers which
incorrectly return 1. Those drivers should be hunted down so that
they return -1.

ok and help from various people. In snaps for about a week now.

Correct x86_atomic_*_ul() to operate on 64bit integers, eliminate
x86_atomic_*_l(), and update the callers of all of those to use the
underlying x86_atomic_*_u32() functions that they were expecting anyway.
Whether the x86_atomic_*_ul() macros should be eliminated has been deferred.

ok kettenis@, tedu@

Get rid of evcount's support for arranging counters in a tree
hierarchy. Everything attached to a single root node anyway, so at
best we had a bush.

"i think it is good" deraadt@

Set the user data of interrupt counters to the interrupt vector number used
for the interrupt. Makes the vmstat -i output similar to what we see on
i386.

ok deraadt@, krw@

Constify the what/name parameter of pci_intr_establish().

Tested by myself, sthen, oga, kettenis, and jasper.
Input from sthen and jasper.

ok kettenis

(Manpage follows shortly.)

revert all changes related to the mpsafe intr handler. i screwed up the commit
and even then it didn't work. we have higher standards than this.

the not MP version didn't compile, mglocker found

mpsafe intr_establish that doesn't get biglock, so that we may dream of the day when this is useful.
mostly macro magic that does nothing. only actually useful on amd64 for now, compliments of art.

Only call pic->pic_delroute() when disestablishing the last interrupt handler
for a pin. Resolves issues with azalia(4) devices that have no usable codecs
when they share an interrupt pin with other devices.

ok weingart@, krw@

replace M_WAITOK with M_NOWAIT

ok krw@ kettenis@

Rename IPL_SOFTSERIAL to IPL_SOFTTY and rename the associated symbols.
Remove IPL_SERIAL since it is unused.

ok krw@, weingart@, deraadt@

When allocating an interrupt 'slot', always check all slots to see if we
already allocated a slot for a particular apic pin.

ok weingart@

MALLOC/FREE -> malloc/free and M_ZERO changes

ok krw@

evcnt & friends were replaced by a proper interrupt counting mechanism 3 years ago; ok miod

ansi/deregister

Fix for ioapic irq routing.

From NetBSD, NetBSD PR 22728 for details

ok beck@ krw@
tested by beck@ krw@ marc@ and a few others on a number of amd64 systems.

Turn CPU_INFO_FOREACH into a real construct, like all queue(3) iterators,
instead of the contents of a for() loop. No functional change.
From the m88k SMP tree; ok art@ deraadt@

[complete diff this time]

In splraise, change an:
if (x > foo->bar)
foo->bar = x;
to:
foo->bar = MAX(x, foo->bar);

This forces gcc to generate much better code even though both
experessions are equivalent. Normally I wouldn't bother with
microoptimizations like this, but I needed some generated assembler
that uses cmov and splraise used so often..

ok toby@ (well, he ok:ed a diff that didn't use the MAX macro, but it's
the same code)

remove trailing newline in panic(9); ok millert@ and deraadt@

If irq-sharing happens over different ioapics, we must not overwrite setup info for one of them

Check for memory allocation failure before zeroing memory, not after;
from mpech@

From NetBSD:
date: 2004/06/28 09:13:11; author: fvdl; state: Exp; lines: +6 -5
Updaing ci_ilevel and testing ci_ipending must be done with all interrupts
off, or priority inversion can occur, which can lead to IPI deadlocks.
Leaves interrupts off for a bit longer, sadly, but with no noticeable
effects on the systems I tested on.

From YAMAMOTO Takashi.

Fixes the IPI rendezvous panics for me.

pass pin instead of level out to userland

Use new event counter API for interrupt counting on amd64. Based in part
on some changes in the i386 codebase.

deinline splraise, spllower and setsoftint.
Makes the kernel smaller and faster.
deraadt@ ok

SMP support. Big parts from NetBSD, but with some really serious debugging
done by me, niklas and others. Especially wrt. NXE support.

Still needs some polishing, especially in dmesg messages, but we're now
building kernel faster than ever.

an amd64 arch support.
hacked by art@ from netbsd sources and then later debugged
by me into the shape where it can host itself.
no bootloader yet as needs redoing from the
recent advanced i386 sources (anyone? ;)

X86_IPI_NAMES's only use was #if 0'ed out; delete both

ok kettenis@ deraadt@

Always grab the KERNEL_LOCK() for !IPL_MPSAFE handlers.

Suggested by kettenis@, ok visa@

reduce the amount of includes in arch/amd64
ok mpi@ deraadt@

Get rid of read_psl() and write_psl() by replacing
read_psl() + disable_intr() with intr_disable(),
and write_psl() with intr_restore().

OK kettenis@

Setup Hyper-V hypercall page and an IDT vector.

ok mlarkin, kettenis, deraadt

Set up an IDT vector for Xen callbacks

This adds support for delivering the combined Xen interrupt
that later fans out into event port specific (device specific)
interrupts via an IDT of a guest system.

The Xen IDT vector is set to be the first of the IPL_NET group
and is implemented the same way LAPIC timer and IPIs are done.
The additional machinery is there to be able to mask it via
standard mechanisms (e.g. splnet).

Discussed with kettenis@, OK mlarkin, reyk

Introduce intr_barrier(4), an interface that guarantees that an interrupt
handler that was running has finished.

ok miod@, guenther@ (both for the equivalent sparc64 diff)

a few more free sizes. ok deraadt

fairly simple sizes for free(); ok tedu

Move grab/release of the kernel_lock for softintrs from the ASM stubs to
softintr_dispatch(). Delete traces of long superseded stats code.

ok beck@ mpi@ uebayasi@

Use "softintr_pic0" instead of "softintr_fakepic" when faking a
struct device so there is enough space in the buffer for a NUL and
the unit is included in the string.

discussed with deraadt and millert

Remove some includes include-what-you-use claims don't
have any direct symbols used. Tested for indirect use by compiling
amd64/i386/sparc64 kernels.

ok tedu@ deraadt@

when we're entering an interrupt handler, record its ipl so splassert
can check if we're entering code we think should only be used from
lower ipls.

modelled a bit on how sparc64 does things.

with help from and ok kettenis@

delete all the simplelocks. ok deraadt

remove uneeded proc.h includes
ok mpi@ kspillner@

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

It's been a quarter century: we can assume volatile is present with that name.

ok dlg@ mpi@ deraadt@

Since interrupts at IPL_AUDIO no longer grab the kernel lock, and nobody
is supposed to establish interrupts at IPL_VM, we only need to "flatten"
mutex ipls up until IPL_TTY. Add an assertion to enforce that IPL_AUDIO
interrupts are establish with the IPL_MPSAFE flag set. Should improve
latency of audio interrupts a tiny bit.

ok matthew@

Implement a mechanism to establish interrupt handlers that don't grab the
kernel lock upon entry through a new IPL_MPSAFE flag/level.

ok mikeb@, mpi@

Take the kernel lock and call the actual interrupt handler from a
single c function. This will hopefully make easier to stop taking
the kernel lock when running "mp safe" interrupt handlers.

help from ok kettenis

Remove excessive sys/cdefs.h inclusion
ok guenther millert kettenis

Make sure we don't accidentally use ISA or EISA interrupt mappings on PCI
busses.

tested by krw@

Add bounds checks for access to mp_busses.

Raise the number of interrupt sources per CPU from 32 to 64. This effectively
triples the number of interrupt vectors that can be handled by the primary
CPU. Important for MSI, but could also fix some issues with large machines
loaded with a lot of devices.

tested by many; ok deraadt@, marco@

Remove out of data XXX comment.

More than a decade ago, interrupt handlers on sparc started returning 0
(interrupt was not for me), 1 (positive interrupt was for me), or -1
(i am not sure...). We have continued with this practice in as many
drivers as possible, throughout the tree.

This makes some of the architectures use that information in their
interrupt handler calling code -- if 1 is returned (and we know
this specific machine does not have edge-shared interrupts), we
finish servicing other possible handlers on the same pin. If the
interrupt pin remains asserted (from a different device), we will
end up back in the interrupt servicing code of course... but this is
cheaper than calling all the chained interrupts on a pin.

This does of course count on shared level interrupts being properly
sorted by IPL.

There have been some concerns about starvation of drivers which
incorrectly return 1. Those drivers should be hunted down so that
they return -1.

ok and help from various people. In snaps for about a week now.

Correct x86_atomic_*_ul() to operate on 64bit integers, eliminate
x86_atomic_*_l(), and update the callers of all of those to use the
underlying x86_atomic_*_u32() functions that they were expecting anyway.
Whether the x86_atomic_*_ul() macros should be eliminated has been deferred.

ok kettenis@, tedu@

Get rid of evcount's support for arranging counters in a tree
hierarchy. Everything attached to a single root node anyway, so at
best we had a bush.

"i think it is good" deraadt@

Set the user data of interrupt counters to the interrupt vector number used
for the interrupt. Makes the vmstat -i output similar to what we see on
i386.

ok deraadt@, krw@

Constify the what/name parameter of pci_intr_establish().

Tested by myself, sthen, oga, kettenis, and jasper.
Input from sthen and jasper.

ok kettenis

(Manpage follows shortly.)

revert all changes related to the mpsafe intr handler. i screwed up the commit
and even then it didn't work. we have higher standards than this.

the not MP version didn't compile, mglocker found

mpsafe intr_establish that doesn't get biglock, so that we may dream of the day when this is useful.
mostly macro magic that does nothing. only actually useful on amd64 for now, compliments of art.

Only call pic->pic_delroute() when disestablishing the last interrupt handler
for a pin. Resolves issues with azalia(4) devices that have no usable codecs
when they share an interrupt pin with other devices.

ok weingart@, krw@

replace M_WAITOK with M_NOWAIT

ok krw@ kettenis@

Rename IPL_SOFTSERIAL to IPL_SOFTTY and rename the associated symbols.
Remove IPL_SERIAL since it is unused.

ok krw@, weingart@, deraadt@

When allocating an interrupt 'slot', always check all slots to see if we
already allocated a slot for a particular apic pin.

ok weingart@

MALLOC/FREE -> malloc/free and M_ZERO changes

ok krw@

evcnt & friends were replaced by a proper interrupt counting mechanism 3 years ago; ok miod

ansi/deregister

Fix for ioapic irq routing.

From NetBSD, NetBSD PR 22728 for details

ok beck@ krw@
tested by beck@ krw@ marc@ and a few others on a number of amd64 systems.

Turn CPU_INFO_FOREACH into a real construct, like all queue(3) iterators,
instead of the contents of a for() loop. No functional change.
From the m88k SMP tree; ok art@ deraadt@

[complete diff this time]

In splraise, change an:
if (x > foo->bar)
foo->bar = x;
to:
foo->bar = MAX(x, foo->bar);

This forces gcc to generate much better code even though both
experessions are equivalent. Normally I wouldn't bother with
microoptimizations like this, but I needed some generated assembler
that uses cmov and splraise used so often..

ok toby@ (well, he ok:ed a diff that didn't use the MAX macro, but it's
the same code)

remove trailing newline in panic(9); ok millert@ and deraadt@

If irq-sharing happens over different ioapics, we must not overwrite setup info for one of them

Check for memory allocation failure before zeroing memory, not after;
from mpech@

From NetBSD:
date: 2004/06/28 09:13:11; author: fvdl; state: Exp; lines: +6 -5
Updaing ci_ilevel and testing ci_ipending must be done with all interrupts
off, or priority inversion can occur, which can lead to IPI deadlocks.
Leaves interrupts off for a bit longer, sadly, but with no noticeable
effects on the systems I tested on.

From YAMAMOTO Takashi.

Fixes the IPI rendezvous panics for me.

pass pin instead of level out to userland

Use new event counter API for interrupt counting on amd64. Based in part
on some changes in the i386 codebase.

deinline splraise, spllower and setsoftint.
Makes the kernel smaller and faster.
deraadt@ ok

SMP support. Big parts from NetBSD, but with some really serious debugging
done by me, niklas and others. Especially wrt. NXE support.

Still needs some polishing, especially in dmesg messages, but we're now
building kernel faster than ever.