arm/pic: Sprinkle dtrace probes on interrupt handlers like x86.

arm/pic: Use a local variable to reduce #ifdef scope.

Avoids straddling a conditional this way.

arm/pic: Mark definitions static to match declarations.

arm/pic: Assert ci_cpl doesn't lower across interrupt handler.

Catch up with the per lwp astpending (rather than per cpu) flag in the
preemption code. NFC as preemption (still) isn't enabled.

pic: Update ci_cpl in pic_set_priority callback.

Not all ICs need interrupts disabled to update the priority. DAIF accesses
are not cheap, so push the update of ci_cpl from pic_set_priority to the
IC's pic_set_priority callback, and let the IC driver determine whether
or not it needs interrupts disabled.

arm: No #if DIAGNOSTIC needed any more.

Compiler can (and gcc does) flush call to strlen when the result is
unused, so no performance impact here.

Fix a bug where pic_establish_intr would fail to call pic_establish_irq
if a free pic__iplsources slot was found, i.e. an interrupt handler at
the same ipl had been disestablished previously.

G/C pic_iplsource

KNF

Only perform pic_unblock_percpu if all of the following are true
- mp_online, i.e. APs are running.
- is_mpsafe, i.e. the interrupt handler is MP safe
- is_percpu, i.e. the interrupt actually requires it!

The last one (is_percpu) is true for GIC PPI+SGI only.

Assert we can sleep in pic_add

Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.

The AP initialisation code in cpu_init_secondary_processor will read and
initialise the required system registers and state for the BP to attach
and report.

Rework the interrupt handler code for this new sequence. Thankfully,
this removes a bunch of code for bcm2836mp.

The VFP detection handler on <= armv7 relies on the global undefined
handler being in place until the BP attaches vfp. That is, after the
APs have been spun up.

gicv3_its.c has a serialisation issue which is protected against in
the gicv3_its_cpu_init, which is called from cpu_hatch, with a spin
lock. The serialisation issue needs addressing more completely.

Tested on RPI3, Apple M1, QEMU, and lx2k

Fixes PR port-arm/56264:
diagnostic assertion "l->l_stat == LSONPROC" failed on RPI3

If an SGI or PPI is established after interrupts are enabled, make sure
we unblock the source on _all_ CPUs and not just the CPU that is
establishing the interrupt.

Re-apply

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

There is(/was) no need to use atomic operations on the percpu / cpu_info
members, so don't.

Finally removng the use of percpu should help avoid problems with "late"
attaching cpus.

Revert recent pic optimizations until I have more time to work on this.

branches: 1.69.2;
Inline pic_set_priority and use cpu_dosoftints_ci when available.

Fixup with __HAVE_PREEMPTION code which is currently unused

Inline "pic_do_pending_ints" in splx and check ci_pending_ipls to optimize
the common case (hw priority, no cascaded interrupts pending).

This also removes the need for the "pic_pending_used" flag, and should fix
booting on Raspberry Pi 3.

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

pic: avoid pic_do_pending_ints if pic_mark_pending_* has never been called

splx: use pic_set_priority_psw in interrupts disabled case to skip a few
more daif accesses.

pic: reduce the number of daif accesses in pic_do_pending_ints

The caller has already provided daif state. No need to keep updating
daif via pic_set_priority if it's already in the state we need (interrupts
disabled).

Use ENABLE_INTERRUPT() / DISABLE_INTERRUPT() instead of cpsie()/cpsid() in
places where we don't care about the cpsie() return value.

branches: 1.61.2;
intr_ipi_send: assert that kcp is either NULL or contains exactly one CPU

Improve a comment

KNF

KASSERT -> KASSERTMSG

Trailing whitespace

KNF

Reduce some ifdefs.

pic_pending_zero is unnecessary; percpu_alloc already zeroes.

Switch arm pic allocation and initialization to percpu_create.

branches: 1.52.2;
Update pic_add to allocate and return an irqbase if passed
PIC_IRQBASE_ALLOC.

Make pic_sourcebase static

Add reference counts to intr_mask/intr_unmask as calls can be nested, spotted by thorpej

Implement acpi_md_intr_mask and acpi_md_intr_unmask

Add intr_establish_xname support to arm and expose it to intrctl

Fix intrctl for pics with non-0 irqbase

Add support for intrctl(8).

Implement intr_string(9)

Add support for setting and getting interrupt affinity.

fix false positive KASSERT.
the case that secondary processors failed to hatch, and not attached.

when "cpu1 at cpus0: disabled (unresponsive)"

branches: 1.42.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.41.2;
Trailing whitespace

spaces to TAB

decouple pic_ipi_kpreempt() from DDB

The assertion at the bottom of intr_ipi_send is invalid for UP configs; add
a text for arm_cpu_max == 1

remove checks for failure after memory allocation calls that cannot fail:

kmem_alloc() with KM_SLEEP
kmem_zalloc() with KM_SLEEP
percpu_alloc()
pserialize_create()
psref_class_create()

all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.

unblock interrupts also when reusing a previously freed irq slot.

Serialise work in pic_add.

Add separate IPI routines for IPI_AST and IPI_KPREEMPT.

__HAVE_PREEEMPTION -> __HAVE_PREEMPTION

Don't send IPIs to ourselves if sending to everyone.

Don't need pic_ipi_sender anymore.

Use right kcpuset call.

Adapt pic to deal with the BCM2836 interrupts.
Move pic_pending_pics, pic_pending_ipls, and pic_blocked_irqs into a
structure and make then per-cpu. There is no global interrupt state anymore.

Add __HAVE_PIC_HAVE_PENDING_INTRS and define it if __HAVE_PIC_SET_PRIORITY
is undefined (also define in mvsoc_intr.h since their use of the latter is
peculiar). This new define controls whether the pending interrupt logic is
compiled. The GIC doesn't use pending interrupts since it uses the priority
level on the GIC to control delivery of interrupts, thus there can never
be a pending interrupt. The kernel shrinks about 4KB with the removal of
the pending interupt support,

Don't clear CI_ASTPENDING in exception return, do it in ast() instead.
Add basic support for __HAVE_PREEMPTION.
Use atomic ops for ci_astpending if __HAVE_PREEMPTION is defined.
Use kpreempt_disable/kpreempt_enable

Add aarch64 support (mostly remapping I32_bit and F32_bit onto aarch64 bits).

branches: 1.25.2;
Include opt_multiprocessor.h

In the MULTIPROCESSOR case mark all interrupt handlers not at IPL_VM as
mpsafe, so the kernel_lock is only taken at IPL_VM as per spl(9).

Comment out a printf that was confusing entering ddb output.

branches: 1.22.2;
Implement MI IPI interface with cross-call support.

branches: 1.21.2;
Support MPSAFE interrupts.

Add a mpsafe flag to the intrsource

Mark a diagnostic-only variable

Mark rv as unused (for now)

Use KASSERTMSG

Whitespace

branches: 1.15.2; 1.15.6;
Fix a bug that incorrect arg is passed to pic_block_irqs() on disestablish.
Fixes PR#45371 by Yuichiro Goto.

branches: 1.14.2;
Add #ifdef DIAGNOSTIC / #endif

Add PIC hooks for MP and for the ARM Generic Interrupt Controller.

Fix botched change to use right value for ipl. Thanks jak

Add hooks for __HAVE_PIC_SET_PRIORITY which allows updating of a hardware
(PIC) priority based on current IPL.

Convert a KASSERT to a KASSERTMSG

Add a KASSERT

branches: 1.8.4; 1.8.10; 1.8.12;
split arch/arm/pic/pic.c so that we can implement more efficient version of splfoo() while using pic framework.

Increment ci->ci_intr_depth for the duration of pic_do_pending_ints().
Fixes locking catastrophy commonly noticed with mvsata(4) on evbarm.

branches: 1.6.2; 1.6.4;
Fix lost interrupt. (2/2)
+ It is likely to lose sight of interrupt when the interrupt of irq_base
that is smaller than the same at the level is generated if PIC_MAXSOURCES
is 33 or more.

Fix lost interrupt. (1/2)
+ Change blocked_irqs that dispach if change pending_irqs in the loop.

branches: 1.4.4; 1.4.6;
Use atomic ops to manipulate the bitmasks.

branches: 1.3.8; 1.3.16;
Remove clause 3 and 4 from TNF licenses

Merge kernel changes in matt-armv6 to HEAD.

branches: 1.1.2; 1.1.20; 1.1.22; 1.1.24;
file pic.c was initially added on branch matt-armv6.

arm/pic: Mark definitions static to match declarations.

arm/pic: Assert ci_cpl doesn't lower across interrupt handler.

Catch up with the per lwp astpending (rather than per cpu) flag in the
preemption code. NFC as preemption (still) isn't enabled.

pic: Update ci_cpl in pic_set_priority callback.

Not all ICs need interrupts disabled to update the priority. DAIF accesses
are not cheap, so push the update of ci_cpl from pic_set_priority to the
IC's pic_set_priority callback, and let the IC driver determine whether
or not it needs interrupts disabled.

arm: No #if DIAGNOSTIC needed any more.

Compiler can (and gcc does) flush call to strlen when the result is
unused, so no performance impact here.

Fix a bug where pic_establish_intr would fail to call pic_establish_irq
if a free pic__iplsources slot was found, i.e. an interrupt handler at
the same ipl had been disestablished previously.

G/C pic_iplsource

KNF

Only perform pic_unblock_percpu if all of the following are true
- mp_online, i.e. APs are running.
- is_mpsafe, i.e. the interrupt handler is MP safe
- is_percpu, i.e. the interrupt actually requires it!

The last one (is_percpu) is true for GIC PPI+SGI only.

Assert we can sleep in pic_add

Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.

The AP initialisation code in cpu_init_secondary_processor will read and
initialise the required system registers and state for the BP to attach
and report.

Rework the interrupt handler code for this new sequence. Thankfully,
this removes a bunch of code for bcm2836mp.

The VFP detection handler on <= armv7 relies on the global undefined
handler being in place until the BP attaches vfp. That is, after the
APs have been spun up.

gicv3_its.c has a serialisation issue which is protected against in
the gicv3_its_cpu_init, which is called from cpu_hatch, with a spin
lock. The serialisation issue needs addressing more completely.

Tested on RPI3, Apple M1, QEMU, and lx2k

Fixes PR port-arm/56264:
diagnostic assertion "l->l_stat == LSONPROC" failed on RPI3

If an SGI or PPI is established after interrupts are enabled, make sure
we unblock the source on _all_ CPUs and not just the CPU that is
establishing the interrupt.

Re-apply

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

There is(/was) no need to use atomic operations on the percpu / cpu_info
members, so don't.

Finally removng the use of percpu should help avoid problems with "late"
attaching cpus.

Revert recent pic optimizations until I have more time to work on this.

branches: 1.69.2;
Inline pic_set_priority and use cpu_dosoftints_ci when available.

Fixup with __HAVE_PREEMPTION code which is currently unused

Inline "pic_do_pending_ints" in splx and check ci_pending_ipls to optimize
the common case (hw priority, no cascaded interrupts pending).

This also removes the need for the "pic_pending_used" flag, and should fix
booting on Raspberry Pi 3.

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

pic: avoid pic_do_pending_ints if pic_mark_pending_* has never been called

splx: use pic_set_priority_psw in interrupts disabled case to skip a few
more daif accesses.

pic: reduce the number of daif accesses in pic_do_pending_ints

The caller has already provided daif state. No need to keep updating
daif via pic_set_priority if it's already in the state we need (interrupts
disabled).

Use ENABLE_INTERRUPT() / DISABLE_INTERRUPT() instead of cpsie()/cpsid() in
places where we don't care about the cpsie() return value.

branches: 1.61.2;
intr_ipi_send: assert that kcp is either NULL or contains exactly one CPU

Improve a comment

KNF

KASSERT -> KASSERTMSG

Trailing whitespace

KNF

Reduce some ifdefs.

pic_pending_zero is unnecessary; percpu_alloc already zeroes.

Switch arm pic allocation and initialization to percpu_create.

branches: 1.52.2;
Update pic_add to allocate and return an irqbase if passed
PIC_IRQBASE_ALLOC.

Make pic_sourcebase static

Add reference counts to intr_mask/intr_unmask as calls can be nested, spotted by thorpej

Implement acpi_md_intr_mask and acpi_md_intr_unmask

Add intr_establish_xname support to arm and expose it to intrctl

Fix intrctl for pics with non-0 irqbase

Add support for intrctl(8).

Implement intr_string(9)

Add support for setting and getting interrupt affinity.

fix false positive KASSERT.
the case that secondary processors failed to hatch, and not attached.

when "cpu1 at cpus0: disabled (unresponsive)"

branches: 1.42.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.41.2;
Trailing whitespace

spaces to TAB

decouple pic_ipi_kpreempt() from DDB

The assertion at the bottom of intr_ipi_send is invalid for UP configs; add
a text for arm_cpu_max == 1

remove checks for failure after memory allocation calls that cannot fail:

kmem_alloc() with KM_SLEEP
kmem_zalloc() with KM_SLEEP
percpu_alloc()
pserialize_create()
psref_class_create()

all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.

unblock interrupts also when reusing a previously freed irq slot.

Serialise work in pic_add.

Add separate IPI routines for IPI_AST and IPI_KPREEMPT.

__HAVE_PREEEMPTION -> __HAVE_PREEMPTION

Don't send IPIs to ourselves if sending to everyone.

Don't need pic_ipi_sender anymore.

Use right kcpuset call.

Adapt pic to deal with the BCM2836 interrupts.
Move pic_pending_pics, pic_pending_ipls, and pic_blocked_irqs into a
structure and make then per-cpu. There is no global interrupt state anymore.

Add __HAVE_PIC_HAVE_PENDING_INTRS and define it if __HAVE_PIC_SET_PRIORITY
is undefined (also define in mvsoc_intr.h since their use of the latter is
peculiar). This new define controls whether the pending interrupt logic is
compiled. The GIC doesn't use pending interrupts since it uses the priority
level on the GIC to control delivery of interrupts, thus there can never
be a pending interrupt. The kernel shrinks about 4KB with the removal of
the pending interupt support,

Don't clear CI_ASTPENDING in exception return, do it in ast() instead.
Add basic support for __HAVE_PREEMPTION.
Use atomic ops for ci_astpending if __HAVE_PREEMPTION is defined.
Use kpreempt_disable/kpreempt_enable

Add aarch64 support (mostly remapping I32_bit and F32_bit onto aarch64 bits).

branches: 1.25.2;
Include opt_multiprocessor.h

In the MULTIPROCESSOR case mark all interrupt handlers not at IPL_VM as
mpsafe, so the kernel_lock is only taken at IPL_VM as per spl(9).

Comment out a printf that was confusing entering ddb output.

branches: 1.22.2;
Implement MI IPI interface with cross-call support.

branches: 1.21.2;
Support MPSAFE interrupts.

Add a mpsafe flag to the intrsource

Mark a diagnostic-only variable

Mark rv as unused (for now)

Use KASSERTMSG

Whitespace

branches: 1.15.2; 1.15.6;
Fix a bug that incorrect arg is passed to pic_block_irqs() on disestablish.
Fixes PR#45371 by Yuichiro Goto.

branches: 1.14.2;
Add #ifdef DIAGNOSTIC / #endif

Add PIC hooks for MP and for the ARM Generic Interrupt Controller.

Fix botched change to use right value for ipl. Thanks jak

Add hooks for __HAVE_PIC_SET_PRIORITY which allows updating of a hardware
(PIC) priority based on current IPL.

Convert a KASSERT to a KASSERTMSG

Add a KASSERT

branches: 1.8.4; 1.8.10; 1.8.12;
split arch/arm/pic/pic.c so that we can implement more efficient version of splfoo() while using pic framework.

Increment ci->ci_intr_depth for the duration of pic_do_pending_ints().
Fixes locking catastrophy commonly noticed with mvsata(4) on evbarm.

branches: 1.6.2; 1.6.4;
Fix lost interrupt. (2/2)
+ It is likely to lose sight of interrupt when the interrupt of irq_base
that is smaller than the same at the level is generated if PIC_MAXSOURCES
is 33 or more.

Fix lost interrupt. (1/2)
+ Change blocked_irqs that dispach if change pending_irqs in the loop.

branches: 1.4.4; 1.4.6;
Use atomic ops to manipulate the bitmasks.

branches: 1.3.8; 1.3.16;
Remove clause 3 and 4 from TNF licenses

Merge kernel changes in matt-armv6 to HEAD.

branches: 1.1.2; 1.1.20; 1.1.22; 1.1.24;
file pic.c was initially added on branch matt-armv6.

pic: Update ci_cpl in pic_set_priority callback.

Not all ICs need interrupts disabled to update the priority. DAIF accesses
are not cheap, so push the update of ci_cpl from pic_set_priority to the
IC's pic_set_priority callback, and let the IC driver determine whether
or not it needs interrupts disabled.

arm: No #if DIAGNOSTIC needed any more.

Compiler can (and gcc does) flush call to strlen when the result is
unused, so no performance impact here.

Fix a bug where pic_establish_intr would fail to call pic_establish_irq
if a free pic__iplsources slot was found, i.e. an interrupt handler at
the same ipl had been disestablished previously.

G/C pic_iplsource

KNF

Only perform pic_unblock_percpu if all of the following are true
- mp_online, i.e. APs are running.
- is_mpsafe, i.e. the interrupt handler is MP safe
- is_percpu, i.e. the interrupt actually requires it!

The last one (is_percpu) is true for GIC PPI+SGI only.

Assert we can sleep in pic_add

Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.

The AP initialisation code in cpu_init_secondary_processor will read and
initialise the required system registers and state for the BP to attach
and report.

Rework the interrupt handler code for this new sequence. Thankfully,
this removes a bunch of code for bcm2836mp.

The VFP detection handler on <= armv7 relies on the global undefined
handler being in place until the BP attaches vfp. That is, after the
APs have been spun up.

gicv3_its.c has a serialisation issue which is protected against in
the gicv3_its_cpu_init, which is called from cpu_hatch, with a spin
lock. The serialisation issue needs addressing more completely.

Tested on RPI3, Apple M1, QEMU, and lx2k

Fixes PR port-arm/56264:
diagnostic assertion "l->l_stat == LSONPROC" failed on RPI3

If an SGI or PPI is established after interrupts are enabled, make sure
we unblock the source on _all_ CPUs and not just the CPU that is
establishing the interrupt.

Re-apply

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

There is(/was) no need to use atomic operations on the percpu / cpu_info
members, so don't.

Finally removng the use of percpu should help avoid problems with "late"
attaching cpus.

Revert recent pic optimizations until I have more time to work on this.

branches: 1.69.2;
Inline pic_set_priority and use cpu_dosoftints_ci when available.

Fixup with __HAVE_PREEMPTION code which is currently unused

Inline "pic_do_pending_ints" in splx and check ci_pending_ipls to optimize
the common case (hw priority, no cascaded interrupts pending).

This also removes the need for the "pic_pending_used" flag, and should fix
booting on Raspberry Pi 3.

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

pic: avoid pic_do_pending_ints if pic_mark_pending_* has never been called

splx: use pic_set_priority_psw in interrupts disabled case to skip a few
more daif accesses.

pic: reduce the number of daif accesses in pic_do_pending_ints

The caller has already provided daif state. No need to keep updating
daif via pic_set_priority if it's already in the state we need (interrupts
disabled).

Use ENABLE_INTERRUPT() / DISABLE_INTERRUPT() instead of cpsie()/cpsid() in
places where we don't care about the cpsie() return value.

branches: 1.61.2;
intr_ipi_send: assert that kcp is either NULL or contains exactly one CPU

Improve a comment

KNF

KASSERT -> KASSERTMSG

Trailing whitespace

KNF

Reduce some ifdefs.

pic_pending_zero is unnecessary; percpu_alloc already zeroes.

Switch arm pic allocation and initialization to percpu_create.

branches: 1.52.2;
Update pic_add to allocate and return an irqbase if passed
PIC_IRQBASE_ALLOC.

Make pic_sourcebase static

Add reference counts to intr_mask/intr_unmask as calls can be nested, spotted by thorpej

Implement acpi_md_intr_mask and acpi_md_intr_unmask

Add intr_establish_xname support to arm and expose it to intrctl

Fix intrctl for pics with non-0 irqbase

Add support for intrctl(8).

Implement intr_string(9)

Add support for setting and getting interrupt affinity.

fix false positive KASSERT.
the case that secondary processors failed to hatch, and not attached.

when "cpu1 at cpus0: disabled (unresponsive)"

branches: 1.42.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.41.2;
Trailing whitespace

spaces to TAB

decouple pic_ipi_kpreempt() from DDB

The assertion at the bottom of intr_ipi_send is invalid for UP configs; add
a text for arm_cpu_max == 1

remove checks for failure after memory allocation calls that cannot fail:

kmem_alloc() with KM_SLEEP
kmem_zalloc() with KM_SLEEP
percpu_alloc()
pserialize_create()
psref_class_create()

all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.

unblock interrupts also when reusing a previously freed irq slot.

Serialise work in pic_add.

Add separate IPI routines for IPI_AST and IPI_KPREEMPT.

__HAVE_PREEEMPTION -> __HAVE_PREEMPTION

Don't send IPIs to ourselves if sending to everyone.

Don't need pic_ipi_sender anymore.

Use right kcpuset call.

Adapt pic to deal with the BCM2836 interrupts.
Move pic_pending_pics, pic_pending_ipls, and pic_blocked_irqs into a
structure and make then per-cpu. There is no global interrupt state anymore.

Add __HAVE_PIC_HAVE_PENDING_INTRS and define it if __HAVE_PIC_SET_PRIORITY
is undefined (also define in mvsoc_intr.h since their use of the latter is
peculiar). This new define controls whether the pending interrupt logic is
compiled. The GIC doesn't use pending interrupts since it uses the priority
level on the GIC to control delivery of interrupts, thus there can never
be a pending interrupt. The kernel shrinks about 4KB with the removal of
the pending interupt support,

Don't clear CI_ASTPENDING in exception return, do it in ast() instead.
Add basic support for __HAVE_PREEMPTION.
Use atomic ops for ci_astpending if __HAVE_PREEMPTION is defined.
Use kpreempt_disable/kpreempt_enable

Add aarch64 support (mostly remapping I32_bit and F32_bit onto aarch64 bits).

branches: 1.25.2;
Include opt_multiprocessor.h

In the MULTIPROCESSOR case mark all interrupt handlers not at IPL_VM as
mpsafe, so the kernel_lock is only taken at IPL_VM as per spl(9).

Comment out a printf that was confusing entering ddb output.

branches: 1.22.2;
Implement MI IPI interface with cross-call support.

branches: 1.21.2;
Support MPSAFE interrupts.

Add a mpsafe flag to the intrsource

Mark a diagnostic-only variable

Mark rv as unused (for now)

Use KASSERTMSG

Whitespace

branches: 1.15.2; 1.15.6;
Fix a bug that incorrect arg is passed to pic_block_irqs() on disestablish.
Fixes PR#45371 by Yuichiro Goto.

branches: 1.14.2;
Add #ifdef DIAGNOSTIC / #endif

Add PIC hooks for MP and for the ARM Generic Interrupt Controller.

Fix botched change to use right value for ipl. Thanks jak

Add hooks for __HAVE_PIC_SET_PRIORITY which allows updating of a hardware
(PIC) priority based on current IPL.

Convert a KASSERT to a KASSERTMSG

Add a KASSERT

branches: 1.8.4; 1.8.10; 1.8.12;
split arch/arm/pic/pic.c so that we can implement more efficient version of splfoo() while using pic framework.

Increment ci->ci_intr_depth for the duration of pic_do_pending_ints().
Fixes locking catastrophy commonly noticed with mvsata(4) on evbarm.

branches: 1.6.2; 1.6.4;
Fix lost interrupt. (2/2)
+ It is likely to lose sight of interrupt when the interrupt of irq_base
that is smaller than the same at the level is generated if PIC_MAXSOURCES
is 33 or more.

Fix lost interrupt. (1/2)
+ Change blocked_irqs that dispach if change pending_irqs in the loop.

branches: 1.4.4; 1.4.6;
Use atomic ops to manipulate the bitmasks.

branches: 1.3.8; 1.3.16;
Remove clause 3 and 4 from TNF licenses

Merge kernel changes in matt-armv6 to HEAD.

branches: 1.1.2; 1.1.20; 1.1.22; 1.1.24;
file pic.c was initially added on branch matt-armv6.

arm: No #if DIAGNOSTIC needed any more.

Compiler can (and gcc does) flush call to strlen when the result is
unused, so no performance impact here.

Fix a bug where pic_establish_intr would fail to call pic_establish_irq
if a free pic__iplsources slot was found, i.e. an interrupt handler at
the same ipl had been disestablished previously.

G/C pic_iplsource

KNF

Only perform pic_unblock_percpu if all of the following are true
- mp_online, i.e. APs are running.
- is_mpsafe, i.e. the interrupt handler is MP safe
- is_percpu, i.e. the interrupt actually requires it!

The last one (is_percpu) is true for GIC PPI+SGI only.

Assert we can sleep in pic_add

Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.

The AP initialisation code in cpu_init_secondary_processor will read and
initialise the required system registers and state for the BP to attach
and report.

Rework the interrupt handler code for this new sequence. Thankfully,
this removes a bunch of code for bcm2836mp.

The VFP detection handler on <= armv7 relies on the global undefined
handler being in place until the BP attaches vfp. That is, after the
APs have been spun up.

gicv3_its.c has a serialisation issue which is protected against in
the gicv3_its_cpu_init, which is called from cpu_hatch, with a spin
lock. The serialisation issue needs addressing more completely.

Tested on RPI3, Apple M1, QEMU, and lx2k

Fixes PR port-arm/56264:
diagnostic assertion "l->l_stat == LSONPROC" failed on RPI3

If an SGI or PPI is established after interrupts are enabled, make sure
we unblock the source on _all_ CPUs and not just the CPU that is
establishing the interrupt.

Re-apply

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

There is(/was) no need to use atomic operations on the percpu / cpu_info
members, so don't.

Finally removng the use of percpu should help avoid problems with "late"
attaching cpus.

Revert recent pic optimizations until I have more time to work on this.

branches: 1.69.2;
Inline pic_set_priority and use cpu_dosoftints_ci when available.

Fixup with __HAVE_PREEMPTION code which is currently unused

Inline "pic_do_pending_ints" in splx and check ci_pending_ipls to optimize
the common case (hw priority, no cascaded interrupts pending).

This also removes the need for the "pic_pending_used" flag, and should fix
booting on Raspberry Pi 3.

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

pic: avoid pic_do_pending_ints if pic_mark_pending_* has never been called

splx: use pic_set_priority_psw in interrupts disabled case to skip a few
more daif accesses.

pic: reduce the number of daif accesses in pic_do_pending_ints

The caller has already provided daif state. No need to keep updating
daif via pic_set_priority if it's already in the state we need (interrupts
disabled).

Use ENABLE_INTERRUPT() / DISABLE_INTERRUPT() instead of cpsie()/cpsid() in
places where we don't care about the cpsie() return value.

branches: 1.61.2;
intr_ipi_send: assert that kcp is either NULL or contains exactly one CPU

Improve a comment

KNF

KASSERT -> KASSERTMSG

Trailing whitespace

KNF

Reduce some ifdefs.

pic_pending_zero is unnecessary; percpu_alloc already zeroes.

Switch arm pic allocation and initialization to percpu_create.

branches: 1.52.2;
Update pic_add to allocate and return an irqbase if passed
PIC_IRQBASE_ALLOC.

Make pic_sourcebase static

Add reference counts to intr_mask/intr_unmask as calls can be nested, spotted by thorpej

Implement acpi_md_intr_mask and acpi_md_intr_unmask

Add intr_establish_xname support to arm and expose it to intrctl

Fix intrctl for pics with non-0 irqbase

Add support for intrctl(8).

Implement intr_string(9)

Add support for setting and getting interrupt affinity.

fix false positive KASSERT.
the case that secondary processors failed to hatch, and not attached.

when "cpu1 at cpus0: disabled (unresponsive)"

branches: 1.42.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.41.2;
Trailing whitespace

spaces to TAB

decouple pic_ipi_kpreempt() from DDB

The assertion at the bottom of intr_ipi_send is invalid for UP configs; add
a text for arm_cpu_max == 1

remove checks for failure after memory allocation calls that cannot fail:

kmem_alloc() with KM_SLEEP
kmem_zalloc() with KM_SLEEP
percpu_alloc()
pserialize_create()
psref_class_create()

all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.

unblock interrupts also when reusing a previously freed irq slot.

Serialise work in pic_add.

Add separate IPI routines for IPI_AST and IPI_KPREEMPT.

__HAVE_PREEEMPTION -> __HAVE_PREEMPTION

Don't send IPIs to ourselves if sending to everyone.

Don't need pic_ipi_sender anymore.

Use right kcpuset call.

Adapt pic to deal with the BCM2836 interrupts.
Move pic_pending_pics, pic_pending_ipls, and pic_blocked_irqs into a
structure and make then per-cpu. There is no global interrupt state anymore.

Add __HAVE_PIC_HAVE_PENDING_INTRS and define it if __HAVE_PIC_SET_PRIORITY
is undefined (also define in mvsoc_intr.h since their use of the latter is
peculiar). This new define controls whether the pending interrupt logic is
compiled. The GIC doesn't use pending interrupts since it uses the priority
level on the GIC to control delivery of interrupts, thus there can never
be a pending interrupt. The kernel shrinks about 4KB with the removal of
the pending interupt support,

Don't clear CI_ASTPENDING in exception return, do it in ast() instead.
Add basic support for __HAVE_PREEMPTION.
Use atomic ops for ci_astpending if __HAVE_PREEMPTION is defined.
Use kpreempt_disable/kpreempt_enable

Add aarch64 support (mostly remapping I32_bit and F32_bit onto aarch64 bits).

branches: 1.25.2;
Include opt_multiprocessor.h

In the MULTIPROCESSOR case mark all interrupt handlers not at IPL_VM as
mpsafe, so the kernel_lock is only taken at IPL_VM as per spl(9).

Comment out a printf that was confusing entering ddb output.

branches: 1.22.2;
Implement MI IPI interface with cross-call support.

branches: 1.21.2;
Support MPSAFE interrupts.

Add a mpsafe flag to the intrsource

Mark a diagnostic-only variable

Mark rv as unused (for now)

Use KASSERTMSG

Whitespace

branches: 1.15.2; 1.15.6;
Fix a bug that incorrect arg is passed to pic_block_irqs() on disestablish.
Fixes PR#45371 by Yuichiro Goto.

branches: 1.14.2;
Add #ifdef DIAGNOSTIC / #endif

Add PIC hooks for MP and for the ARM Generic Interrupt Controller.

Fix botched change to use right value for ipl. Thanks jak

Add hooks for __HAVE_PIC_SET_PRIORITY which allows updating of a hardware
(PIC) priority based on current IPL.

Convert a KASSERT to a KASSERTMSG

Add a KASSERT

branches: 1.8.4; 1.8.10; 1.8.12;
split arch/arm/pic/pic.c so that we can implement more efficient version of splfoo() while using pic framework.

Increment ci->ci_intr_depth for the duration of pic_do_pending_ints().
Fixes locking catastrophy commonly noticed with mvsata(4) on evbarm.

branches: 1.6.2; 1.6.4;
Fix lost interrupt. (2/2)
+ It is likely to lose sight of interrupt when the interrupt of irq_base
that is smaller than the same at the level is generated if PIC_MAXSOURCES
is 33 or more.

Fix lost interrupt. (1/2)
+ Change blocked_irqs that dispach if change pending_irqs in the loop.

branches: 1.4.4; 1.4.6;
Use atomic ops to manipulate the bitmasks.

branches: 1.3.8; 1.3.16;
Remove clause 3 and 4 from TNF licenses

Merge kernel changes in matt-armv6 to HEAD.

branches: 1.1.2; 1.1.20; 1.1.22; 1.1.24;
file pic.c was initially added on branch matt-armv6.

Fix a bug where pic_establish_intr would fail to call pic_establish_irq
if a free pic__iplsources slot was found, i.e. an interrupt handler at
the same ipl had been disestablished previously.

G/C pic_iplsource

KNF

Only perform pic_unblock_percpu if all of the following are true
- mp_online, i.e. APs are running.
- is_mpsafe, i.e. the interrupt handler is MP safe
- is_percpu, i.e. the interrupt actually requires it!

The last one (is_percpu) is true for GIC PPI+SGI only.

Assert we can sleep in pic_add

Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.

The AP initialisation code in cpu_init_secondary_processor will read and
initialise the required system registers and state for the BP to attach
and report.

Rework the interrupt handler code for this new sequence. Thankfully,
this removes a bunch of code for bcm2836mp.

The VFP detection handler on <= armv7 relies on the global undefined
handler being in place until the BP attaches vfp. That is, after the
APs have been spun up.

gicv3_its.c has a serialisation issue which is protected against in
the gicv3_its_cpu_init, which is called from cpu_hatch, with a spin
lock. The serialisation issue needs addressing more completely.

Tested on RPI3, Apple M1, QEMU, and lx2k

Fixes PR port-arm/56264:
diagnostic assertion "l->l_stat == LSONPROC" failed on RPI3

If an SGI or PPI is established after interrupts are enabled, make sure
we unblock the source on _all_ CPUs and not just the CPU that is
establishing the interrupt.

Re-apply

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

There is(/was) no need to use atomic operations on the percpu / cpu_info
members, so don't.

Finally removng the use of percpu should help avoid problems with "late"
attaching cpus.

Revert recent pic optimizations until I have more time to work on this.

branches: 1.69.2;
Inline pic_set_priority and use cpu_dosoftints_ci when available.

Fixup with __HAVE_PREEMPTION code which is currently unused

Inline "pic_do_pending_ints" in splx and check ci_pending_ipls to optimize
the common case (hw priority, no cascaded interrupts pending).

This also removes the need for the "pic_pending_used" flag, and should fix
booting on Raspberry Pi 3.

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

pic: avoid pic_do_pending_ints if pic_mark_pending_* has never been called

splx: use pic_set_priority_psw in interrupts disabled case to skip a few
more daif accesses.

pic: reduce the number of daif accesses in pic_do_pending_ints

The caller has already provided daif state. No need to keep updating
daif via pic_set_priority if it's already in the state we need (interrupts
disabled).

Use ENABLE_INTERRUPT() / DISABLE_INTERRUPT() instead of cpsie()/cpsid() in
places where we don't care about the cpsie() return value.

branches: 1.61.2;
intr_ipi_send: assert that kcp is either NULL or contains exactly one CPU

Improve a comment

KNF

KASSERT -> KASSERTMSG

Trailing whitespace

KNF

Reduce some ifdefs.

pic_pending_zero is unnecessary; percpu_alloc already zeroes.

Switch arm pic allocation and initialization to percpu_create.

branches: 1.52.2;
Update pic_add to allocate and return an irqbase if passed
PIC_IRQBASE_ALLOC.

Make pic_sourcebase static

Add reference counts to intr_mask/intr_unmask as calls can be nested, spotted by thorpej

Implement acpi_md_intr_mask and acpi_md_intr_unmask

Add intr_establish_xname support to arm and expose it to intrctl

Fix intrctl for pics with non-0 irqbase

Add support for intrctl(8).

Implement intr_string(9)

Add support for setting and getting interrupt affinity.

fix false positive KASSERT.
the case that secondary processors failed to hatch, and not attached.

when "cpu1 at cpus0: disabled (unresponsive)"

branches: 1.42.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.41.2;
Trailing whitespace

spaces to TAB

decouple pic_ipi_kpreempt() from DDB

The assertion at the bottom of intr_ipi_send is invalid for UP configs; add
a text for arm_cpu_max == 1

remove checks for failure after memory allocation calls that cannot fail:

kmem_alloc() with KM_SLEEP
kmem_zalloc() with KM_SLEEP
percpu_alloc()
pserialize_create()
psref_class_create()

all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.

unblock interrupts also when reusing a previously freed irq slot.

Serialise work in pic_add.

Add separate IPI routines for IPI_AST and IPI_KPREEMPT.

__HAVE_PREEEMPTION -> __HAVE_PREEMPTION

Don't send IPIs to ourselves if sending to everyone.

Don't need pic_ipi_sender anymore.

Use right kcpuset call.

Adapt pic to deal with the BCM2836 interrupts.
Move pic_pending_pics, pic_pending_ipls, and pic_blocked_irqs into a
structure and make then per-cpu. There is no global interrupt state anymore.

Add __HAVE_PIC_HAVE_PENDING_INTRS and define it if __HAVE_PIC_SET_PRIORITY
is undefined (also define in mvsoc_intr.h since their use of the latter is
peculiar). This new define controls whether the pending interrupt logic is
compiled. The GIC doesn't use pending interrupts since it uses the priority
level on the GIC to control delivery of interrupts, thus there can never
be a pending interrupt. The kernel shrinks about 4KB with the removal of
the pending interupt support,

Don't clear CI_ASTPENDING in exception return, do it in ast() instead.
Add basic support for __HAVE_PREEMPTION.
Use atomic ops for ci_astpending if __HAVE_PREEMPTION is defined.
Use kpreempt_disable/kpreempt_enable

Add aarch64 support (mostly remapping I32_bit and F32_bit onto aarch64 bits).

branches: 1.25.2;
Include opt_multiprocessor.h

In the MULTIPROCESSOR case mark all interrupt handlers not at IPL_VM as
mpsafe, so the kernel_lock is only taken at IPL_VM as per spl(9).

Comment out a printf that was confusing entering ddb output.

branches: 1.22.2;
Implement MI IPI interface with cross-call support.

branches: 1.21.2;
Support MPSAFE interrupts.

Add a mpsafe flag to the intrsource

Mark a diagnostic-only variable

Mark rv as unused (for now)

Use KASSERTMSG

Whitespace

branches: 1.15.2; 1.15.6;
Fix a bug that incorrect arg is passed to pic_block_irqs() on disestablish.
Fixes PR#45371 by Yuichiro Goto.

branches: 1.14.2;
Add #ifdef DIAGNOSTIC / #endif

Add PIC hooks for MP and for the ARM Generic Interrupt Controller.

Fix botched change to use right value for ipl. Thanks jak

Add hooks for __HAVE_PIC_SET_PRIORITY which allows updating of a hardware
(PIC) priority based on current IPL.

Convert a KASSERT to a KASSERTMSG

Add a KASSERT

branches: 1.8.4; 1.8.10; 1.8.12;
split arch/arm/pic/pic.c so that we can implement more efficient version of splfoo() while using pic framework.

Increment ci->ci_intr_depth for the duration of pic_do_pending_ints().
Fixes locking catastrophy commonly noticed with mvsata(4) on evbarm.

branches: 1.6.2; 1.6.4;
Fix lost interrupt. (2/2)
+ It is likely to lose sight of interrupt when the interrupt of irq_base
that is smaller than the same at the level is generated if PIC_MAXSOURCES
is 33 or more.

Fix lost interrupt. (1/2)
+ Change blocked_irqs that dispach if change pending_irqs in the loop.

branches: 1.4.4; 1.4.6;
Use atomic ops to manipulate the bitmasks.

branches: 1.3.8; 1.3.16;
Remove clause 3 and 4 from TNF licenses

Merge kernel changes in matt-armv6 to HEAD.

branches: 1.1.2; 1.1.20; 1.1.22; 1.1.24;
file pic.c was initially added on branch matt-armv6.

Only perform pic_unblock_percpu if all of the following are true
- mp_online, i.e. APs are running.
- is_mpsafe, i.e. the interrupt handler is MP safe
- is_percpu, i.e. the interrupt actually requires it!

The last one (is_percpu) is true for GIC PPI+SGI only.

Assert we can sleep in pic_add

Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.

The AP initialisation code in cpu_init_secondary_processor will read and
initialise the required system registers and state for the BP to attach
and report.

Rework the interrupt handler code for this new sequence. Thankfully,
this removes a bunch of code for bcm2836mp.

The VFP detection handler on <= armv7 relies on the global undefined
handler being in place until the BP attaches vfp. That is, after the
APs have been spun up.

gicv3_its.c has a serialisation issue which is protected against in
the gicv3_its_cpu_init, which is called from cpu_hatch, with a spin
lock. The serialisation issue needs addressing more completely.

Tested on RPI3, Apple M1, QEMU, and lx2k

Fixes PR port-arm/56264:
diagnostic assertion "l->l_stat == LSONPROC" failed on RPI3

If an SGI or PPI is established after interrupts are enabled, make sure
we unblock the source on _all_ CPUs and not just the CPU that is
establishing the interrupt.

Re-apply

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

There is(/was) no need to use atomic operations on the percpu / cpu_info
members, so don't.

Finally removng the use of percpu should help avoid problems with "late"
attaching cpus.

Revert recent pic optimizations until I have more time to work on this.

branches: 1.69.2;
Inline pic_set_priority and use cpu_dosoftints_ci when available.

Fixup with __HAVE_PREEMPTION code which is currently unused

Inline "pic_do_pending_ints" in splx and check ci_pending_ipls to optimize
the common case (hw priority, no cascaded interrupts pending).

This also removes the need for the "pic_pending_used" flag, and should fix
booting on Raspberry Pi 3.

Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few
instructions in splx.

pic: avoid pic_do_pending_ints if pic_mark_pending_* has never been called

splx: use pic_set_priority_psw in interrupts disabled case to skip a few
more daif accesses.

pic: reduce the number of daif accesses in pic_do_pending_ints

The caller has already provided daif state. No need to keep updating
daif via pic_set_priority if it's already in the state we need (interrupts
disabled).

Use ENABLE_INTERRUPT() / DISABLE_INTERRUPT() instead of cpsie()/cpsid() in
places where we don't care about the cpsie() return value.

branches: 1.61.2;
intr_ipi_send: assert that kcp is either NULL or contains exactly one CPU

Improve a comment

KNF

KASSERT -> KASSERTMSG

Trailing whitespace

KNF

Reduce some ifdefs.

pic_pending_zero is unnecessary; percpu_alloc already zeroes.

Switch arm pic allocation and initialization to percpu_create.

branches: 1.52.2;
Update pic_add to allocate and return an irqbase if passed
PIC_IRQBASE_ALLOC.

Make pic_sourcebase static

Add reference counts to intr_mask/intr_unmask as calls can be nested, spotted by thorpej

Implement acpi_md_intr_mask and acpi_md_intr_unmask

Add intr_establish_xname support to arm and expose it to intrctl

Fix intrctl for pics with non-0 irqbase

Add support for intrctl(8).

Implement intr_string(9)

Add support for setting and getting interrupt affinity.

fix false positive KASSERT.
the case that secondary processors failed to hatch, and not attached.

when "cpu1 at cpus0: disabled (unresponsive)"

branches: 1.42.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.41.2;
Trailing whitespace

spaces to TAB

decouple pic_ipi_kpreempt() from DDB

The assertion at the bottom of intr_ipi_send is invalid for UP configs; add
a text for arm_cpu_max == 1

remove checks for failure after memory allocation calls that cannot fail:

kmem_alloc() with KM_SLEEP
kmem_zalloc() with KM_SLEEP
percpu_alloc()
pserialize_create()
psref_class_create()

all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.

unblock interrupts also when reusing a previously freed irq slot.

Serialise work in pic_add.

Add separate IPI routines for IPI_AST and IPI_KPREEMPT.

__HAVE_PREEEMPTION -> __HAVE_PREEMPTION

Don't send IPIs to ourselves if sending to everyone.

Don't need pic_ipi_sender anymore.

Use right kcpuset call.

Adapt pic to deal with the BCM2836 interrupts.
Move pic_pending_pics, pic_pending_ipls, and pic_blocked_irqs into a
structure and make then per-cpu. There is no global interrupt state anymore.

Add __HAVE_PIC_HAVE_PENDING_INTRS and define it if __HAVE_PIC_SET_PRIORITY
is undefined (also define in mvsoc_intr.h since their use of the latter is
peculiar). This new define controls whether the pending interrupt logic is
compiled. The GIC doesn't use pending interrupts since it uses the priority
level on the GIC to control delivery of interrupts, thus there can never
be a pending interrupt. The kernel shrinks about 4KB with the removal of
the pending interupt support,