#
347700 |
|
16-May-2019 |
markj |
MFC r337715, r337751, r337754, r337758, r337813, r338354, r338687, r339124, r341821: Add support for boot-time Intel microcode loading.
|
#
338419 |
|
01-Sep-2018 |
kib |
MFC r338312: Unify amd64 and i386 vmspace0 pmap activation.
|
#
338300 |
|
24-Aug-2018 |
kib |
MFC r337773, r337838, r338112, r338202: Fixes for early EFIRT usage on amd64.
Activate vmspace0 pmap, not kernel_pmap. Use special version of the pmap activation code, pmap_activate_boot(). Ensure that PCID state for the vmspace0 pmap is valid early.
|
#
335817 |
|
30-Jun-2018 |
kib |
MFC r333087 (by cem): amd64/mp_machdep.c: Fix GCC build after r333059.
Noted by: bde
|
#
335570 |
|
22-Jun-2018 |
kib |
MFC r333059 (by tychon): Expand the checks for UCR3 == PMAP_NO_CR3 to enable processes to be excluded from PTI.
|
#
333369 |
|
08-May-2018 |
emaste |
MFC r333368: Prepare DB# handler for deferred trigger of watchpoints.
Prepare DB# handler for deferred trigger of watchpoints.
Since pop %ss/mov %ss instructions defer all interrupts and exceptions for the next instruction, it is possible that the userspace watchpoint trap executes on the first instruction of the kernel entry for syscall/bpt.
In this case, DB# should be treated similarly to NMI: on amd64 we must always load GSBASE even if the trap comes from kernel mode, and load the kernel page table root into %cr3. Moreover, the trap must use the dedicated stack, because we are still on the user stack when trapped on syscall entry.
For i386, we must reload %cr3. The syscall instruction is not configured, so there is no issue with executing on user stack when trapping.
Due to some CPU erratas it is not always possible to detect that the userspace watchpoint triggered by inspecting %dr6. In trap(), compare the trap %rip with the known unsafe entry points and if matched pretend that the watchpoint did not fire at all.
Thank you to the MSRC Incident Response Team, and in particular Greg Lenti and Nate Warfield, for coordinating the response to this issue across multiple vendors.
Thanks to Computer Recycling at The Working Center of Kitchener for making hardware available to allow us to test the patch on additional CPU families.
Reviewed by: jhb Discussed with: Matthew Dillon Tested by: emaste Approved by: re (so blanket) Security: CVE-2018-8897 Security: FreeBSD-SA-18:06.debugreg Sponsored by: The FreeBSD Foundation
|
#
331722 |
|
29-Mar-2018 |
eadler |
Revert r330897:
This was intended to be a non-functional change. It wasn't. The commit message was thus wrong. In addition it broke arm, and merged crypto related code.
Revert with prejudice.
This revert skips files touched in r316370 since that commit was since MFCed. This revert also skips files that require $FreeBSD$ property changes.
Thank you to those who helped me get out of this mess including but not limited to gonzo, kevans, rgrimes.
Requested by: gjb (re)
|
#
330897 |
|
14-Mar-2018 |
eadler |
Partial merge of the SPDX changes
These changes are incomplete but are making it difficult to determine what other changes can/should be merged.
No objections from: pfg
|
#
329462 |
|
17-Feb-2018 |
kib |
MFC r328083,328096,328116,328119,328120,328128,328135,328153,328157, 328166,328177,328199,328202,328205,328468,328470,328624,328625,328627, 328628,329214,329297,329365:
Meltdown mitigation by PTI, PCID optimization of PTI, and kernel use of IBRS for some mitigations of Spectre.
Tested by: emaste, Arshan Khanifar <arshankhanifar@gmail.com> Discussed with: jkim Sponsored by: The FreeBSD Foundation
|
#
328123 |
|
18-Jan-2018 |
kib |
MFC r327818: Move the hardware setup for fast syscalls into a common function.
|
#
307856 |
|
24-Oct-2016 |
kib |
MFC r306680: Reduce the cost of TLB invalidation on x86 by using per-CPU completion flags.
|
#
302408 |
|
07-Jul-2016 |
gjb |
Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle. Prune svn:mergeinfo from the new branch, as nothing has been merged here.
Additional commits post-branch will follow.
Approved by: re (implicit) Sponsored by: The FreeBSD Foundation |
#
297857 |
|
12-Apr-2016 |
avg |
re-enable AMD Topology extension on certain models if disabled by BIOS
Some BIOSes disable AMD Topology extension on AMD Family 15h notebook processors. We re-enable the extension, so that we can properly discover core and cache topology. Linux seems to do the same.
Reported by: Johannes Dieterich <dieterich.joh@gmail.com> Reviewed by: jhb, kib Tested by: Johannes Dieterich <dieterich.joh@gmail.com> (earlier version) MFC after: 3 weeks Differential Revision: https://reviews.freebsd.org/D5883
|
#
291688 |
|
03-Dec-2015 |
kib |
For amd64 non-PCID machines, and for i386 machines with support for the PG_G global pte flag, pmap_invalidate_all() fails to flush global TLB entries [*]. This is because TLB shootdown handler for such configs reloads CR3, and on i386 pmap_invalidate_all() does the same for the initiating CPU. Note that current code does not issue total invalidation requests for the kernel_pmap.
Rename amd64 function invltlb_globpcid() to invltlb_glob(), it is not specific for PCID for quite some time, and implement the same functionality for i386. Use the function instead of invltlb() in shootdown handlers and in i386 pmap_invalidate_all(), but only for the kernel pmap (which maps pages with the PG_G attribute set), which takes care of PG_G TLB entries on flush.
To detect the affected pmap in i386 TLB shootdown handler, pmap should be passed to the smp_masked_invltlb() function, which makes amd64 and i386 TLB shootdown code almost identical. Merge the code under x86/.
Noted by: jhb [*] Reviewed by: cem, jhb, pho Tested by: pho Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D4346
|
#
286658 |
|
12-Aug-2015 |
kib |
Initialization of smp_tlb_wait does not require release semantic, no data is synchronized by store/load to the variable. The lapic_write_icr() function ensures that store buffers are flushed before IPI command is issued.
Discussed with: bde Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
|
#
286657 |
|
12-Aug-2015 |
kib |
AP should load aps_ready with acquire semantic to see BSP updates to the SMP structures, synchronized with the load by release store in release_aps().
The change is formal, x86 strong memory model implicitely provided the guarantees.
Discussed with: bde Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
|
#
286584 |
|
10-Aug-2015 |
kib |
Make kstack_pages a tunable on arm, x86, and powepc. On i386, the initial thread stack is not adjusted by the tunable, the stack is allocated too early to get access to the kernel environment. See TD0_KSTACK_PAGES for the thread0 stack sizing on i386.
The tunable was tested on x86 only. From the visual inspection, it seems that it might work on arm and powerpc. The arm USPACE_SVC_STACK_TOP and powerpc USPACE macros seems to be already incorrect for the threads with non-default kstack size. I only changed the macros to use variable instead of constant, since I cannot test.
On arm64, mips and sparc64, some static data structures are sized by KSTACK_PAGES, so the tunable is disabled.
Sponsored by: The FreeBSD Foundation MFC after: 2 week
|
#
286228 |
|
03-Aug-2015 |
kib |
Clear the IA32_MISC_ENABLE MSR bit, which limits the max CPUID reported, on APs. We already did this on BSP.
Otherwise, the userspace software which depends on the features reported by the high CPUID levels is misbehaving. In particular, AVX detection is non-functional, depending on which CPU thread happens to execute when doing CPUID. Another victim is the libthr signal handlers interposer, which needs to save full FPU extended state.
Reported and tested by: Andre Meiser <ortadur@web.de> Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
|
#
282684 |
|
09-May-2015 |
kib |
Rewrite amd64 PCID implementation to follow an algorithm described in the Vahalia' "Unix Internals" section 15.12 "Other TLB Consistency Algorithms". The same algorithm is already utilized by the MIPS pmap to handle ASIDs.
The PCID for the address space is now allocated per-cpu during context switch to the thread using pmap, when no PCID on the cpu was ever allocated, or the current PCID is invalidated. If the PCID is reused, bit 63 of %cr3 can be set to avoid TLB flush.
Each cpu has PCID' algorithm generation count, which is saved in the pmap pcpu block when pcpu PCID is allocated. On invalidation, the pmap generation count is zeroed, which signals the context switch code that already allocated PCID is no longer valid. The implication is the TLB shootdown for the given cpu/address space, due to the allocation of new PCID.
The pm_save mask is no longer has to be tracked, which (significantly) reduces the targets of the TLB shootdown IPIs. Previously, pm_save was reset only on pmap_invalidate_all(), which made it accumulate the cpuids of all processors on which the thread was scheduled between full TLB shootdowns.
Besides reducing the amount of TLB shootdowns and removing atomics to update pm_saves in the context switch code, the algorithm is much simpler than the maintanence of pm_save and selection of the right address space in the shootdown IPI handler.
Reviewed by: alc Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 3 weeks
|
#
281940 |
|
24-Apr-2015 |
kib |
Move common code from sys/i386/i386/mp_machdep.c and sys/amd64/amd64/mp_machdep.c, to the new common x86 source sys/x86/x86/mp_x86.c.
Proposed and reviewed by: jhb Review: https://reviews.freebsd.org/D2347 Sponsored by: The FreeBSD Foundation
|
#
280866 |
|
30-Mar-2015 |
jhb |
Wait 100 microseconds for a local APIC to dispatch each startup-related IPI rather than 20. The MP 1.4 specification states in Appendix B.2:
"A period of 20 microseconds should be sufficient for IPI dispatch to complete under normal operating conditions".
(Note that this appears to be separate from the 10 millisecond (INIT) and 200 microsecond (STARTUP) waits after the IPIs are dispatched.) The Intel SDM is silent on this issue as far as I can tell.
At least some hardware requires 60 microseconds as noted in the PR, so bump this to 100 to be on the safe side.
PR: 197756 Reported by: zaphod@berentweb.com MFC after: 1 week
|
#
279699 |
|
06-Mar-2015 |
jhb |
Only schedule interrupts on a single hyperthread of a modern Intel CPU core by default. Previously we used a single hyperthread on Pentium4-era cores but used both hyperthreads on more recent CPUs.
MFC after: 2 weeks
|
#
279406 |
|
28-Feb-2015 |
kib |
Supposed fix for some SandyBridge mobile CPUs hang on AP startup when x2APIC mode is detected and enabled. Current theory is that switching the APIC mode while an IPI is in flight might be the issue.
Postpone switching to x2APIC mode until we are guaranteed that all starting IPIs are already send and aknowledged. Use aps_ready signal as an indication that the BSP is done with us.
Tested by: adrian Sponsored by: The FreeBSD Foundation MFC after: 2 months
|
#
278869 |
|
16-Feb-2015 |
kib |
Initialize x2APIC mode on the resume path before accessing LAPIC.
Remove unneeded disable of LAPIC in the native_lapic_xapic_mode(). We attempt to send wakeup IPI on the resume path right after BSP wakeup, so disabling is wrong.
Reported and tested by: glebius, "Ranjan1018 ." <214748mv@gmail.com> Sponsored by: The FreeBSD Foundation MFC after: 2 months
|
#
278473 |
|
09-Feb-2015 |
kib |
Add x2APIC support. Enable it by default if CPU is capable. The hw.x2apic_enable tunable allows disabling it from the loader prompt.
To closely repeat effects of the uncached memory ops when accessing registers in the xAPIC mode, the x2APIC writes to MSRs are preceeded by mfence, except for the EOI notifications. This is probably too strict, only ICR writes to send IPI require serialization to ensure that other CPUs see the previous actions when IPI is delivered. This may be changed later.
In vmm justreturn IPI handler, call doreti_iret instead of doing iretd inline, to handle corner conditions.
Note that the patch only switches LAPICs into x2APIC mode. It does not enables FreeBSD to support > 255 CPUs, which requires parsing x2APIC MADT entries and doing interrupts remapping, but is the required step on the way.
Reviewed by: neel Tested by: pho (real hardware), neel (on bhyve) Discussed with: jhb, grehan Sponsored by: The FreeBSD Foundation MFC after: 2 months
|
#
278325 |
|
06-Feb-2015 |
jhb |
Revert the IPI startup sequence to match what is described in the Intel Multiprocessor Specification v1.4. The Intel SDM claims that the INIT IPIs here are invalid, but other systems follow the MP spec instead.
While here, fix the IPI wait routine to accept a timeout in microseconds instead of a raw spin count, and don't spin forever during AP startup. Instead, panic if a STARTUP IPI is not delivered after 20 us.
PR: 196542 Differential Revision: https://reviews.freebsd.org/D1719 MFC after: 2 weeks
|
#
271409 |
|
10-Sep-2014 |
jhb |
MFamd64: Use initializecpu() to set various model-specific registers on AP startup and AP resume (it was already used for BSP startup and BSP resume). - Split code to do one-time probing of cache properties out of initializecpu() and into initializecpucache(). This is called once on the BSP during boot. - Move enable_sse() into initializecpu(). - Call initializecpu() for AP startup instead of enable_sse() and manually frobbing MSR_EFER to enable PG_NX. - Call initializecpu() when an AP resumes. In theory this will now properly re-enable PG_NX in MSR_EFER when resuming a PAE kernel on APs.
|
#
271192 |
|
06-Sep-2014 |
jhb |
Create a separate structure for per-CPU state saved across suspend and resume that is a superset of a pcb. Move the FPU state out of the pcb and into this new structure. As part of this, move the FPU resume code on amd64 into a C function. This allows resumectx() to still operate only on a pcb and more closely mirrors the i386 code.
Reviewed by: kib (earlier version)
|
#
267526 |
|
16-Jun-2014 |
royger |
amd64/i386: introduce APIC hooks for different APIC implementations.
This is needed for Xen PV(H) guests, since there's no hardware lapic available on this kind of domains. This commit should not change functionality.
Sponsored by: Citrix Systems R&D Reviewed by: jhb Approved by: gibbs
amd64/include/cpu.h: amd64/amd64/mp_machdep.c: i386/include/cpu.h: i386/i386/mp_machdep.c: - Remove lapic_ipi_vectored hook from cpu_ops, since it's now implemented in the lapic hooks.
amd64/amd64/mp_machdep.c: i386/i386/mp_machdep.c: - Use lapic_ipi_vectored directly, since it's now an inline function that will call the appropiate hook.
x86/x86/local_apic.c: - Prefix bare metal public lapic functions with native_ and mark them as static. - Define default implementation of apic_ops.
x86/include/apicvar.h: - Declare the apic_ops structure and create inline functions to access the hooks, so the change is transparent to existing users of the lapic_ functions.
x86/xen/hvm.c: - Switch to use the new apic_ops.
|
#
266093 |
|
14-May-2014 |
neel |
Increase the TSS limit by one byte. The processor requires an additional byte with all bits set to 1 beyond the I/O permission bitmap.
Prior to this change accessing I/O ports [0xFFF8-0xFFFF] would trigger a #GP fault even though the I/O bitmap allowed access to those ports.
For more details see section "I/O Permission Bit Map" in the Intel SDM, Vol 1.
Reviewed by: kib
|
#
265004 |
|
27-Apr-2014 |
kib |
Same as it was done in r263878 for invlrng_handler(), fix order of checks for special pcid values in invlpg_pcid_handler(). Forst check for special values, and only then do PCID-specific page invalidation.
Minor fix to the style compliance, declare local variable at the function start.
Sponsored by: The FreeBSD Foundation MFC after: 1 week
|
#
264984 |
|
26-Apr-2014 |
scottl |
Retire smp_active. It was racey and caused demonstrated problems with the cpufreq code. Replace its use with smp_started. There's at least one userland tool that still looks at the kern.smp.active sysctl, so preserve it but point it to smp_started as well.
Discussed with: peter, jhb MFC after: 3 days Obtained from: Netflix
|
#
263878 |
|
28-Mar-2014 |
kib |
Several fixes for the PCID implementation:
- When clearing a bit for a cpuid in pmap->pm_save, ensure that the cpuid is not set in pm_active. The pm_save indicates which CPUs may have cached translations for given PCID, which implies that a CPU executing with the given pmap active have the translations cached. [1]
- In smp_masked_invltlb(), pass pmap to smp_targeted_tlb_shootdown(). [1]
- In invlrng_handler(), check for the special values of pcid (0 and -1) and do corresponding global or total invalidations before checking for performing PCID-specific range invalidation with INVPCID_ADDR. [2]
- In invltlb_pcid_handler(), do not read %cr3 unless needed. [2]
- Do minor style tweaks. [2]
Submitted by: Henrik Gulbrandsen <henrik@gulbra.net> [1] Other parts sponsored by: The FreeBSD Foundation [2] Tested by: Henrik Gulbrandsen, pho MFC after: 1 week
|
#
263014 |
|
11-Mar-2014 |
royger |
xen: add a hook to perform AP startup
AP startup on PVH follows the PV method, so we need to add a hook in order to diverge from bare metal.
Approved by: gibbs Sponsored by: Citrix Systems R&D
amd64/amd64/machdep.c: - Add hook for start_all_aps on native (using native_start_all_aps defined in mp_machdep).
amd64/amd64/mp_machdep.c: - Make some variables global because they will also be used by the Xen PVH AP startup code. - Use the start_all_aps hook to start APs. - Rename start_all_aps to native_start_all_aps.
amd64/include/smp.h: - Add declaration for native_start_all_aps.
x86/include/init.h: - Declare start_all_aps hook in init_ops.
x86/xen/pv.c: - Pick external declarations from mp_machdep. - Introduce Xen PV code to start APs on PVH. - Set start_all_aps init hook to use the Xen PVH implementation.
|
#
263001 |
|
11-Mar-2014 |
royger |
amd64/i386: switch IPI handlers to C code.
Move asm IPIs handlers to C code, so both Xen and native IPI handlers share the same code.
Reviewed by: jhb Approved by: gibbs Sponsored by: Citrix Systems R&D
amd64/amd64/apic_vector.S: i386/i386/apic_vector.s: - Remove asm coded IPI handlers and instead call the newly introduced C variants.
amd64/amd64/mp_machdep.c: i386/i386/mp_machdep.c: - Add C coded clones to the asm IPI handlers (moved from x86/xen/hvm.c).
i386/include/smp.h: amd64/include/smp.h: - Add prototypes for the C IPI handlers.
x86/xen/hvm.c: - Move the C IPI handlers to mp_machdep and call those in the Xen IPI handlers.
i386/xen/mp_machdep.c: - Add dummy IPI handlers to the i386 Xen PV port (this port doesn't support SMP).
|
#
262752 |
|
04-Mar-2014 |
jkim |
Move fpusave() wrapper for suspend hander to sys/amd64/amd64/fpu.c.
Inspired by: jhb
|
#
259782 |
|
23-Dec-2013 |
jhb |
Add a resume hook for bhyve that runs a function on all CPUs during resume. For Intel CPUs, invoke vmxon for CPUs that were in VMX mode at the time of suspend.
Reviewed by: neel
|
#
255744 |
|
20-Sep-2013 |
gibbs |
Merge Xen PVHVM support into the GENERIC kernel config for both amd64 and i386.
Submitted by: Roger Pau Monné Sponsored by: Citrix Systems R&D Reviewed by: gibbs Approved by: re (blanket Xen) MFC after: 2 weeks
sys/amd64/amd64/mp_machdep.c: sys/amd64/include/cpu.h: sys/i386/i386/mp_machdep.c: sys/i386/include/cpu.h: - Introduce two new CPU hooks for initialization and resume purposes. This allows us to get rid of the XENHVM ifdefs in mp_machdep, and also sets some hooks into common code that can be used by other hypervisor implementations.
sys/amd64/conf/XENHVM: sys/i386/conf/XENHVM: - Remove these configs now that GENERIC has builtin support for Xen HVM.
sys/kern/subr_smp.c: - Make sure there are no pending IPIs when suspending a system.
sys/x86/xen/hvm.c: - Add cpu init and resume vectors that are called from mp_machdep using the new hooks. - Only clear the vcpu_info mapping data on resume. It is already clear for the BSP on a cold boot and is set correctly as APs are started. - Gate xen_hvm_init_cpu only to systems running under Xen.
sys/x86/xen/xen_intr.c: - Gate the setup of event channels only to systems running under Xen.
|
#
255726 |
|
20-Sep-2013 |
gibbs |
Add support for suspend/resume/migration operations when running as a Xen PVHVM guest.
Submitted by: Roger Pau Monné Sponsored by: Citrix Systems R&D Reviewed by: gibbs Approved by: re (blanket Xen) MFC after: 2 weeks
sys/amd64/amd64/mp_machdep.c: sys/i386/i386/mp_machdep.c: - Make sure that are no MMU related IPIs pending on migration. - Reset pending IPI_BITMAP on resume. - Init vcpu_info on resume.
sys/amd64/include/intr_machdep.h: sys/i386/include/intr_machdep.h: sys/x86/acpica/acpi_wakeup.c: sys/x86/x86/intr_machdep.c: sys/x86/isa/atpic.c: sys/x86/x86/io_apic.c: sys/x86/x86/local_apic.c: - Add a "suspend_cancelled" parameter to pic_resume(). For the Xen PIC, restoration of interrupt services differs between the aborted suspend and normal resume cases, so we must provide this information.
sys/dev/acpica/acpi_timer.c: sys/dev/xen/timer/timer.c: sys/timetc.h: - Don't swap out "suspend safe" timers across a suspend/resume cycle. This includes the Xen PV and ACPI timers.
sys/dev/xen/control/control.c: - Perform proper suspend/resume process for PVHVM: - Suspend all APs before going into suspension, this allows us to reset the vcpu_info on resume for each AP. - Reset shared info page and callback on resume.
sys/dev/xen/timer/timer.c: - Implement suspend/resume support for the PV timer. Since FreeBSD doesn't perform a per-cpu resume of the timer, we need to call smp_rendezvous in order to correctly resume the timer on each CPU.
sys/dev/xen/xenpci/xenpci.c: - Don't reset the PCI interrupt on each suspend/resume.
sys/kern/subr_smp.c: - When suspending a PVHVM domain make sure there are no MMU IPIs in-flight, or we will get a lockup on resume due to the fact that pending event channels are not carried over on migration. - Implement a generic version of restart_cpus that can be used by suspended and stopped cpus.
sys/x86/xen/hvm.c: - Implement resume support for the hypercall page and shared info. - Clear vcpu_info so it can be reset by APs when resuming from suspension.
sys/dev/xen/xenpci/xenpci.c: sys/x86/xen/hvm.c: sys/x86/xen/xen_intr.c: - Support UP kernel configurations.
sys/x86/xen/xen_intr.c: - Properly rebind per-cpus VIRQs and IPIs on resume.
|
#
255331 |
|
06-Sep-2013 |
gibbs |
Implement PV IPIs for PVHVM guests and further converge PV and HVM IPI implmementations.
Submitted by: Roger Pau Monné Sponsored by: Citrix Systems R&D Submitted by: gibbs (misc cleanup, table driven config) Reviewed by: gibbs MFC after: 2 weeks
sys/amd64/include/cpufunc.h: sys/amd64/amd64/pmap.c: Move invltlb_globpcid() into cpufunc.h so that it can be used by the Xen HVM version of tlb shootdown IPI handlers.
sys/x86/xen/xen_intr.c: sys/xen/xen_intr.h: Rename xen_intr_bind_ipi() to xen_intr_alloc_and_bind_ipi(), and remove the ipi vector parameter. This api allocates an event channel port that can be used for ipi services, but knows nothing of the actual ipi for which that port will be used. Removing the unused argument and cleaning up the comments surrounding its declaration helps clarify its actual role.
sys/amd64/amd64/mp_machdep.c: sys/amd64/include/cpu.h: sys/i386/i386/mp_machdep.c: sys/i386/include/cpu.h: Implement a generic framework for amd64 and i386 that allows the implementation of certain CPU management functions to be selected at runtime. Currently this is only used for the ipi send function, which we optimize for Xen when running on a Xen hypervisor, but can easily be expanded to support more operations.
sys/x86/xen/hvm.c: Implement Xen PV IPI handlers and operations, replacing native send IPI.
sys/amd64/include/pcpu.h: sys/i386/include/pcpu.h: sys/i386/include/smp.h: Remove NR_VIRQS and NR_IPIS from FreeBSD headers. NR_VIRQS is defined already for us in the xen interface files. NR_IPIS is only needed in one file per Xen platform and is easily inferred by the IPI vector table that is defined in those files.
sys/i386/xen/mp_machdep.c: Restructure to more closely match the HVM implementation by performing table driven IPI setup.
|
#
255217 |
|
04-Sep-2013 |
kib |
Tidy up some loose ends in the PCID code:
- Restore the pre-PCID TLB shootdown handlers for whole address space and single page invalidation asm code, and assign the IPI handler to them when PCID is not supported or disabled. Old handlers have linear control flow. But, still use the common return sequence.
- Stop using pcpu for INVPCID descriptors in the invlrg handler. It is enough to allocate descriptors on the stack. As result, two SWAPGS instructions are shaved off from the code for Haswell+.
- Fix the reverted condition in invlrng for checking of the PCID support [1], also in invlrng check that pmap is kernel pmap before performing other tests. For the kernel pmap, which provides global mappings, the INVLPG must be used for invalidation always.
- Save the pre-computed pmap' %CR3 register in the struct pmap. This allows to remove several checks for pm_pcid validity when %CR3 is reloaded [2].
Noted by: gibbs [1] Discussed with: alc [2] Tested by: pho, flo Sponsored by: The FreeBSD Foundation
|
#
255060 |
|
30-Aug-2013 |
kib |
Implement support for the process-context identifiers ('PCID') on Intel CPUs. The feature tags TLB entries with the Id of the address space and allows to avoid TLB invalidation on the context switch, it is available only in the long mode. In the microbenchmarks, using the PCID decreased latency of the context switches by ~30% on SandyBridge class desktop CPUs, measured with the lat_ctx program from lmbench.
If available, use INVPCID instruction when a TLB entry in non-current address space needs to be invalidated. The instruction is typically available on the Haswell.
If needed, the use of PCID can be turned off with the vm.pmap.pcid_enabled loader tunable set to 0. The state of the feature is reported by the vm.pmap.pcid_enabled sysctl. The sysctl vm.pmap.pcid_save_cnt reports the number of context switches which avoided invalidating the TLB; compare with the total number of context switches, available as sysctl vm.stats.sys.v_swtch.
Sponsored by: The FreeBSD Foundation Reviewed by: alc Tested by: pho, bf
|
#
255040 |
|
29-Aug-2013 |
gibbs |
Implement vector callback for PVHVM and unify event channel implementations
Re-structure Xen HVM support so that: - Xen is detected and hypercalls can be performed very early in system startup. - Xen interrupt services are implemented using FreeBSD's native interrupt delivery infrastructure. - the Xen interrupt service implementation is shared between PV and HVM guests. - Xen interrupt handlers can optionally use a filter handler in order to avoid the overhead of dispatch to an interrupt thread. - interrupt load can be distributed among all available CPUs. - the overhead of accessing the emulated local and I/O apics on HVM is removed for event channel port events. - a similar optimization can eventually, and fairly easily, be used to optimize MSI.
Early Xen detection, HVM refactoring, PVHVM interrupt infrastructure, and misc Xen cleanups:
Sponsored by: Spectra Logic Corporation
Unification of PV & HVM interrupt infrastructure, bug fixes, and misc Xen cleanups:
Submitted by: Roger Pau Monné Sponsored by: Citrix Systems R&D
sys/x86/x86/local_apic.c: sys/amd64/include/apicvar.h: sys/i386/include/apicvar.h: sys/amd64/amd64/apic_vector.S: sys/i386/i386/apic_vector.s: sys/amd64/amd64/machdep.c: sys/i386/i386/machdep.c: sys/i386/xen/exception.s: sys/x86/include/segments.h: Reserve IDT vector 0x93 for the Xen event channel upcall interrupt handler. On Hypervisors that support the direct vector callback feature, we can request that this vector be called directly by an injected HVM interrupt event, instead of a simulated PCI interrupt on the Xen platform PCI device. This avoids all of the overhead of dealing with the emulated I/O APIC and local APIC. It also means that the Hypervisor can inject these events on any CPU, allowing upcalls for different ports to be handled in parallel.
sys/amd64/amd64/mp_machdep.c: sys/i386/i386/mp_machdep.c: Map Xen per-vcpu area during AP startup.
sys/amd64/include/intr_machdep.h: sys/i386/include/intr_machdep.h: Increase the FreeBSD IRQ vector table to include space for event channel interrupt sources.
sys/amd64/include/pcpu.h: sys/i386/include/pcpu.h: Remove Xen HVM per-cpu variable data. These fields are now allocated via the dynamic per-cpu scheme. See xen_intr.c for details.
sys/amd64/include/xen/hypercall.h: sys/dev/xen/blkback/blkback.c: sys/i386/include/xen/xenvar.h: sys/i386/xen/clock.c: sys/i386/xen/xen_machdep.c: sys/xen/gnttab.c: Prefer FreeBSD primatives to Linux ones in Xen support code.
sys/amd64/include/xen/xen-os.h: sys/i386/include/xen/xen-os.h: sys/xen/xen-os.h: sys/dev/xen/balloon/balloon.c: sys/dev/xen/blkback/blkback.c: sys/dev/xen/blkfront/blkfront.c: sys/dev/xen/console/xencons_ring.c: sys/dev/xen/control/control.c: sys/dev/xen/netback/netback.c: sys/dev/xen/netfront/netfront.c: sys/dev/xen/xenpci/xenpci.c: sys/i386/i386/machdep.c: sys/i386/include/pmap.h: sys/i386/include/xen/xenfunc.h: sys/i386/isa/npx.c: sys/i386/xen/clock.c: sys/i386/xen/mp_machdep.c: sys/i386/xen/mptable.c: sys/i386/xen/xen_clock_util.c: sys/i386/xen/xen_machdep.c: sys/i386/xen/xen_rtc.c: sys/xen/evtchn/evtchn_dev.c: sys/xen/features.c: sys/xen/gnttab.c: sys/xen/gnttab.h: sys/xen/hvm.h: sys/xen/xenbus/xenbus.c: sys/xen/xenbus/xenbus_if.m: sys/xen/xenbus/xenbusb_front.c: sys/xen/xenbus/xenbusvar.h: sys/xen/xenstore/xenstore.c: sys/xen/xenstore/xenstore_dev.c: sys/xen/xenstore/xenstorevar.h: Pull common Xen OS support functions/settings into xen/xen-os.h.
sys/amd64/include/xen/xen-os.h: sys/i386/include/xen/xen-os.h: sys/xen/xen-os.h: Remove constants, macros, and functions unused in FreeBSD's Xen support.
sys/xen/xen-os.h: sys/i386/xen/xen_machdep.c: sys/x86/xen/hvm.c: Introduce new functions xen_domain(), xen_pv_domain(), and xen_hvm_domain(). These are used in favor of #ifdefs so that FreeBSD can dynamically detect and adapt to the presence of a hypervisor. The goal is to have an HVM optimized GENERIC, but more is necessary before this is possible.
sys/amd64/amd64/machdep.c: sys/dev/xen/xenpci/xenpcivar.h: sys/dev/xen/xenpci/xenpci.c: sys/x86/xen/hvm.c: sys/sys/kernel.h: Refactor magic ioport, Hypercall table and Hypervisor shared information page setup, and move it to a dedicated HVM support module.
HVM mode initialization is now triggered during the SI_SUB_HYPERVISOR phase of system startup. This currently occurs just after the kernel VM is fully setup which is just enough infrastructure to allow the hypercall table and shared info page to be properly mapped.
sys/xen/hvm.h: sys/x86/xen/hvm.c: Add definitions and a method for configuring Hypervisor event delievery via a direct vector callback.
sys/amd64/include/xen/xen-os.h: sys/x86/xen/hvm.c:
sys/conf/files: sys/conf/files.amd64: sys/conf/files.i386: Adjust kernel build to reflect the refactoring of early Xen startup code and Xen interrupt services.
sys/dev/xen/blkback/blkback.c: sys/dev/xen/blkfront/blkfront.c: sys/dev/xen/blkfront/block.h: sys/dev/xen/control/control.c: sys/dev/xen/evtchn/evtchn_dev.c: sys/dev/xen/netback/netback.c: sys/dev/xen/netfront/netfront.c: sys/xen/xenstore/xenstore.c: sys/xen/evtchn/evtchn_dev.c: sys/dev/xen/console/console.c: sys/dev/xen/console/xencons_ring.c Adjust drivers to use new xen_intr_*() API.
sys/dev/xen/blkback/blkback.c: Since blkback defers all event handling to a taskqueue, convert this task queue to a "fast" taskqueue, and schedule it via an interrupt filter. This avoids an unnecessary ithread context switch.
sys/xen/xenstore/xenstore.c: The xenstore driver is MPSAFE. Indicate as much when registering its interrupt handler.
sys/xen/xenbus/xenbus.c: sys/xen/xenbus/xenbusvar.h: Remove unused event channel APIs.
sys/xen/evtchn.h: Remove all kernel Xen interrupt service API definitions from this file. It is now only used for structure and ioctl definitions related to the event channel userland device driver.
Update the definitions in this file to match those from NetBSD. Implementing this interface will be necessary for Dom0 support.
sys/xen/evtchn/evtchnvar.h: Add a header file for implemenation internal APIs related to managing event channels event delivery. This is used to allow, for example, the event channel userland device driver to access low-level routines that typical kernel consumers of event channel services should never access.
sys/xen/interface/event_channel.h: sys/xen/xen_intr.h: Standardize on the evtchn_port_t type for referring to an event channel port id. In order to prevent low-level event channel APIs from leaking to kernel consumers who should not have access to this data, the type is defined twice: Once in the Xen provided event_channel.h, and again in xen/xen_intr.h. The double declaration is protected by __XEN_EVTCHN_PORT_DEFINED__ to ensure it is never declared twice within a given compilation unit.
sys/xen/xen_intr.h: sys/xen/evtchn/evtchn.c: sys/x86/xen/xen_intr.c: sys/dev/xen/xenpci/evtchn.c: sys/dev/xen/xenpci/xenpcivar.h: New implementation of Xen interrupt services. This is similar in many respects to the i386 PV implementation with the exception that events for bound to event channel ports (i.e. not IPI, virtual IRQ, or physical IRQ) are further optimized to avoid mask/unmask operations that aren't necessary for these edge triggered events.
Stubs exist for supporting physical IRQ binding, but will need additional work before this implementation can be fully shared between PV and HVM.
sys/amd64/amd64/mp_machdep.c: sys/i386/i386/mp_machdep.c: sys/i386/xen/mp_machdep.c sys/x86/xen/hvm.c: Add support for placing vcpu_info into an arbritary memory page instead of using HYPERVISOR_shared_info->vcpu_info. This allows the creation of domains with more than 32 vcpus.
sys/i386/i386/machdep.c: sys/i386/xen/clock.c: sys/i386/xen/xen_machdep.c: sys/i386/xen/exception.s: Add support for new event channle implementation.
|
#
254025 |
|
07-Aug-2013 |
jeff |
Replace kernel virtual address space allocation with vmem. This provides transparent layering and better fragmentation.
- Normalize functions that allocate memory to use kmem_* - Those that allocate address space are named kva_* - Those that operate on maps are named kmap_* - Implement recursive allocation handling for kmem_arena in vmem.
Reviewed by: alc Tested by: pho Sponsored by: EMC / Isilon Storage Division
|
#
251703 |
|
13-Jun-2013 |
jeff |
- Add a BIT_FFS() macro and use it to replace cpusetffs_obj()
Discussed with: attilio Sponsored by: EMC / Isilon Storage Division
|
#
250851 |
|
21-May-2013 |
kib |
Fix the hardware watchpoints on SMP amd64. Load the updated %dr registers also on other CPUs, besides the CPU which happens to execute the ddb. The debugging registers are stored in the pcpu area, together with the command which is executed by the IPI stop handler upon resume.
Reviewed by: jhb Sponsored by: The FreeBSD Foundation MFC after: 1 week
|
#
246248 |
|
02-Feb-2013 |
avg |
cpususpend_handler: mark AP as resumed only after fully setting up lapic
Reviewed by: jhb Tested by: Sergey V. Dyatko <sergey.dyatko@gmail.com>, KAHO Toshikazu <kaho@elam.kais.kyoto-u.ac.jp> MFC after: 12 days
|
#
241371 |
|
09-Oct-2012 |
attilio |
Reverts r234074,234105,234564,234723,234989,235231-235232 and part of r234247. Use, instead, the static intializer introduced in r239923 for x86 and sparc64 intr_cpus, unwinding the code to the initial version.
Reviewed by: marius
|
#
239241 |
|
13-Aug-2012 |
jhb |
Remove the deassert INIT IPI from the IPI startup sequence for APs. It is not listed in the boot sequence in the MP specification (1.4), and it is explicitly ignored on modern CPUs. It was only ever required when bootstrapping systems with external APICs (that is, SMP machines with 486s), which FreeBSD has never supported (and never will).
While here, tidy some comments and remove some banal ones.
|
#
239235 |
|
13-Aug-2012 |
jhb |
Add a 10 millisecond delay after sending the initial INIT IPI. This matches the algorithm in the MP specification (1.4). Previously we were sending out the deassert INIT IPI immediately after the initial INIT IPI was sent.
|
#
237037 |
|
13-Jun-2012 |
jkim |
- Remove unused code for CR3 and CR4. - Fix few style(9) nits while I am here.
|
#
236938 |
|
11-Jun-2012 |
iwasaki |
Share IPI init and startup code of mp_machdep.c with acpi_wakeup.c as ipi_startup().
|
#
236830 |
|
10-Jun-2012 |
iwasaki |
Some fixes for r236772.
- Remove cpuset stopped_cpus which is no longer used. - Add a short comment for cpuset suspended_cpus clearing. - Fix the un-ordered x86/acpica/acpi_wakeup.c in conf/files.amd64 and i386.
Pointed-out by: attilio@
|
#
236772 |
|
08-Jun-2012 |
iwasaki |
Add x86/acpica/acpi_wakeup.c for amd64 and i386. Difference of suspend/resume procedures are minimized among them.
common: - Add global cpuset suspended_cpus to indicate APs are suspended/resumed. - Remove acpi_waketag and acpi_wakemap from acpivar.h (no longer used). - Add some variables in acpi_wakecode.S in order to minimize the difference among amd64 and i386. - Disable load_cr3() because now CR3 is restored in resumectx().
amd64: - Add suspend/resume related members (such as MSR) in PCB. - Modify savectx() for above new PCB members. - Merge acpi_switch.S into cpu_switch.S as resumectx().
i386: - Merge(and remove) suspendctx() into savectx() in order to match with amd64 code.
Reviewed by: attilio@, acpi@
|
#
234208 |
|
13-Apr-2012 |
avg |
add actual interrupt counters to back ipi_invlcache_counts
Otherwise one could run into a panic with COUNT_IPIS when cache invalidation actually happened.
Reviewed by: jhb MFC after: 1 week
|
#
234074 |
|
09-Apr-2012 |
attilio |
BSP is not added to the mask of valid target CPUs for interrupts in set_apic_interrupt_ids(). Besides, set_apic_interrupts_ids() is not called in the !SMP case too. Fix this by: - Adding the BSP as an interrupt target directly in cpu_startup(). - Remove an obsolete optimization where the BSP are skipped in set_apic_interrupt_ids().
Reported by: jh Reviewed by: jhb MFC after: 3 days X-MFC: r233961 Pointy hat to: me
|
#
233704 |
|
30-Mar-2012 |
jkim |
Re-initialize model-specific MSRs when we resume CPUs.
MFC after: 1 week
|
#
231797 |
|
15-Feb-2012 |
jkim |
Clean up RFLAG and CR3 register handling and nearby comments. For BSP, use spinlock_enter()/spinlock_exit() to save/restore RFLAGS. We know interrupt is disabled when returning from S3. For AP, we do not have to save/restore it because IRET will do it for us any way. Do not save CR3 locally because savectx() does it and BSP does not have to switch to kernel map for amd64. Change contigmalloc(9) flag while I am in the neighborhood.
|
#
230426 |
|
21-Jan-2012 |
kib |
Add support for the extended FPU states on amd64, both for native 64bit and 32bit ABIs. As a side-effect, it enables AVX on capable CPUs.
In particular:
- Query the CPU support for XSAVE, list of the supported extensions and the required size of FPU save area. The hw.use_xsave tunable is provided for disabling XSAVE, and hw.xsave_mask may be used to select the enabled extensions.
- Remove the FPU save area from PCB and dynamically allocate the (run-time sized) user save area on the top of the kernel stack, right above the PCB. Reorganize the thread0 PCB initialization to postpone it after BSP is queried for save area size.
- The dumppcb, stoppcbs and susppcbs now do not carry the FPU state as well. FPU state is only useful for suspend, where it is saved in dynamically allocated suspfpusave area.
- Use XSAVE and XRSTOR to save/restore FPU state, if supported and enabled.
- Define new mcontext_t flag _MC_HASFPXSTATE, indicating that mcontext_t has a valid pointer to out-of-struct extended FPU state. Signal handlers are supplied with stack-allocated fpu state. The sigreturn(2) and setcontext(2) syscall honour the flag, allowing the signal handlers to inspect and manipilate extended state in the interrupted context.
- The getcontext(2) never returns extended state, since there is no place in the fixed-sized mcontext_t to place variable-sized save area. And, since mcontext_t is embedded into ucontext_t, makes it impossible to fix in a reasonable way. Instead of extending getcontext(2) syscall, provide a sysarch(2) facility to query extended FPU state.
- Add ptrace(2) support for getting and setting extended state; while there, implement missed PT_I386_{GET,SET}XMMREGS for 32bit binaries.
- Change fpu_kern KPI to not expose struct fpu_kern_ctx layout to consumers, making it opaque. Internally, struct fpu_kern_ctx now contains a space for the extended state. Convert in-kernel consumers of fpu_kern KPI both on i386 and amd64.
First version of the support for AVX was submitted by Tim Bird <tim.bird am sony com> on behalf of Sony. This version was written from scratch.
Tested by: pho (previous version), Yamagi Burmeister <lists yamagi org> MFC after: 1 month
|
#
227309 |
|
07-Nov-2011 |
ed |
Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs.
The SYSCTL_NODE macro defines a list that stores all child-elements of that node. If there's no SYSCTL_DECL macro anywhere else, there's no reason why it shouldn't be static.
|
#
223758 |
|
04-Jul-2011 |
attilio |
With retirement of cpumask_t and usage of cpuset_t for representing a mask of CPUs, pc_other_cpus and pc_cpumask become highly inefficient.
Remove them and replace their usage with custom pc_cpuid magic (as, atm, pc_cpumask can be easilly represented by (1 << pc_cpuid) and pc_other_cpus by (all_cpus & ~(1 << pc_cpuid))).
This change is not targeted for MFC because of struct pcpu members removal and dependency by cpumask_t retirement.
MD review by: marcel, marius, alc Tested by: pluknet MD testing by: marcel, marius, gonzo, andreast
|
#
222853 |
|
08-Jun-2011 |
avg |
remove code for dynamic offlining/onlining of CPUs on x86
The code has definitely been broken for SCHED_ULE, which is a default scheduler. It may have been broken for SCHED_4BSD in more subtle ways, e.g. with manually configured CPU affinities and for interrupt devilery purposes. We still provide a way to disable individual CPUs or all hyperthreading "twin" CPUs before SMP startup. See the UPDATING entry for details.
Interaction between building CPU topology and disabling CPUs still remains fuzzy: topology is first built using all availble CPUs and then the disabled CPUs should be "subtracted" from it. That doesn't work well if the resulting topology becomes non-uniform.
This work is done in cooperation with Attilio Rao who in addition to reviewing also provided parts of code.
PR: kern/145385 Discussed with: gcooper, ambrisko, mdf, sbruno Reviewed by: attilio Tested by: pho, pluknet X-MFC after: never
|
#
222813 |
|
07-Jun-2011 |
attilio |
etire the cpumask_t type and replace it with cpuset_t usage.
This is intended to fix the bug where cpu mask objects are capped to 32. MAXCPU, then, can now arbitrarely bumped to whatever value. Anyway, as long as several structures in the kernel are statically allocated and sized as MAXCPU, it is suggested to keep it as low as possible for the time being.
Technical notes on this commit itself: - More functions to handle with cpuset_t objects are introduced. The most notable are cpusetobj_ffs() (which calculates a ffs(3) for a cpuset_t object), cpusetobj_strprint() (which prepares a string representing a cpuset_t object) and cpusetobj_strscan() (which creates a valid cpuset_t starting from a string representation). - pc_cpumask and pc_other_cpus are target to be removed soon. With the moving from cpumask_t to cpuset_t they are now inefficient and not really useful. Anyway, for the time being, please note that access to pcpu datas is protected by sched_pin() in order to avoid migrating the CPU while reading more than one (possible) word - Please note that size of cpuset_t objects may differ between kernel and userland. While this is not directly related to the patch itself, it is good to understand that concept and possibly use the patch as a reference on how to deal with cpuset_t objects in userland, when accessing kernland members. - KTR_CPUMASK is changed and now is represented through a string, to be set as the example reported in NOTES.
Please additively note that no MAXCPU is bumped in this patch, but private testing has been done until to MAXCPU=128 on a real 8x8x2(htt) machine (amd64).
Please note that the FreeBSD version is not yet bumped because of the upcoming pcpu changes. However, note that this patch is not targeted for MFC.
People to thank for the time spent on this patch: - sbruno, pluknet and Nicholas Esborn (nick AT desert DOT net) tested several revision of the patches and really helped in improving stability of this work. - marius fixed several bugs in the sparc64 implementation and reviewed patches related to ktr. - jeff and jhb discussed the basic approach followed. - kib and marcel made targeted review on some specific part of the patch. - marius, art, nwhitehorn and andreast reviewed MD specific part of the patch. - marius, andreast, gonzo, nwhitehorn and jceel tested MD specific implementations of the patch. - Other people have made contributions on other patches that have been already committed and have been listed separately.
Companies that should be mentioned for having participated at several degrees: - Yahoo! for having offered the machines used for testing on big count of CPUs. - The FreeBSD Foundation for having sponsored my devsummit attendance, which has been instrumental. - Sandvine for having offered offices and infrastructure during development.
(I really hope I didn't forget anyone, if it happened I apologize in advance).
|
#
222756 |
|
06-Jun-2011 |
avg |
don't use cpuid level 4 in x86 cpu topology detection if it's not supported
This regression was introduced in r213323. There are probably no Intel cpus that support amd64 mode, but do not support cpuid level 4, but it's better to keep i386 and amd64 versions of this code in sync.
Discovered by: pho Tested by: pho MFC after: 2 weeks
|
#
221527 |
|
06-May-2011 |
avg |
prepare code that does topology detection for amd cpus for bulldozer
This also introduces a new detection path for family 10h and newer pre-bulldozer cpus, pre-10h hardware should not be affected.
Tested by: Gary Jennejohn <gljennjohn@googlemail.com> (with pre-10h hardware) MFC after: 2 weeks
|
#
220058 |
|
27-Mar-2011 |
alc |
Amd64 doesn't have a lazypmap ipi.
|
#
217368 |
|
13-Jan-2011 |
mdf |
Fix up a few more sysctl(9) mis-typing found in various LINT builds.
|
#
215753 |
|
23-Nov-2010 |
jkim |
Reinitialize PAT MSR via pmap_init_pat() while resuming. This function does better job since r215703 and it is safer now.
|
#
215002 |
|
08-Nov-2010 |
jhb |
A few small style and whitespace fixes.
|
#
214774 |
|
04-Nov-2010 |
avg |
x86 topo_probe: do not probe smp topology if only one cpu is visible
This could lead to a division by zero if hardware is multi-core and/or multi-threaded, but for some (quite unusual) reason FreeBSD sees only one logical processor. This could happen, for example, if neither MADT nor MP Table are presented by BIOS.
Also: - assert in topo_probe_0x4 that BSP is accounted for - neither cpu_cores nor cpu_logical should be zero after successful probing, so either being zero is an indication of failed probing
Reported by: vwe, Dan Allen <danallen46@airwired.net> Tested by: Dan Allen <danallen46@airwired.net> MFC after: 3 days
|
#
214631 |
|
01-Nov-2010 |
jhb |
Move <machine/apicreg.h> to <x86/apicreg.h>.
|
#
214630 |
|
01-Nov-2010 |
jhb |
Move the <machine/mca.h> header to <x86/mca.h>.
|
#
213323 |
|
01-Oct-2010 |
avg |
i386 and amd64 mp_machdep: improve topology detection for Intel CPUs
This patch is significantly based on previous work by jkim. List of changes: - added comments that describe topology uniformity assumption - added reference to Intel Processor Topology Enumeration article - documented a few global variables that describe topology - retired weirdly set and used logical_cpus variable - changed fallback code for mp_ncpus > 0 case, so that CPUs are treated as being different packages rather than cores in a single package - moved AMD-specific code to topo_probe_amd [jkim] - in topo_probe_0x4() follow Intel-prescribed procedure of deriving SMT and core masks and match APIC IDs against those masks [started by jkim] - in topo_probe_0x4() drop code for double-checking topology parameters by looking at L1 cache properties [jkim] - in topo_probe_0xb() add fallback path to topo_probe_0x4() as prescribed by Intel [jkim]
Still to do: - prepare for upcoming AMD CPUs by using new mechanism of uniform topology description [pointed by jkim] - probe cache topology in addition to CPU topology and probably use that for scheduler affinity topology; e.g. Core2 Duo and Athlon II X2 have the same CPU topology, but Athlon cores do not share L2 cache while Core2's do (no L3 cache in both cases) - think of supporting non-uniform topologies if they are ever implemented for platforms in question - think how to better described old HTT vs new HTT distinction, HTT vs SMT can be confusing as SMT is a generic term - more robust code for marking CPUs as "logical" and/or "hyperthreaded", use HTT mask instead of modulo operation - correct support for halting logical and/or hyperthreaded CPUs, let scheduler know that it shouldn't schedule any threads on those CPUs
PR: kern/145385 (related) In collaboration with: jkim Tested by: Sergey Kandaurov <pluknet@gmail.com>, Jeremy Chadwick <freebsd@jdc.parodius.com>, Chip Camden <sterling@camdensoftware.com>, Steve Wills <steve@mouf.net>, Olivier Smedts <olivier@gid0.org>, Florian Smeets <flo@smeets.im> MFC after: 1 month
|
#
212541 |
|
13-Sep-2010 |
mav |
Refactor timer management code with priority to one-shot operation mode. The main goal of this is to generate timer interrupts only when there is some work to do. When CPU is busy interrupts are generating at full rate of hz + stathz to fullfill scheduler and timekeeping requirements. But when CPU is idle, only minimum set of interrupts (down to 8 interrupts per second per CPU now), needed to handle scheduled callouts is executed. This allows significantly increase idle CPU sleep time, increasing effect of static power-saving technologies. Also it should reduce host CPU load on virtualized systems, when guest system is idle.
There is set of tunables, also available as writable sysctls, allowing to control wanted event timer subsystem behavior: kern.eventtimer.timer - allows to choose event timer hardware to use. On x86 there is up to 4 different kinds of timers. Depending on whether chosen timer is per-CPU, behavior of other options slightly differs. kern.eventtimer.periodic - allows to choose periodic and one-shot operation mode. In periodic mode, current timer hardware taken as the only source of time for time events. This mode is quite alike to previous kernel behavior. One-shot mode instead uses currently selected time counter hardware to schedule all needed events one by one and program timer to generate interrupt exactly in specified time. Default value depends of chosen timer capabilities, but one-shot mode is preferred, until other is forced by user or hardware. kern.eventtimer.singlemul - in periodic mode specifies how much times higher timer frequency should be, to not strictly alias hardclock() and statclock() events. Default values are 2 and 4, but could be reduced to 1 if extra interrupts are unwanted. kern.eventtimer.idletick - makes each CPU to receive every timer interrupt independently of whether they busy or not. By default this options is disabled. If chosen timer is per-CPU and runs in periodic mode, this option has no effect - all interrupts are generating.
As soon as this patch modifies cpu_idle() on some platforms, I have also refactored one on x86. Now it makes use of MONITOR/MWAIT instrunctions (if supported) under high sleep/wakeup rate, as fast alternative to other methods. It allows SMP scheduler to wake up sleeping CPUs much faster without using IPI, significantly increasing performance on some highly task-switching loads.
Tested by: many (on i386, amd64, sparc64 and powerc) H/W donated by: Gheorghe Ardelean Sponsored by: iXsystems, Inc.
|
#
211518 |
|
19-Aug-2010 |
attilio |
Revert part of the r211149 as I erroneously ported the logical_cpus from Yahoo! patchset as a mask (and according manipulating variables) while it is actually a CPU count.
Submitted by: neel MFC after: 1 month X-MFC: 211149
|
#
211292 |
|
13-Aug-2010 |
jkim |
Reset switchtime to zero rather than the current CPU ticker (TSC) value. It is more appropriate in this context because TSC MSR is reset to zero when the CPU is restarted from S3 and above. Move acpi_resync_clock() back to where it was before r211202. It does not make a difference any more.
|
#
211220 |
|
12-Aug-2010 |
attilio |
Revert r211176: As long as interrupts are disabled and there is not explicit call to sched_add() there can't be any preemption there, thus the calls may be consistent.
Reported by: kib, jhb
|
#
211202 |
|
11-Aug-2010 |
jkim |
Reset switchtime and switchticks after resynchronizing the system clock. This should fix weird runtime problem after resume on amd64. It also fixes "calcru: runtime went backwards" warnings with bootverbose.
|
#
211197 |
|
11-Aug-2010 |
jhb |
Update various places that store or manipulate CPU masks to use cpumask_t instead of int or u_int. Since cpumask_t is currently u_int on all platforms this should just be a cosmetic change.
|
#
211176 |
|
11-Aug-2010 |
attilio |
IPI handlers may run generally with interrupts disabled because they are served via an interrupt gate.
However, that doesn't explicitly prevent preemption and thread migration thus scheduler pinning may be necessary in some handlers. Fix that.
Tested by: gianni MFC after: 1 month
|
#
211151 |
|
10-Aug-2010 |
attilio |
Fix a typo due to a stale version of the patch.
Reported by: gianni, rdivacky MFC after: 1 month X-MFC: 211149
|
#
211149 |
|
10-Aug-2010 |
attilio |
Fix some places that may use cpumask_t while they still use 'int' types. While there, also fix some places assuming cpu type is 'int' while u_int is really meant.
Note: this will also fix some possible races in per-cpu data accessings to be addressed in further commits.
In collabouration with: Yahoo! Incorporated (via sbruno and peter) Tested by: gianni MFC after: 1 month
|
#
211117 |
|
09-Aug-2010 |
attilio |
Simplify the logic for handling ipi_selected() and ipi_cpu() in the amd64/i386 case.
Reviewed by: jhb Tested by: gianni MFC after: 1 month X-MFC: 210939
|
#
210939 |
|
06-Aug-2010 |
jhb |
Add a new ipi_cpu() function to the MI IPI API that can be used to send an IPI to a specific CPU by its cpuid. Replace calls to ipi_selected() that constructed a mask for a single CPU with calls to ipi_cpu() instead. This will matter more in the future when we transition from cpumask_t to cpuset_t for CPU masks in which case building a CPU mask is more expensive.
Submitted by: peter, sbruno Reviewed by: rookie Obtained from: Yahoo! (x86) MFC after: 1 month
|
#
210777 |
|
02-Aug-2010 |
jkim |
- Merge savectx2() with savectx() and struct xpcb with struct pcb. [1] savectx() is only used for panic dump (dumppcb) and kdb (stoppcbs). Thus, saving additional information does not hurt and it may be even beneficial. Unfortunately, struct pcb has grown larger to accommodate more data. Move 512-byte long pcb_user_save to the end of struct pcb while I am here. - savectx() now saves FPU state unconditionally and copy it to the PCB of FPU thread if necessary. This gives panic dump and kdb a chance to take a look at the current FPU state even if the FPU is "supposedly" not used. - Resuming CPU now unconditionally reinitializes FPU. If the saved FPU state was irrelevant, it could be in an unknown state.
Suggested by: bde [1]
|
#
210514 |
|
26-Jul-2010 |
jkim |
Re-implement FPU suspend/resume for amd64. This removes superfluous uses of critical_enter(9) and critical_exit(9) by fpugetregs() and fpusetregs(). Also, we do not touch PCB flags any more.
MFC after: 1 month
|
#
209432 |
|
22-Jun-2010 |
mav |
Some style fixes for r209371.
Submitted by: jhb@
|
#
209371 |
|
20-Jun-2010 |
mav |
Implement new event timers infrastructure. It provides unified APIs for writing event timer drivers, for choosing best possible drivers by machine independent code and for operating them to supply kernel with hardclock(), statclock() and profclock() events in unified fashion on various hardware.
Infrastructure provides support for both per-CPU (independent for every CPU core) and global timers in periodic and one-shot modes. MI management code at this moment uses only periodic mode, but one-shot mode use planned for later, as part of tickless kernel project.
For this moment infrastructure used on i386 and amd64 architectures. Other archs are welcome to follow, while their current operation should not be affected.
This patch updates existing drivers (i8254, RTC and LAPIC) for the new order, and adds event timers support into the HPET driver. These drivers have different capabilities: LAPIC - per-CPU timer, supports periodic and one-shot operation, may freeze in C3 state, calibrated on first use, so may be not exactly precise. HPET - depending on hardware can work as per-CPU or global, supports periodic and one-shot operation, usually provides several event timers. i8254 - global, limited to periodic mode, because same hardware used also as time counter. RTC - global, supports only periodic mode, set of frequencies in Hz limited by powers of 2.
Depending on hardware capabilities, drivers preferred in following orders, either LAPIC, HPETs, i8254, RTC or HPETs, LAPIC, i8254, RTC. User may explicitly specify wanted timers via loader tunables or sysctls: kern.eventtimer.timer1 and kern.eventtimer.timer2. If requested driver is unavailable or unoperational, system will try to replace it. If no more timers available or "NONE" specified for second, system will operate using only one timer, multiplying it's frequency by few times and uing respective dividers to honor hz, stathz and profhz values, set during initial setup.
|
#
209248 |
|
17-Jun-2010 |
mav |
Merge COUNT_XINVLTLB_HITS and COUNT_IPIS kernel options from i386 to amd64. This information can be very valuable for CPU sleep-time (and respectively idle power consumption) optimization.
Add counters for timer-related IPIs.
Reviewed by: jhb@ (previous version)
|
#
209212 |
|
15-Jun-2010 |
jhb |
Restore the machine check register banks on resume. For banks being monitored via CMCI, reset the interrupt threshold to 1 on resume.
Reviewed by: jkim MFC after: 2 weeks
|
#
209174 |
|
14-Jun-2010 |
jkim |
Fix ACPI suspend/resume on amd64, which was broken since r208833. We need actual storage for FPU state to save and restore.
|
#
208833 |
|
05-Jun-2010 |
kib |
Introduce the x86 kernel interfaces to allow kernel code to use FPU/SSE hardware. Caller should provide a save area that is chained into the stack of the areas; pcb save_area for usermode FPU state is on top. The pcb now contains a pointer to the current FPU saved area, used during FPUDNA handling and context switches. There is also a facility to allow the kernel thread to use pcb save_area.
Change the dreaded warnings "npxdna in kernel mode!" into the panics when FPU usage is not registered.
KPI discussed with: fabient Tested by: pho, fabient Hardware provided by: Sentex Communications MFC after: 1 month
|
#
208494 |
|
24-May-2010 |
mav |
- Implement MI helper functions, dividing one or two timer interrupts with arbitrary frequencies into hardclock(), statclock() and profclock() calls. Same code with minor variations duplicated several times over the tree for different timer drivers and architectures. - Switch all x86 archs to new functions, simplifying the code and removing extra logic from timer drivers. Other archs are also welcome.
|
#
202047 |
|
10-Jan-2010 |
alc |
Eliminate unused declarations.
|
#
198931 |
|
04-Nov-2009 |
jkim |
Tweak memory allocation for amd64 suspend/resume CPU context.
|
#
196196 |
|
13-Aug-2009 |
attilio |
* Completely Remove the option STOP_NMI from the kernel. This option has proven to have a good effect when entering KDB by using a NMI, but it completely violates all the good rules about interrupts disabled while holding a spinlock in other occasions. This can be the cause of deadlocks on events where a normal IPI_STOP is expected. * Adds an new IPI called IPI_STOP_HARD on all the supported architectures. This IPI is responsible for sending a stop message among CPUs using a privileged channel when disponible. In other cases it just does match a normal IPI_STOP. Right now the IPI_STOP_HARD functionality uses a NMI on ia32 and amd64 architectures, while on the other has a normal IPI_STOP effect. It is responsibility of maintainers to eventually implement an hard stop when necessary and possible. * Use the new IPI facility in order to implement a new userend SMP kernel function called stop_cpus_hard(). That is specular to stop_cpu() but it does use the privileged channel for the stopping facility. * Let KDB use the newly introduced function stop_cpus_hard() and leave stop_cpus() for all the other cases * Disable interrupts on CPU0 when starting the process of APs suspension. * Style cleanup and comments adding
This patch should fix the reboot/shutdown deadlocks many users are constantly reporting on mailing lists.
Please don't forget to update your config file with the STOP_NMI option removal
Reviewed by: jhb Tested by: pho, bz, rink Approved by: re (kib)
|
#
194784 |
|
23-Jun-2009 |
jeff |
Implement a facility for dynamic per-cpu variables. - Modules and kernel code alike may use DPCPU_DEFINE(), DPCPU_GET(), DPCPU_SET(), etc. akin to the statically defined PCPU_*. Requires only one extra instruction more than PCPU_* and is virtually the same as __thread for builtin and much faster for shared objects. DPCPU variables can be initialized when defined. - Modules are supported by relocating the module's per-cpu linker set over space reserved in the kernel. Modules may fail to load if there is insufficient space available. - Track space available for modules with a one-off extent allocator. Free may block for memory to allocate space for an extent.
Reviewed by: jhb, rwatson, kan, sam, grehan, marius, marcel, stas
|
#
192114 |
|
14-May-2009 |
attilio |
FreeBSD right now support 32 CPUs on all the architectures at least. With the arrival of 128+ cores it is necessary to handle more than that. One of the first thing to change is the support for cpumask_t that needs to handle more than 32 bits masking (which happens now). Some places, however, still assume that cpumask_t is a 32 bits mask. Fix that situation by using always correctly cpumask_t when needed.
While here, remove the part under STOP_NMI for the Xen support as it is broken in any case.
Additively make ipi_nmi_pending as static.
Reviewed by: jhb, kmacy Tested by: Giovanni Trematerra <giovanni dot trematerra at gmail dot com>
|
#
192050 |
|
13-May-2009 |
jhb |
Implement simple machine check support for amd64 and i386. - For CPUs that only support MCE (the machine check exception) but not MCA (i.e. Pentium), all this does is print out the value of the machine check registers and then panic when a machine check exception occurs. - For CPUs that support MCA (the machine check architecture), the support is a bit more involved. - First, there is limited support for decoding the CPU-independent MCA error codes in the kernel, and the kernel uses this to output a short description of any machine check events that occur. - When a machine check exception occurs, all of the MCx banks on the current CPU are scanned and any events are reported to the console before panic'ing. - To catch events for correctable errors, a periodic timer kicks off a task which scans the MCx banks on all CPUs. The frequency of these checks is controlled via the "hw.mca.interval" sysctl. - Userland can request an immediate scan of the MCx banks by writing a non-zero value to "hw.mca.force_scan". - If any correctable events are encountered, the appropriate details are stored in a 'struct mca_record' (defined in <machine/mca.h>). The "hw.mca.count" is a count of such records and each record may be queried via the "hw.mca.records" tree by specifying the record index (0 .. count - 1) as the next name in the MIB similar to using PIDs with the kern.proc.* sysctls. The idea is to export machine check events to userland for more detailed processing. - The periodic timer and hw.mca sysctls are only present if the CPU supports MCA.
Discussed with: emaste (briefly) MFC after: 1 month
|
#
191744 |
|
02-May-2009 |
mav |
Add support for using i8254 and rtc timers as event sources for amd64 SMP system. Redistribute hard-/stat-/profclock events to other CPUs using IPIs.
|
#
191708 |
|
30-Apr-2009 |
jkim |
- Fix divide-by-zero panic when SMP kernel is used on UP system[1]. - Avoid possible divide-by-zero panic on SMP system when the CPUID is disabled, unsupported, or buggy.
Submitted by: pluknet (pluknet at gmail dot com)[1]
|
#
191648 |
|
29-Apr-2009 |
jeff |
- Add support for cpuid leaf 0xb. This allows us to determine the topology of nehalem/corei7 based systems. - Remove the cpu_cores/cpu_logical detection from identcpu. - Describe the layout of the system in cpu_mp_announce().
Sponsored by: Nokia
|
#
191405 |
|
22-Apr-2009 |
jhb |
Adjust the way we number CPUs on x86 so that we attempt to "group" all logical CPUs in a package. We do this by numbering the non-boot CPUs by starting with the first CPU whose APIC ID is after the boot CPU and wrapping back around to APIC ID 0 if needed rather than always starting at APIC ID 0. While here, adjust the cpu_mp_announce() routine to list CPUs based on the mapping established by assign_cpu_ids() rather than making assumptions about the algorithm assign_cpu_ids() uses.
MFC after: 1 month
|
#
190620 |
|
01-Apr-2009 |
kib |
Save and restore segment registers on amd64 when entering and leaving the kernel on amd64. Fill and read segment registers for mcontext and signals. Handle traps caused by restoration of the invalidated selectors.
Implement user-mode creation and manipulation of the process-specific LDT descriptors for amd64, see sysarch(2).
Implement support for TSS i/o port access permission bitmap for amd64.
Context-switch LDT and TSS. Do not save and restore segment registers on the context switch, that is handled by kernel enter/leave trampolines now. Remove segment restore code from the signal trampolines for freebsd/amd64, freebsd/ia32 and linux/i386 for the same reason.
Implement amd64-specific compat shims for sysarch.
Linuxolator (temporary ?) switched to use gsbase for thread_area pointer.
TODO: Currently, gdb is not adapted to show segment registers from struct reg. Also, no machine-depended ptrace command is added to set segment registers for debugged process.
In collaboration with: pho Discussed with: peter Reviewed by: jhb Linuxolator tested by: dchagin
|
#
189903 |
|
16-Mar-2009 |
jkim |
Initial suspend/resume support for amd64.
This code is heavily inspired by Takanori Watanabe's experimental SMP patch for i386 and large portion was shamelessly cut and pasted from Peter Wemm's AP boot code.
|
#
189509 |
|
08-Mar-2009 |
sobomax |
Small comment nit: "run time" -> "run-time".
Submitted by: rwatson
|
#
189057 |
|
25-Feb-2009 |
sobomax |
Fix typo in comments in r189023.
|
#
189023 |
|
25-Feb-2009 |
sobomax |
Make machdep.hyperthreading_enabled tunable working with the SCHED_ULE. Unlike with SCHED_BSD, however, it can only be set to 0 at boot time, it's not possible to change it at runtime.
Reviewed by: jhb MFC after: 1 month
|
#
188065 |
|
03-Feb-2009 |
jkoshy |
Improve robustness of NMI handling, for NMIs recognized in kernel mode.
- Make the NMI handler run on its own stack (TSS_IST2). - Store the GSBASE value for each CPU just before the start of each NMI stack, permitting efficient retrieval using %rsp-relative addressing. - For NMIs taken from kernel mode, program MSR_GSBASE explicitly since one or both of MSR_GSBASE and MSR_KGSBASE can be potentially invalid. The current contents of MSR_GSBASE are saved and restored at exit. - For NMIs handled from user mode, continue to use 'swapgs' to load the per-CPU GSBASE.
Reviewed by: jeff Debugging help: jeff Tested by: gnn, Artem Belevich <artemb at gmail dot com>
|
#
187880 |
|
29-Jan-2009 |
jeff |
- Allocate apic vectors on a per-cpu basis. This allows us to allocate more irqs as we have more cpus. This is principally useful on systems with msi devices which may want many irqs per-cpu.
Discussed with: jhb Sponsored by: Nokia
|
#
185341 |
|
26-Nov-2008 |
jkim |
Introduce cpu_vendor_id and replace a lot of strcmp(cpu_vendor, "...").
Reviewed by: jhb, peter (early amd64 version)
|
#
183439 |
|
28-Sep-2008 |
marius |
Remove ipi_all() and ipi_self() as the former hasn't been used at all to date and the latter also is only used in ia64 and powerpc code which no longer serves a real purpose after bring-up and just can be removed as well. Note that architectures like sun4u also provide no means of implementing IPI'ing a CPU itself natively in the first place.
Suggested by: jhb Reviewed by: arch, grehan, jhb
|
#
182868 |
|
08-Sep-2008 |
kib |
The pcb_gs32p should be per-cpu, not per-thread pointer. This is location in GDT where the segment descriptor from pcb_gs32sd is copied, and the location is in GDT local to CPU.
Noted and reviewed by: peter MFC after: 1 week
|
#
182867 |
|
08-Sep-2008 |
kib |
Provide private per-CPU GDTs on amd64. This is required at least for the linux CB_GS32BIT to work.
Noted by: nox Reviewed by: peter MFC after: 1 week
|
#
180393 |
|
09-Jul-2008 |
peter |
Band-aid a problem with 32 bit selector setup.
Initialize %ds, %es, and %fs during CPU startup. Otherwise a garbage value could leak to a 32-bit process if a process migrated to a different CPU after exec and the new CPU had never exec'd a 32-bit process.
A more complete fix is needed, but this mitigates the most frequent manifestations.
Obtained from: ups
|
#
180255 |
|
04-Jul-2008 |
alc |
Eliminate an unused declaration. (In fact, the declaration is bogus because the variable is defined static to pmap.c on i386.)
Found by: CScout
|
#
177006 |
|
09-Mar-2008 |
jeff |
- Rather than repeating the same preemption code everywhere call the scheduler specific sched_preempt() routine.
|
#
176734 |
|
02-Mar-2008 |
jeff |
- Remove the old smp cpu topology specification with a new, more flexible tree structure that encodes the level of cache sharing and other properties. - Provide several convenience functions for creating one and two level cpu trees as well as a default flat topology. The system now always has some topology. - On i386 and amd64 create a seperate level in the hierarchy for HTT and multi-core cpus. This will allow the scheduler to intelligently load balance non-uniform cores. Presently we don't detect what level of the cache hierarchy is shared at each level in the topology. - Add a mechanism for testing common topologies that have more information than the MD code is able to provide via the kern.smp.topology tunable. This should be considered a debugging tool only and not a stable api.
Sponsored by: Nokia
|
#
171702 |
|
02-Aug-2007 |
peter |
Move mp_topology() from apic_init(i386) and apic_setup_local(amd64) to cpu_start_mp(). This is after we have read the cpuid registers to calculate the hyperthreading_cpus value for the sysctl that enables or disables hyperthread cores. Change mp_topology() to use that information rather than trying to do it itself.
This solves the problem of ULE being incorrectly told that dual core Athlon64 X2 or Operton cpus are hyperthreading cores. At the very least, we now have a single piece of code to identify hyperthreading.
Obtained from: jhb Approved by: re (kensmith)
|
#
170303 |
|
04-Jun-2007 |
jeff |
Commit 10/14 of sched_lock decomposition. - Use sched_throw() rather than replicating the same cpu_throw() code for each architecture. This also allows the scheduler to use any locking it may want to. - Use the thread_lock() rather than sched_lock when preempting. - The scheduler lock is not required to synchronize release_aps.
Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
|
#
169731 |
|
19-May-2007 |
kan |
Remove extern struct pcpu __pcpu[]; from the header file and move it the the only file where it appears to be used.
|
#
169395 |
|
08-May-2007 |
jhb |
Handle CPUs with APIC IDs higher than 32 (at least one IBM server uses an APIC ID of 38 for its second CPU): - Add a new MAX_APIC_ID constant for the highest valid APIC ID for modern systems. - Size the various arrays in the MADT, MP Table, and SMP code that are indexed by APIC IDs to allow for up to MAX_APIC_ID. - Explicitly go through and assign logical cpu ids to local APICs before starting any of the APs up rather than doing it while starting up the APs. This step is now where we honor MAXCPU.
MFC after: 1 week
|
#
167273 |
|
06-Mar-2007 |
jhb |
Change the x86 interrupt code to use FreeBSD CPU IDs (i.e. PCPU_GET(cpuid)) rather than local APIC IDs to keep track of CPUs which can handle interrupts.
|
#
167250 |
|
05-Mar-2007 |
alc |
Acquiring smp_ipi_mtx on every call to pmap_invalidate_*() is wasteful. For example, during a buildworld more than half of the calls do not generate an IPI because the only TLB entry invalidated is on the calling processor. This revision pushes down the acquisition and release of smp_ipi_mtx into smp_tlb_shootdown() and smp_targeted_tlb_shootdown() and instead uses sched_pin() and sched_unpin() in pmap_invalidate_*() so that thread migration doesn't lead to a missed TLB invalidation.
Reviewed by: jhb MFC after: 3 weeks
|
#
166569 |
|
08-Feb-2007 |
jhb |
Don't send interrupts to CPUs disabled via lapic hints.
Reported by: Ludger Bolmerg <lbolmerg ! web.de> MFC after: 3 days Pointy hat to: jhb
|
#
166187 |
|
23-Jan-2007 |
jeff |
- Allow the schedulers to IPI_PREEMPT idlethread. This puts the decision for this behavior on the initiator side.
|
#
165929 |
|
10-Jan-2007 |
jeff |
- Use the correct test in the ipi bitmask handler for IPI_PREEMPT so that we actually issue preemptions. - Remove the #ifdef IPI_PREEMPTION so it is always compiled in. Leave the option which optionally enables support in sched_4bsd. sched_ule.c will soon use this functionality as a run time rather than compile time option. - Compare against the idlethread rather than the priority. There are some idle prio tasks that we can preempt.
Discussed with: ups Tested on: i386, amd64
|
#
163219 |
|
10-Oct-2006 |
jhb |
Change the x86 interrupt code to suspend/resume interrupt controllers (PICs) rather than interrupt sources. This allows interrupt controllers with no interrupt pics (such as the 8259As when APIC is in use) to participate in suspend/resume. - Always register the 8259A PICs even if we don't use any of their pins. - Explicitly reset the 8259As on resume on amd64 if 'device atpic' isn't included. - Add a "dummy" PIC for the local APIC on the BSP to reset the local APIC on resume. This gets suspend/resume working with APIC on UP systems. SMP still needs more work to bring the APs back to life.
The MFC after is tentative.
Tested by: anholt (i386) Submitted by: Andrea Bittau <a.bittau at cs.ucl.ac.uk> (3) MFC after: 1 week
|
#
162232 |
|
11-Sep-2006 |
jhb |
Actually hook up the IPI_INVLCACHE IDT vectors backing pmap_invalidate_cache() in the SMP case so pmap_mapdev() in multiuser doesn't panic with a trap 30. I broke this many months ago when I added pmap_invalidate_cache() as early parts of the PAT work.
Patience from: jmg Pointy hat: jhb
|
#
158651 |
|
16-May-2006 |
phk |
Since DELAY() was moved, most <machine/clock.h> #includes have been unnecessary.
|
#
158238 |
|
01-May-2006 |
jhb |
Add various constants for the PAT MSR and the PAT PTE and PDE flags. Initialize the PAT MSR during boot to map PAT type 2 to Write-Combining (WC) instead of Uncached (UC-).
MFC after: 1 month
|
#
158236 |
|
01-May-2006 |
jhb |
Add a new 'pmap_invalidate_cache()' to flush the CPU caches via the wbinvd() instruction. This includes a new IPI so that all CPU caches on all CPUs are flushed for the SMP case.
MFC after: 1 month
|
#
158007 |
|
24-Apr-2006 |
jkim |
Check if reported HTT cores are physical cores. This commit does not affect AMD CPUs at all because HTT bit is disabled earlier. Intel multicore CPUs and ULE scheduler may be affected.
|
#
158000 |
|
24-Apr-2006 |
cperciva |
Adjust dangerous-shared-cache-detection logic from "all shared data caches are dangerous" to "a shared L1 data cache is dangerous". This is a compromise between paranoia and performance: Unlike the L1 cache, nobody has publicly demonstrated a cryptographic side channel which exploits the L2 cache -- this is harder due to the larger size, lower bandwidth, and greater associativity -- and prohibiting shared L2 caches turns Intel Core Duo processors into Intel Core Solo processors.
As before, the 'machdep.hyperthreading_allowed' sysctl will allow even the L1 data cache to be shared.
Discussed with: jhb, scottl Security: See FreeBSD-SA-05:09.htt for background material.
|
#
156847 |
|
18-Mar-2006 |
ups |
Enable global pages TLB extension on Application Processors.
MFC after: 3 days
|
#
156504 |
|
09-Mar-2006 |
jhb |
Flip the switch and don't route interrupts to hyperthreads in a HT system. In at least one benchmark this showed around a 20% performance increase. If other workloads do benefit from having hyperthreads service interrupts, we can always make this a loader tunable.
MFC after: 3 days Tested by: ps
|
#
156124 |
|
28-Feb-2006 |
jhb |
Rework how we wire up interrupt sources to CPUs: - Throw out all of the logical APIC ID stuff. The Intel docs are somewhat ambiguous, but it seems that the "flat" cluster model we are currently using is only supported on Pentium and P6 family CPUs. The other "hierarchy" cluster model that is supported on all Intel CPUs with local APICs is severely underdocumented. For example, it's not clear if the OS needs to glean the topology of the APIC hierarchy from somewhere (neither ACPI nor MP Table include it) and setup the logical clusters based on the physical hierarchy or not. Not only that, but on certain Intel chipsets, even though there were 4 CPUs in a logical cluster, all the interrupts were only sent to one CPU anyway. - We now bind interrupts to individual CPUs using physical addressing via the local APIC IDs. This code has also moved out of the ioapic PIC driver and into the common interrupt source code so that it can be shared with MSI interrupt sources since MSI is addressed to APICs the same way that I/O APIC pins are. - Interrupt source classes grow a new method pic_assign_cpu() to bind an interrupt source to a specific local APIC ID. - The SMP code now tells the interrupt code which CPUs are avaiable to handle interrupts in a simpler and more intuitive manner. For one thing, it means we could now choose to not route interrupts to HT cores if we wanted to (this code is currently in place in fact, but under an #if 0 for now). - For now we simply do static round-robin of IRQs to CPUs when the first interrupt handler just as before, with the change that IRQs are now bound to individual CPUs rather than groups of up to 4 CPUs. - Because the IRQ to CPU mapping has now been moved up a layer, it would be easier to manage this mapping from higher levels. For example, we could allow drivers to specify a CPU affinity map for their interrupts, or we could allow a userland tool to bind IRQs to specific CPUs.
The MFC is tentative, but I want to see if this fixes problems some folks had with UP APIC kernels on 6.0 on SMP machines (an SMP kernel would work fine, but a UP APIC kernel (such as GENERIC in RELENG_6) would lose interrupts).
MFC after: 1 week
|
#
155444 |
|
07-Feb-2006 |
phk |
Modify the way we account for CPU time spent (step 1)
Keep track of time spent by the cpu in various contexts in units of "cputicks" and scale to real-world microsec^H^H^H^H^H^H^H^Hclock_t only when somebody wants to inspect the numbers.
For now "cputicks" are still derived from the current timecounter and therefore things should by definition remain sensible also on SMP machines. (The main reason for this first milestone commit is to verify that hypothesis.)
On slower machines, the avoided multiplications to normalize timestams at every context switch, comes out as a 5-7% better score on the unixbench/context1 microbenchmark. On more modern hardware no change in performance is seen.
|
#
153666 |
|
22-Dec-2005 |
jhb |
Tweak how the MD code calls the fooclock() methods some. Instead of passing a pointer to an opaque clockframe structure and requiring the MD code to supply CLKF_FOO() macros to extract needed values out of the opaque structure, just pass the needed values directly. In practice this means passing the pair (usermode, pc) to hardclock() and profclock() and passing the boolean (usermode) to hardclock_cpu() and hardclock_process(). Other details: - Axe clockframe and CLKF_FOO() macros on all architectures. Basically, all the archs were taking a trapframe and converting it into a clockframe one way or another. Now they can just extract the PC and usermode values directly out of the trapframe and pass it to fooclock(). - Renamed hardclock_process() to hardclock_cpu() as the latter is more accurate. - On Alpha, we now run profclock() at hz (profhz == hz) rather than at the slower stathz. - On Alpha, for the TurboLaser machines that don't have an 8254 timecounter, call hardclock() directly. This removes an extra conditional check from every clock interrupt on Alpha on the BSP. There is probably room for even further pruning here by changing Alpha to use the simplified timecounter we use on x86 with the lapic timer since we don't get interrupts from the 8254 on Alpha anyway. - On x86, clkintr() shouldn't ever be called now unless using_lapic_timer is false, so add a KASSERT() to that affect and remove a condition to slightly optimize the non-lapic case. - Change prototypeof arm_handler_execute() so that it's first arg is a trapframe pointer rather than a void pointer for clarity. - Use KCOUNT macro in profclock() to lookup the kernel profiling bucket.
Tested on: alpha, amd64, arm, i386, ia64, sparc64 Reviewed by: bde (mostly)
|
#
153241 |
|
08-Dec-2005 |
jhb |
MFi386: - Move PUSH_FRAME and POP_FRAME to asmacros.h and use PUSH_FRAME in atpic entry points. - Move PCPU_* asm macros out of the middle of the asm profiling macros. - Pass IRQ vector argument as an int rather than void * to reduce diffs with i386. - EOI the lapic in C for the lapic timer handler. - GC unused Xcpuast function. - Split IPI_STOP handling code of ipi_nmi_handler() out into a cpustop_handler() function and call it from Xcpustop rather than duplicating all the logic in assembly. - Fixup the list of symbols with interrupt frames in ddb traces. Xatpic_fastintr* have never existed on amd64, and the lapic timer handler and various IPI handlers were missing. - Use trapframe instead of intrframe for interrupt entry points (on amd64 the interrupt vector was already a separate argument, so the two frames were already identical) and GC intrframe.
Submitted by: peter (3)
|
#
152531 |
|
16-Nov-2005 |
jhb |
Revert a part of the previous commits to these files that made the NMI IPI_STOP handling code use atomic_readandclear() to execute the restart function on the first CPU to resume and restore the behavior of always executing the restart function on the BSP since this is in fact what the non-NMI IPI_STOP handler does. I did add back in a statement to clear the restart function pointer after it is executed to match the behavior of the non-NMI IPI_STOP handler.
|
#
151634 |
|
24-Oct-2005 |
jhb |
Rename the KDB_STOP_NMI kernel option to STOP_NMI and make it apply to all IPI_STOP IPIs. - Change the i386 and amd64 MD IPI code to send an NMI if STOP_NMI is enabled if an attempt is made to send an IPI_STOP IPI. If the kernel option is enabled, there is also a sysctl to change the behavior at runtime (debug.stop_cpus_with_nmi which defaults to enabled). This includes removing stop_cpus_nmi() and making ipi_nmi_selected() a private function for i386 and amd64. - Fix ipi_all(), ipi_all_but_self(), and ipi_self() on i386 and amd64 to properly handle bitmapped IPIs as well as IPI_STOP IPIs when STOP_NMI is enabled. - Fix ipi_nmi_handler() to execute the restart function on the first CPU that is restarted making use of atomic_readandclear() rather than assuming that the BSP is always included in the set of restarted CPUs. Also, the NMI handler didn't clear the function pointer meaning that subsequent stop and restarts could execute the function again. - Define a new macro HAVE_STOPPEDPCBS on i386 and amd64 to control the use of stoppedpcbs[] and always enable it for i386 and amd64 instead of being dependent on KDB_STOP_NMI. It works fine in both the NMI and non-NMI cases.
|
#
151631 |
|
24-Oct-2005 |
jhb |
- Various small whitespace and style nits. - Use PCPU_GET(cpumask) in preference to 1 << PCPU_GET(cpuid) in a few places.
|
#
149300 |
|
19-Aug-2005 |
pjd |
Avoid code duplication and implement bitcount32() function in systm.h only.
Reviewed by: cperciva MFC after: 3 days
|
#
148262 |
|
21-Jul-2005 |
peter |
Actually create the double fault stack page for AP cpus so that we have a chance of getting a working double fault instead of converting it to an instant triple fault reset.
|
#
147674 |
|
29-Jun-2005 |
peter |
Move the KDB_STOP_NMI option from opt_global.h to opt_kdb.h
Approved by: re
|
#
147181 |
|
09-Jun-2005 |
ups |
Add IPI support for preempting a thread on another CPU.
MFC after: 3 weeks
|
#
146551 |
|
23-May-2005 |
obrien |
Sync the style of these two files.
|
#
146457 |
|
20-May-2005 |
obrien |
Adjust the start_ap delay to match i386.
|
#
146456 |
|
20-May-2005 |
obrien |
Fix mismerge in rev 1.226: wait 5 seconds as the comment documents, not .5 seconds.
|
#
146172 |
|
12-May-2005 |
nectar |
Default hyperthreading on in -CURRENT. No seatbelts in CURRENT (^_^)
Requested by: peter, jhb
|
#
146170 |
|
12-May-2005 |
nectar |
Add a knob for disabling/enabling HTT, "machdep.hyperthreading_allowed". Default off due to information disclosure on multi-user systems.
Submitted by: cperciva Reviewed by: jhb
|
#
145727 |
|
30-Apr-2005 |
dwhite |
Implement an alternate method to stop CPUs when entering DDB. Normally we use a regular IPI vector, but this vector is blocked when interrupts are disabled. With "options KDB_STOP_NMI" and debug.kdb.stop_cpus_with_nmi set, KDB will send an NMI to each CPU instead. The code also has a context-stuffing feature which helps ddb extract the state of processes running on the stopped CPUs.
KDB_STOP_NMI is only useful with SMP and complains if SMP is not defined. This feature only applies to i386 and amd64 at the moment, but could be used on other architectures with the appropriate MD bits.
Submitted by: ups
|
#
144696 |
|
05-Apr-2005 |
cperciva |
Fully initialize the required TSS fields so that the io permission bitmap is set correctly.
Patch from: peter Security: FreeBSD-SA-05:03.amd64
|
#
144637 |
|
04-Apr-2005 |
jhb |
Divorce critical sections from spinlocks. Critical sections as denoted by critical_enter() and critical_exit() are now solely a mechanism for deferring kernel preemptions. They no longer have any affect on interrupts. This means that standalone critical sections are now very cheap as they are simply unlocked integer increments and decrements for the common case.
Spin mutexes now use a separate KPI implemented in MD code: spinlock_enter() and spinlock_exit(). This KPI is responsible for providing whatever MD guarantees are needed to ensure that a thread holding a spin lock won't be preempted by any other code that will try to lock the same lock. For now all archs continue to block interrupts in a "spinlock section" as they did formerly in all critical sections. Note that I've also taken this opportunity to push a few things into MD code rather than MI. For example, critical_fork_exit() no longer exists. Instead, MD code ensures that new threads have the correct state when they are created. Also, we no longer try to fixup the idlethreads for APs in MI code. Instead, each arch sets the initial curthread and adjusts the state of the idle thread it borrows in order to perform the initial context switch.
This change is largely a big NOP, but the cleaner separation it provides will allow for more efficient alternative locking schemes in other parts of the kernel (bare critical sections rather than per-CPU spin mutexes for per-CPU data for example).
Reviewed by: grehan, cognet, arch@, others Tested on: i386, alpha, sparc64, powerpc, arm, possibly more
|
#
142839 |
|
28-Feb-2005 |
peter |
MFi386: Bring over John's local apic timer code
|
#
140555 |
|
21-Jan-2005 |
peter |
JumboMFi386: use bitmapped IPI handler. Update elcr and default mptable config handler. Tidy up various local apic initialization.
|
#
137262 |
|
05-Nov-2004 |
peter |
MFi386 1.238 (jhb): Allow hints to disable cpus
|
#
135914 |
|
28-Sep-2004 |
peter |
MFi386: rev 1.239 - invalidate tlb after pte update
|
#
134591 |
|
01-Sep-2004 |
julian |
Give the 4bsd scheduler the ability to wake up idle processors when there is new work to be done.
MFC after: 5 days
|
#
134553 |
|
30-Aug-2004 |
peter |
Add the mp_watchdog hooks, although it locks up my SMP test box. It might be useable to somebody.
|
#
134416 |
|
27-Aug-2004 |
obrien |
s/smp_rv_mtx/smp_ipi_mtx/g
Requested by: jhb
|
#
134227 |
|
23-Aug-2004 |
peter |
Commit Doug White and Alan Cox's fix for the cross-ipi smp deadlock. We were obtaining different spin mutexes (which disable interrupts after aquisition) and spin waiting for delivery. For example, KSE processes do LDT operations which use smp_rendezvous, while other parts of the system are doing things like tlb shootdowns with a different mutex.
This patch uses the common smp_rendezvous mutex for all MD home-grown IPIs that spinwait for delivery. Having the single mutex means that the spinloop to aquire it will enable interrupts periodically, thus avoiding the cross-ipi deadlock.
Obtained from: dwhite, alc Reviewed by: jhb
|
#
133672 |
|
13-Aug-2004 |
ambrisko |
Fix the memory scaling bug when basemem was converted to Kbytes from bytes for AMD64. Otherwise the AP will be started at 640K which won't work. Bug found on a Xeon 64bit system.
|
#
133195 |
|
06-Aug-2004 |
markm |
MFi386: Fix mem device. Grrr.
|
#
133129 |
|
04-Aug-2004 |
markm |
Fix module builds for i386 and amd64.
|
#
130225 |
|
07-Jun-2004 |
peter |
Set up the fpu *after* enabling SSE mode on AP's
Submitted by: (argh, I can't find the email)
|
#
130224 |
|
07-Jun-2004 |
peter |
Initial PG_NX support (no-execute page bit) - export the rest of the cpu features (and amd's features). - turn on EFER_NXE, depending on the NX amd feature bit - reorg the identcpu stuff a bit in order to stop treating the amd features as second class features (since it is now a primary feature bit set) and make it easier to export.
|
#
129305 |
|
16-May-2004 |
peter |
Kill the LAZYPMAP ifdefs. While they worked, they didn't do anything to help the AMD cpus (which have a hardware tlb flush filter). I held off to see what the 64 bit Intel cpus did, but it doesn't seem to help much there either. Oh well, store it in the Attic.
|
#
127158 |
|
18-Mar-2004 |
obrien |
Document machdep.hlt_cpus.
Submitted by: Craig Rodrigues <rodrigc@crodrigues.org>
|
#
126735 |
|
07-Mar-2004 |
peter |
Stop depending on #include pollution from cpufunc.h
|
#
125221 |
|
29-Jan-2004 |
peter |
Merge some more changes from i386.
|
#
125174 |
|
28-Jan-2004 |
peter |
MFi386: mp_topology().
|
#
124950 |
|
25-Jan-2004 |
alc |
MFi386 revision 1.230 - Move smp_topology to subr_smp.c so that it is defined on all architectures.
|
#
123428 |
|
11-Dec-2003 |
peter |
MFi386: (jhb): Deal with MAXCPU etc correctly
|
#
123126 |
|
03-Dec-2003 |
jhb |
Fix all users of mp_maxid to use the same semantics, namely:
1) mp_maxid is a valid FreeBSD CPU ID in the range 0 .. MAXCPU - 1. 2) For all active CPUs in the system, PCPU_GET(cpuid) <= mp_maxid.
Approved by: re (scottl) Tested on: i386, amd64, alpha
|
#
123074 |
|
30-Nov-2003 |
jeff |
- Make mp_maxid reflect the same meaning as it does on other architectures. It is one past the last valid cpuid. This relied on a different bug in UMA to work properly.
Reported/Tested by: phk Approved by: rwatson
|
#
122947 |
|
21-Nov-2003 |
jhb |
- Split cpu_mp_probe() into two parts. cpu_mp_setmaxid() is still called very early (SI_SUB_TUNABLES - 1) and is responsible for setting mp_maxid. cpu_mp_probe() is now called at SI_SUB_CPU and determines if SMP is actually present and sets mp_ncpus and all_cpus. Splitting these up allows an architecture to probe CPUs later than SI_SUB_TUNABLES by just setting mp_maxid to MAXCPU in cpu_mp_setmaxid(). This could allow the CPU probing code to live in a module, for example, since modules sysinit's in modules cannot be invoked prior to SI_SUB_KLD. This is needed to re-enable the ACPI module on i386. - For the alpha SMP probing code, use LOCATE_PCS() instead of duplicating its contents in a few places. Also, add a smp_cpu_enabled() function to avoid duplicating some code. There is room for further code reduction later since much of this code is also present in cpu_mp_start(). - All archs besides i386 still set mp_maxid to the same values they set it to before this change. i386 now sets mp_maxid to MAXCPU.
Tested on: alpha, amd64, i386, ia64, sparc64 Approved by: re (scottl)
|
#
122940 |
|
21-Nov-2003 |
peter |
Cosmetic and/or trivial sync up with i386.
Approved by: re (rwatson)
|
#
122849 |
|
17-Nov-2003 |
peter |
Initial landing of SMP support for FreeBSD/amd64.
- This is heavily derived from John Baldwin's apic/pci cleanup on i386. - I have completely rewritten or drastically cleaned up some other parts. (in particular, bootstrap) - This is still a WIP. It seems that there are some highly bogus bioses on nVidia nForce3-150 boards. I can't stress how broken these boards are. I have a workaround in mind, but right now the Asus SK8N is broken. The Gigabyte K8NPro (nVidia based) is also mind-numbingly hosed. - Most of my testing has been with SCHED_ULE. SCHED_4BSD works. - the apic and acpi components are 'standard'. - If you have an nVidia nForce3-150 board, you are stuck with 'device atpic' in addition, because they somehow managed to forget to connect the 8254 timer to the apic, even though its in the same silicon! ARGH! This directly violates the ACPI spec.
|
#
122771 |
|
15-Nov-2003 |
bde |
Localized the cy driver's locking.
|
#
122572 |
|
12-Nov-2003 |
jhb |
- Move manipulation of td_intr_nesting_level out of assembly interrupt vector stubs and into the C functions they call. - Move disabling and EOIing of interrupt sources out of PIC driver entry points and into intr_execute_handlers(). Intr_execute_handlers() only disables a source for an interrupt if it is a stray interrupt or has threaded handlers. Sources with fast handlers no longer disable (mask) the source while executing the handlers. - Move the setting of clkintr_pending into intr_execute_handlers() and set the variable for any interrupt source with a vector of 0. (Should only be true for IRQ 0.) This fixes clkintr_pending in the NO_MIXED_MODE case. - Implement lapic_eoi() and use it to implement ioapic_eoi_source(). - Rename atpic_sched_ithd() to atpic_handle_intr() since it is used to handle all atpic interrupts and not just threaded ones.
Inspired by: peter's changes to amd64 in p4 (1) Requested by: bde (2)
|
#
122491 |
|
11-Nov-2003 |
jhb |
Enable HTT CPUs by default instead of halting them by default. Users should now only have HTT CPUs if they have explicitly asked for them either by enabling HyperThreading in the BIOS or by using the MPTABLE_FORCE_HTT kernel option.
|
#
122438 |
|
10-Nov-2003 |
jhb |
MFamd64 (via P4, not in CVS yet): - Use the static boot_address variable directly rather than passing it around to several functions. - Clean up a couple of magic numbers.
|
#
122156 |
|
05-Nov-2003 |
peter |
OK, this might be a bit silly, but add another popcnt() candidate.
|
#
121996 |
|
03-Nov-2003 |
jhb |
New i386 SMP code:
- The MP code no longer knows anything specific about an MP Table. Instead, the local APIC code adds CPUs via the cpu_add() function when a local APIC is enumerated by an APIC enumerator. - Don't divide the argument to mp_bootaddress() by 1024 just so that we can turn around and mulitply it by 1024 again. - We no longer panic if SMP is enabled but we are booted on a UP machine. - init_secondary(), the asm code between init_secondary() and ap_init() in mpboot.s and ap_init() have all been merged together in C into init_secondary(). - We now use the cpuid feature bits to determine if we should enable PSE, PGE, or VME on each AP. - Due to the change in the implementation of critical sections, acquire the SMP TLB mutex around a slightly larger chunk of code for TLB shootdowns. - Remove some of the debug code from the original SMP implementation that is no longer used or no longer applies to the new APIC code. - Use a temporary hack to disable the ACPI module until the SMP code has been further reorganized to allow ACPI to work as a module again. - Add a DDB command to dump the interesting contents of the IDT.
|
#
121754 |
|
30-Oct-2003 |
jhb |
Always export r_gdt and r_idt and give them extern declarations in machine/segments.h.
|
#
121481 |
|
24-Oct-2003 |
jhb |
A few whitespace and comment tweaks.
|
#
120654 |
|
01-Oct-2003 |
peter |
Commit Bosko's patch to clean up the PSE/PG_G initialization to and avoid problems with some Pentium 4 cpus and some older PPro/Pentium2 cpus. There are several problems, some documented in Intel errata. This patch: 1) moves the kernel to the second page in the PSE case. There is an errata that says that you Must Not point a 4MB page at physical address zero on older cpus. We avoided bugs here due to sheer luck. 2) sets up PSE page tables right from the start in locore, rather than trying to switch from 4K to 4M (or 2M) pages part way through the boot sequence at the same time that we're messing with PG_G.
For some reason, the pmap work over the last 18 months seems to tickle the problems, and the PAE infrastructure changes disturb the cpu bugs even more.
A couple of people have reported a problem with APM bios calls during boot. I'll work with people to get this resolved.
Obtained from: bmilekic
|
#
119941 |
|
09-Sep-2003 |
jhb |
Remove an XXX comment by using the per CPU mask added after this comment was added.
|
#
119452 |
|
25-Aug-2003 |
obrien |
Fix copyright comment & FBSDID style nits.
Requested by: bde
|
#
118832 |
|
12-Aug-2003 |
ps |
Halted CPU's should not accumulate time.
Reviewed by: jhb
|
#
117928 |
|
23-Jul-2003 |
jhb |
Use macros from apic.h to when writing to the ICR to send IPIs to startup APs rather than magic numbers.
Tested by: scottl
|
#
117385 |
|
10-Jul-2003 |
markm |
Protect lint(1) from a #error.
|
#
117372 |
|
09-Jul-2003 |
peter |
unifdef -DLAZY_SWITCH and start to tidy up the associated glue.
|
#
117006 |
|
28-Jun-2003 |
jeff |
- Construct a cpu topology map for Hyper Threading systems so that ULE may take advantage of them.
|
#
115683 |
|
02-Jun-2003 |
obrien |
Use __FBSDID().
|
#
115016 |
|
15-May-2003 |
alc |
Initialize logical_cpus_mask when the logical CPUs are enumerated in the mptable. (Previously, logical_cpus_mask was only initialized if the hyperthreading fixup was executed.)
Approved by: re (jhb) Reviewed by: ps
|
#
112993 |
|
02-Apr-2003 |
peter |
Commit a partial lazy thread switch mechanism for i386. it isn't as lazy as it could be and can do with some more cleanup. Currently its under options LAZY_SWITCH. What this does is avoid %cr3 reloads for short context switches that do not involve another user process. ie: we can take an interrupt, switch to a kthread and return to the user without explicitly flushing the tlb. However, this isn't as exciting as it could be, the interrupt overhead is still high and too much blocks on Giant still. There are some debug sysctls, for stats and for an on/off switch.
The main problem with doing this has been "what if the process that you're running on exits while we're borrowing its address space?" - in this case we use an IPI to give it a kick when we're about to reclaim the pmap.
Its not compiled in unless you add the LAZY_SWITCH option. I want to fix a few more things and get some more feedback before turning it on by default.
This is NOT a replacement for Bosko's lazy interrupt stuff. This was more meant for the kthread case, while his was for interrupts. Mine helps a little for interrupts, but his helps a lot more.
The stats are enabled with options SWTCH_OPTIM_STATS - this has been a pseudo-option for years, I just added a bunch of stuff to it.
One non-trivial change was to select a new thread before calling cpu_switch() in the first place. This allows us to catch the silly case of doing a cpu_switch() to the current process. This happens uncomfortably often. This simplifies a bit of the asm code in cpu_switch (no longer have to call choosethread() in the middle). This has been implemented on i386 and (thanks to jake) sparc64. The others will come soon. This is actually seperate to the lazy switch stuff.
Glanced at by: jake, jhb
|
#
112687 |
|
26-Mar-2003 |
ps |
Nuke options HTT infavor of machdep.hlt_logical_cpus tunable/sysctl. This keeps the logical cpu's halted in the idle loop. By default the logical cpu's are halted at startup. It is also possible to halt any cpu in the idle loop now using machdep.hlt_cpus.
Examples of how to use this: machdep.hlt_cpus=1 halt cpu0 machdep.hlt_cpus=2 halt cpu1 machdep.hlt_cpus=4 halt cpu2 machdep.hlt_cpus=3 halt cpu0,cpu1
Reviewed by: jhb, peter
|
#
112569 |
|
24-Mar-2003 |
jake |
- Add vm_paddr_t, a physical address type. This is required for systems where physical addresses larger than virtual addresses, such as i386s with PAE. - Use this to represent physical addresses in the MI vm system and in the i386 pmap code. This also changes the paddr parameter to d_mmap_t. - Fix printf formats to handle physical addresses >4G in the i386 memory detection code, and due to kvtop returning vm_paddr_t instead of u_long.
Note that this is a name change only; vm_paddr_t is still the same as vm_offset_t on all currently supported platforms.
Sponsored by: DARPA, Network Associates Laboratories Discussed with: re, phk (cdevsw change)
|
#
111878 |
|
04-Mar-2003 |
jhb |
Wrap the hyperthreading support code with the HTT kernel option. Hyperthreading support is now off unless the HTT option is added.
MFC-after: 3 days
|
#
111428 |
|
24-Feb-2003 |
nyan |
The mpbiosreason variable does not used for pc98.
|
#
111382 |
|
23-Feb-2003 |
tegge |
Allow machines with one CPU and a valid mp table to boot an SMP kernel.
|
#
111119 |
|
19-Feb-2003 |
imp |
Back out M_* changes, per decision of the TRB.
Approved by: trb
|
#
111002 |
|
16-Feb-2003 |
phk |
Remove #include <sys/dkstat.h>
|
#
110296 |
|
03-Feb-2003 |
jake |
Split statclock into statclock and profclock, and made the method for driving statclock based on profhz when profiling is enabled MD, since most platforms don't use this anyway. This removes the need for statclock_process, whose only purpose was to subdivide profhz, and gets the profiling clock running outside of sched_lock on platforms that implement suswintr. Also changed the interface for starting and stopping the profiling clock to do just that, instead of changing the rate of statclock, since they can now be separate.
Reviewed by: jhb, tmm Tested on: i386, sparc64
|
#
110190 |
|
01-Feb-2003 |
julian |
Reversion of commit by Davidxu plus fixes since applied.
I'm not convinced there is anything major wrong with the patch but them's the rules..
I am using my "David's mentor" hat to revert this as he's offline for a while.
|
#
109898 |
|
26-Jan-2003 |
julian |
Fix KSE related patch. Make it compile for the SMP case.. statclock_process() has changed prototypes.
|
#
109700 |
|
22-Jan-2003 |
jhb |
- Move enable_sse()'s prototype to machine/md_var.h. - Sort definition of cpu_* variables appropriately. - Move cpu_fxsr out of the magic non-BSS set of variables and stick it in the BSS along with hw_instruction_sse (make the latter static as well).
Submitted by: bde (partially)
|
#
109696 |
|
22-Jan-2003 |
jhb |
Rename cpuid_cpuinfo to cpu_procinfo. bde requested that I rename this variable to something in the cpu_* namespace since that's what all the other cpuid variables were named and cpu_procinfo is what I came up with.
Requested by: bde
|
#
109623 |
|
21-Jan-2003 |
alfred |
Remove M_TRYWAIT/M_WAITOK/M_WAIT. Callers should use 0. Merge M_NOWAIT/M_DONTWAIT into a single flag M_NOWAIT.
|
#
108914 |
|
07-Jan-2003 |
jhb |
Enumerate logical hyperthread CPUs manually if they aren't already listed in the mptable. The way this works is that we determine if the system has hyperthreading and how many logical CPU's should be in each physical CPU by using the information returned by cpuid. During the first pass of the mptable, we build a bitmask of the APIC IDs of the CPUs listed in the mptable. We then scan that bitmask to see if the CPUs are already listed by the mptable, or if there are any APIC IDs already in use that would conflict with the APIC IDs of the logical CPUs. If that test succeeds, then we fixup the count of application processors. Later on during the second pass of the mptable we create fake processor entries for logical CPUs and add them to the system.
We only need this type of fixup hack when using the mptable to enumerate CPUs. The ACPI MADT table properly enumerates all logical CPUs.
|
#
105216 |
|
16-Oct-2002 |
phk |
Be consistent about functions being static.
Spotted by: FlexeLint.
|
#
104215 |
|
30-Sep-2002 |
obrien |
Turn back on the "SMP: AP CPU #N Launched!" message on normal boots. Peter's rev 1.189 should fix the lost console on SCSI-based systems due to this message.
|
#
104175 |
|
30-Sep-2002 |
obrien |
Only print out the "SMP: AP CPU #N Launched!" message on verbose boots. The kernel printf() isn't race-free
|
#
102543 |
|
28-Aug-2002 |
peter |
OK, I have had it with losing my console because the AP's print their "I am alive!" message right as the scsi probe messages happen. This is a bit nasty, but it seems to work. At the point that we unlock the AP's, briefly wait till they are all done while we hold the console on their behalf.
|
#
99862 |
|
12-Jul-2002 |
peter |
Revive backed out pmap related changes from Feb 2002. The highlights are: - It actually works this time, honest! - Fine grained TLB shootdowns for SMP on i386. IPI's are very expensive, so try and optimize things where possible. - Introduce ranged shootdowns that can be done as a single IPI. - PG_G support for i386 - Specific-cpu targeted shootdowns. For example, there is no sense in globally purging the TLB cache for where we are stealing a page from the local unshared process on the local cpu. Use pm_active to track this. - Add some instrumentation for the tlb shootdown code. - Rip out SMP code from <machine/cpufunc.h> - Try and fix some very bogus PG_G and PG_PS interactions that were bad enough to cause vm86 bios calls to break. vm86 depended on our existing bugs and this was the cause of the VESA panics last time. - Fix the silly one-line error that caused the 'panic: bad pte' last time. - Fix a couple of other silly one-line errors that should have caused more pain than they did.
Some more work is needed: - pmap_{zero,copy}_page[_idle]. These can be done without IPI's if we have a hook in cpu_switch. - The IPI handlers need some cleanup. I have a bogus %ds load that can be avoided. - APTD handling is rather bogus and appears to be a large source of global TLB IPI shootdowns for no really good reason.
I see speedups of between 1.5% and ~4% on buildworlds in a while 1 loop. I expect to see a bigger difference when there is significant pageout activity or the system otherwise has memory shortages.
I have backed out a few optimizations that I had been using over the last few days in order to be a little more conservative. I'll revisit these again over the next few days as the dust settles.
New option: DISABLE_PG_G - In case I missed something.
|
#
99766 |
|
11-Jul-2002 |
peter |
Bah, move the invltlb counter to C code and hook a debug sysctl onto it.
|
#
94967 |
|
17-Apr-2002 |
tegge |
Fix typo in adjusted panic message.
Submitted by: cokane
|
#
94962 |
|
17-Apr-2002 |
tegge |
Update io_apic_ints array properly when revoking an irq mapping. Adjust panic message.
Submitted by: David Xu <bsddiy@yahoo.com>
|
#
93818 |
|
04-Apr-2002 |
jhb |
Change callers of mtx_init() to pass in an appropriate lock type name. In most cases NULL is passed, but in some cases such as network driver locks (which use the MTX_NETWORK_LOCK macro) and UMA zone locks, a name is used.
Tested on: i386, alpha, sparc64
|
#
93264 |
|
27-Mar-2002 |
dillon |
Compromise for critical*()/cpu_critical*() recommit. Cleanup the interrupt disablement assumptions in kern_fork.c by adding another API call, cpu_critical_fork_exit(). Cleanup the td_savecrit field by moving it from MI to MD. Temporarily move cpu_critical*() from <arch>/include/cpufunc.h to <arch>/<arch>/critical.c (stage-2 will clean this up).
Implement interrupt deferral for i386 that allows interrupts to remain enabled inside critical sections. This also fixes an IPI interlock bug, and requires uses of icu_lock to be enclosed in a true interrupt disablement.
This is the stage-1 commit. Stage-2 will occur after stage-1 has stabilized, and will move cpu_critical*() into its own header file(s) + other things. This commit may break non-i386 architectures in trivial ways. This should be temporary.
Reviewed by: core Approved by: core
|
#
93017 |
|
23-Mar-2002 |
bde |
Fixed some style bugs in the removal of __P(()). The main ones were not removing tabs before "__P((", and not outdenting continuation lines to preserve non-KNF lining up of code with parentheses. Switch to KNF formatting and/or rewrap the whole prototype in some cases.
|
#
92770 |
|
20-Mar-2002 |
alfred |
Remove __P.
|
#
91778 |
|
07-Mar-2002 |
jake |
Add needed includes of machine/smp.h, remove nested include in sys/smp.h so that inlines in machine/smp.h can use variables declared in sys/smp.h.
|
#
91673 |
|
05-Mar-2002 |
jeff |
Add a new variable mp_maxid. This is used so that per cpu datastructures may be allocated as arrays indexed by the cpu id. Previously the only reliable way to know the max cpu id was through MAXCPU. mp_ncpus isn't useful here because cpu ids may be sparsely mapped, although x86 and alpha do not do this.
Also, call cpu_mp_probe much earlier so the max cpu id is known before the VM starts up. This is intended to help support per cpu queues for the new allocator, but may be useful elsewhere.
Reviewed by: jake Approved by: jake
|
#
91367 |
|
27-Feb-2002 |
peter |
Back out all the pmap related stuff I've touched over the last few days. There is some unresolved badness that has been eluding me, particularly affecting uniprocessor kernels. Turning off PG_G helped (which is a bad sign) but didn't solve it entirely. Userland programs still crashed.
|
#
91328 |
|
26-Feb-2002 |
dillon |
revert last commit temporarily due to whining on the lists.
|
#
91315 |
|
26-Feb-2002 |
dillon |
STAGE-1 of 3 commit - allow (but do not require) interrupts to remain enabled in critical sections and streamline critical_enter() and critical_exit().
This commit allows an architecture to leave interrupts enabled inside critical sections if it so wishes. Architectures that do not wish to do this are not effected by this change.
This commit implements the feature for the I386 architecture and provides a sysctl, debug.critical_mode, which defaults to 1 (use the feature). For now you can turn the sysctl on and off at any time in order to test the architectural changes or track down bugs.
This commit is just the first stage. Some areas of the code, specifically the MACHINE_CRITICAL_ENTER #ifdef'd code, is strictly temporary and will be cleaned up in the STAGE-2 commit when the critical_*() functions are moved entirely into MD files.
The following changes have been made:
* critical_enter() and critical_exit() for I386 now simply increment and decrement curthread->td_critnest. They no longer disable hard interrupts. When critical_exit() decrements the counter to 0 it effectively calls a routine to deal with whatever interrupts were deferred during the time the code was operating in a critical section.
Other architectures are unaffected.
* fork_exit() has been conditionalized to remove MD assumptions for the new code. Old code will still use the old MD assumptions in regards to hard interrupt disablement. In STAGE-2 this will be turned into a subroutine call into MD code rather then hardcoded in MI code.
The new code places the burden of entering the critical section in the trampoline code where it belongs.
* I386: interrupts are now enabled while we are in a critical section. The interrupt vector code has been adjusted to deal with the fact. If it detects that we are in a critical section it currently defers the interrupt by adding the appropriate bit to an interrupt mask.
* In order to accomplish the deferral, icu_lock is required. This is i386-specific. Thus icu_lock can only be obtained by mainline i386 code while interrupts are hard disabled. This change has been made.
* Because interrupts may or may not be hard disabled during a context switch, cpu_switch() can no longer simply assume that PSL_I will be in a consistent state. Therefore, it now saves and restores eflags.
* FAST INTERRUPT PROVISION. Fast interrupts are currently deferred. The intention is to eventually allow them to operate either while we are in a critical section or, if we are able to restrict the use of sched_lock, while we are not holding the sched_lock.
* ICU and APIC vector assembly for I386 cleaned up. The ICU code has been cleaned up to match the APIC code in regards to format and macro availability. Additionally, the code has been adjusted to deal with deferred interrupts.
* Deferred interrupts use a per-cpu boolean int_pending, and masks ipending, spending, and fpending. Being per-cpu variables it is not currently necessary to lock; bus cycles modifying them.
Note that the same mechanism will enable preemption to be incorporated as a true software interrupt without having to further hack up the critical nesting code.
* Note: the old critical_enter() code in kern/kern_switch.c is currently #ifdef to be compatible with both the old and new methodology. In STAGE-2 it will be moved entirely to MD code.
Performance issues:
One of the purposes of this commit is to enhance critical section performance, specifically to greatly reduce bus overhead to allow the critical section code to be used to protect per-cpu caches. These caches, such as Jeff's slab allocator work, can potentially operate very quickly making the effective savings of the new critical section code's performance very significant.
The second purpose of this commit is to allow architectures to enable certain interrupts while in a critical section. Specifically, the intention is to eventually allow certain FAST interrupts to operate rather then defer.
The third purpose of this commit is to begin to clean up the critical_enter()/critical_exit()/cpu_critical_enter()/ cpu_critical_exit() API which currently has serious cross pollution in MI code (in fork_exit() and ast() for example).
The fourth purpose of this commit is to provide a framework that allows kernel-preempting software interrupts to be implemented cleanly. This is currently used for two forward interrupts in I386. Other architectures will have the choice of using this infrastructure or building the functionality directly into critical_enter()/ critical_exit().
Finally, this commit is designed to greatly improve the flexibility of various architectures to manage critical section handling, software interrupts, preemption, and other highly integrated architecture-specific details.
|
#
91260 |
|
25-Feb-2002 |
peter |
Work-in-progress commit syncing up pmap cleanups that I have been working on for a while: - fine grained TLB shootdown for SMP on i386 - ranged TLB shootdowns.. eg: specify a range of pages to shoot down with a single IPI, since the IPI is very expensive. Adjust some callers that used to trigger this inside tight loops to do a ranged shootdown at the end instead. - PG_G support for SMP on i386 (options ENABLE_PG_G) - defer PG_G activation till after we decide what we are going to do with PSE and the 4MB pages at the start of the kernel. This should solve some rumored strangeness about stale PG_G entries getting stuck underneath the 4MB pages. - add some instrumentation for the fine TLB shootdown - convert some asm instruction wrappers from functions to inlines. gcc seems to do a fair bit better with this. - [temporarily!] pessimize the tlb shootdown IPI handlers. I will fix this again shortly.
This has been working fairly well for me for a while, but I have tweaked it again prior to commit since my last major testing round. The only outstanding problem that I know of is PG_G related, which is why there is an option for it (not on by default for SMP). I have seen a world speedups by a few percent (as much as 4 or 5% in one case) but I have *not* accurately measured this - I am a bit sceptical of these numbers.
|
#
91066 |
|
22-Feb-2002 |
phk |
Convert p->p_runtime and PCPU(switchtime) to bintime format.
|
#
89489 |
|
18-Jan-2002 |
peter |
Avoid __func__ string concatenation
|
#
89410 |
|
15-Jan-2002 |
peter |
Ensure that we set all the %cr0 bits to a known state for the AP's before they make it through to userland. This should fix the p5-smp problem without affecting the other cpus (eg: cyrix, see initcpu.c and the special cache handling for these cpu types).
|
#
88903 |
|
05-Jan-2002 |
peter |
Convert a bunch of 1 << PCPU_GET(cpuid) to PCPU_GET(cpumask).
|
#
88322 |
|
20-Dec-2001 |
jhb |
Introduce a standard name for the lock protecting an interrupt controller and it's associated state variables: icu_lock with the name "icu". This renames the imen_mtx for x86 SMP, but also uses the lock to protect access to the 8259 PIC on x86 UP. This also adds an appropriate lock to the various Alpha chipsets which fixes problems with Alpha SMP machines dropping interrupts with an SMP kernel.
|
#
88085 |
|
17-Dec-2001 |
jhb |
Small cleanups to the SMP code: - Axe inlvtlb_ok as it was completely redundant with smp_active. - Remove references to non-existent variable and non-existent file in i386/include/smp.h. - Don't perform initializations local to each CPU while holding the ap boot lock on i386 while an AP bootstraps itself. - Reorganize the AP startup code some to unify the latter half of the functions to bring an AP up. Eventually this might be broken out into a MI function in subr_smp.c.
|
#
87721 |
|
12-Dec-2001 |
jhb |
Axe an unneeded PCPU_SET(spinlocks, NULL) that I missed earlier.
|
#
87702 |
|
11-Dec-2001 |
jhb |
Overhaul the per-CPU support a bit:
- The MI portions of struct globaldata have been consolidated into a MI struct pcpu. The MD per-CPU data are specified via a macro defined in machine/pcpu.h. A macro was chosen over a struct mdpcpu so that the interface would be cleaner (PCPU_GET(my_md_field) vs. PCPU_GET(md.md_my_md_field)). - All references to globaldata are changed to pcpu instead. In a UP kernel, this data was stored as global variables which is where the original name came from. In an SMP world this data is per-CPU and ideally private to each CPU outside of the context of debuggers. This also included combining machine/globaldata.h and machine/globals.h into machine/pcpu.h. - The pointer to the thread using the FPU on i386 was renamed from npxthread to fpcurthread to be identical with other architectures. - Make the show pcpu ddb command MI with a MD callout to display MD fields. - The globaldata_register() function was renamed to pcpu_init() and now init's MI fields of a struct pcpu in addition to registering it with the internal array and list. - A pcpu_destroy() function was added to remove a struct pcpu from the internal array and list.
Tested on: alpha, i386 Reviewed by: peter, jake
|
#
85793 |
|
31-Oct-2001 |
mjacob |
Remove previous revision. smp_started back in subr_smp where it belongs.
|
#
85788 |
|
31-Oct-2001 |
mjacob |
Make the actual volatile int smp_started live *somewhere*. This is a temporary fix so that we can compile kernels. I waited 30 minutes for a response from the person who would likely know, but any longer is too long to wait with breakage at ToT.
|
#
84812 |
|
11-Oct-2001 |
jhb |
Add missing includes of sys/ktr.h.
|
#
83366 |
|
12-Sep-2001 |
julian |
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process.
Sorry john! (your next MFC will be a doosie!)
Reviewed by: peter@freebsd.org, dillon@freebsd.org
X-MFC after: ha ha ha ha
|
#
82309 |
|
25-Aug-2001 |
peter |
Optionize UPAGES for the i386. As part of this I split some of the low level implementation stuff out of machine/globaldata.h to avoid exposing UPAGES to lots more places. The end result is that we can double the kernel stack size with 'options UPAGES=4' etc.
This is mainly being done for the benefit of a MFC to RELENG_4 at some point. -current doesn't really need this so much since each interrupt runs on its own kstack.
|
#
80399 |
|
26-Jul-2001 |
bmilekic |
- Do not handle the per-CPU containers in mbuf code as though the cpuids were indices in a dense array. The cpuids are a sparse set and treat them as such, setting up containers only for CPUs activated during mb_init().
- Fix netstat(1) and systat(1) to treat the per-CPU stats area as a sparse map, in accordance with the above.
This allows us to properly boot with certain CPUs disactivated. However, if we later decide to re-activate said CPUs, we will barf until we decide to implement CPU spinon/spinoff callback hooks to allow for said CPUs' per-CPU containers to get configured on their activation.
Reported by: mjacob Partially (sys/ diffs) Submitted by: mjacob
|
#
79609 |
|
12-Jul-2001 |
peter |
Activate SSE/SIMD. This is the extra context switching support that we are required to do if we let user processes use the extra 128 bit registers etc.
This is the base part of the diff I got from: http://www.issei.org/issei/FreeBSD/sse.html I believe this is by: Mr. SUZUKI Issei <issei@issei.org> SMP support apparently by: Takekazu KATO <kato@chino.it.okayama-u.ac.jp> Test code by: NAKAMURA Kazushi <kaz@kobe1995.net>, see http://kobe1995.net/~kaz/FreeBSD/SSE.en.html
I have fixed a couple of style(9) deviations. I have some followup commits to fix a couple of non-style things.
|
#
78908 |
|
28-Jun-2001 |
jhb |
Get kernel profiling on SMP systems closer to working by replacing the mcount spin mutex with a very simple non-recursive spinlock implemented using atomic operations.
|
#
78425 |
|
18-Jun-2001 |
jhb |
Include sys/pcpu.h to get the prototype for globaldata_register() to quiet a warning.
|
#
76650 |
|
15-May-2001 |
jhb |
Remove unneeded includes of sys/ipl.h and machine/ipl.h.
|
#
76166 |
|
01-May-2001 |
markm |
Undo part of the tangle of having sys/lock.h and sys/mutex.h included in other "system" header files.
Also help the deprecation of lockmgr.h by making it a sub-include of sys/lock.h and removing sys/lockmgr.h form kernel .c files.
Sort sys/*.h includes where possible in affected files.
OK'ed by: bde (with reservations)
|
#
76078 |
|
27-Apr-2001 |
jhb |
Overhaul of the SMP code. Several portions of the SMP kernel support have been made machine independent and various other adjustments have been made to support Alpha SMP.
- It splits the per-process portions of hardclock() and statclock() off into hardclock_process() and statclock_process() respectively. hardclock() and statclock() call the *_process() functions for the current process so that UP systems will run as before. For SMP systems, it is simply necessary to ensure that all other processors execute the *_process() functions when the main clock functions are triggered on one CPU by an interrupt. For the alpha 4100, clock interrupts are delievered in a staggered broadcast fashion, so we simply call hardclock/statclock on the boot CPU and call the *_process() functions on the secondaries. For x86, we call statclock and hardclock as usual and then call forward_hardclock/statclock in the MD code to send an IPI to cause the AP's to execute forwared_hardclock/statclock which then call the *_process() functions. - forward_signal() and forward_roundrobin() have been reworked to be MI and to involve less hackery. Now the cpu doing the forward sets any flags, etc. and sends a very simple IPI_AST to the other cpu(s). AST IPIs now just basically return so that they can execute ast() and don't bother with setting the astpending or needresched flags themselves. This also removes the loop in forward_signal() as sched_lock closes the race condition that the loop worked around. - need_resched(), resched_wanted() and clear_resched() have been changed to take a process to act on rather than assuming curproc so that they can be used to implement forward_roundrobin() as described above. - Various other SMP variables have been moved to a MI subr_smp.c and a new header sys/smp.h declares MI SMP variables and API's. The IPI API's from machine/ipl.h have moved to machine/smp.h which is included by sys/smp.h. - The globaldata_register() and globaldata_find() functions as well as the SLIST of globaldata structures has become MI and moved into subr_smp.c. Also, the globaldata list is only available if SMP support is compiled in.
Reviewed by: jake, peter Looked over by: eivind
|
#
75724 |
|
19-Apr-2001 |
jhb |
Make the ap_boot_mtx mutex static.
|
#
75570 |
|
17-Apr-2001 |
jhb |
Blow away the panic mutex in favor of using a single atomic_cmpset() on a panic_cpu shared variable. I used a simple atomic operation here instead of a spin lock as it seemed to be excessive overhead. Also, this can avoid recursive panics if, for example, witness is broken.
|
#
75421 |
|
11-Apr-2001 |
jhb |
Rename the IPI API from smp_ipi_* to ipi_* since the smp_ prefix is just "redundant noise" and to match the IPI constant namespace (IPI_*).
Requested by: bde
|
#
75393 |
|
10-Apr-2001 |
jhb |
Remove the BETTER_CLOCK #ifdef's. The code is on by default and is here to stay for the foreseeable future.
OK'd by: peter (the idea)
|
#
75392 |
|
10-Apr-2001 |
jhb |
Add an MI API for sending IPI's. I used the same API present on the alpha because: - it used a better namespace (smp_ipi_* rather than *_ipi), - it used better constant names for the IPI's (IPI_* rather than X*_OFFSET), and - this API also somewhat exists for both alpha and ia64 already.
|
#
74912 |
|
28-Mar-2001 |
jhb |
Rework the witness code to work with sx locks as well as mutexes. - Introduce lock classes and lock objects. Each lock class specifies a name and set of flags (or properties) shared by all locks of a given type. Currently there are three lock classes: spin mutexes, sleep mutexes, and sx locks. A lock object specifies properties of an additional lock along with a lock name and all of the extra stuff needed to make witness work with a given lock. This abstract lock stuff is defined in sys/lock.h. The lockmgr constants, types, and prototypes have been moved to sys/lockmgr.h. For temporary backwards compatability, sys/lock.h includes sys/lockmgr.h. - Replace proc->p_spinlocks with a per-CPU list, PCPU(spinlocks), of spin locks held. By making this per-cpu, we do not have to jump through magic hoops to deal with sched_lock changing ownership during context switches. - Replace proc->p_heldmtx, formerly a list of held sleep mutexes, with proc->p_sleeplocks, which is a list of held sleep locks including sleep mutexes and sx locks. - Add helper macros for logging lock events via the KTR_LOCK KTR logging level so that the log messages are consistent. - Add some new flags that can be passed to mtx_init(): - MTX_NOWITNESS - specifies that this lock should be ignored by witness. This is used for the mutex that blocks a sx lock for example. - MTX_QUIET - this is not new, but you can pass this to mtx_init() now and no events will be logged for this lock, so that one doesn't have to change all the individual mtx_lock/unlock() operations. - All lock objects maintain an initialized flag. Use this flag to export a mtx_initialized() macro that can be safely called from drivers. Also, we on longer walk the all_mtx list if MUTEX_DEBUG is defined as witness performs the corresponding checks using the initialized flag. - The lock order reversal messages have been improved to output slightly more accurate file and line numbers.
|
#
74283 |
|
15-Mar-2001 |
peter |
Kill the 4MB kernel limit dead. [I hope :-)]. For UP, we were using $tmp_stk as a stack from the data section. If the kernel text section grew beyond ~3MB, the data section would be pushed beyond the temporary 4MB P==V mapping. This would cause the trampoline up to high memory to fault. The hack workaround I did was to use all of the page table pages that we already have while preparing the initial P==V mapping, instead of just the first one. For SMP, the AP bootstrap process suffered the same sort of problem and got the same treatment.
MFC candidate - this breaks on 4.x just the same..
Thanks to: Richard Todd <rmtodd@ichotolot.servalan.com>
|
#
72930 |
|
22-Feb-2001 |
peter |
Activate USER_LDT by default. The new thread libraries are going to depend on this. The linux ABI emulator tries to use it for some linux binaries too. VM86 had a bigger cost than this and it was made default a while ago.
Reviewed by: jhb, imp
|
#
72225 |
|
09-Feb-2001 |
jhb |
Woops, remove an obsolete reference to gd_cpu_lockid.
|
#
72200 |
|
09-Feb-2001 |
bmilekic |
Change and clean the mutex lock interface.
mtx_enter(lock, type) becomes:
mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized)
similarily, for releasing a lock, we now have:
mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument.
The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind.
Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two:
MTX_QUIET and MTX_NOSWITCH
The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers:
mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively.
Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case.
Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled.
Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those.
Finally, caught up to the interface changes in all sys code.
Contributors: jake, jhb, jasone (in no particular order)
|
#
71818 |
|
30-Jan-2001 |
peter |
Remove some leftovers from the CMAP* stuff in globaldata and the BSP and AP startup.
|
#
71728 |
|
28-Jan-2001 |
bmilekic |
Move the setting of curproc to idleproc up earlier in ap_init(). The problem is that a mutex lock, prior to this change, is acquired before the curproc is set to idleproc, so we mess ourselves up by calling the mutex lock routine with curproc == NULL.
Moving it up after the aps_ready spin-wait has us hopefully setting it after idleproc is setup.
Solved by: jake (the allmighty) :-)
|
#
71727 |
|
27-Jan-2001 |
tegge |
Defer assignment of low level interrupt handlers for PCI interrupts described in the MP table until something asks for the interrupt number later on.
|
#
71576 |
|
24-Jan-2001 |
jasone |
Convert all simplelocks to mutexes and remove the simplelock implementations.
|
#
71525 |
|
24-Jan-2001 |
jhb |
- Relocate portions of this file to get it into an order closer to that of the alpha mp_machdep.c. - Proc locking. - Catch up to the P_FOO -> PS_FOO proc flags changes. - Stick ap_init()'s prototype with the other prototypes. - Remove the Xforwardirq IPI. - Remove unused simplelocks. - Don't try to psignal() from forward_statclock(), but set the appropriate signal pending flag in p_sflag instead. - Add in KTR_SMP tracepoints for various SMP functions. (Brought over from the alpha port)
|
#
71337 |
|
21-Jan-2001 |
jake |
Make intr_nesting_level per-process, rather than per-cpu. Setup interrupt threads to run with it always >= 1, so that malloc can detect M_WAITOK from "interrupt" context. This is also necessary in order to context switch from sched_ithd() directly.
Reviewed By: peter
|
#
71321 |
|
21-Jan-2001 |
peter |
Remove APIC_INTR_DIAGNOSTIC - this has been disabled for some time now. Remove some leftovers of removed SMP options.
|
#
71318 |
|
21-Jan-2001 |
jake |
Remove the per-cpu pages used for copy and zero-ing pages of memory for SMP; just use the same ones as UP. These weren't used without holding Giant anyway, and the routines that use them would have to be protected from pre-emption to avoid migrating cpus.
|
#
71243 |
|
19-Jan-2001 |
peter |
apic_itrace_splz[] is unused
|
#
71211 |
|
18-Jan-2001 |
jhb |
Protect p_stat and p_oncpu with sched_lock in forward_signal().
|
#
70954 |
|
12-Jan-2001 |
jake |
Change return ??? to return -1 in some #if 0'ed code.
|
#
70861 |
|
10-Jan-2001 |
jake |
Use PCPU_GET, PCPU_PTR and PCPU_SET to access all per-cpu variables other then curproc.
|
#
70798 |
|
08-Jan-2001 |
jake |
Fix a warning. The type of globaldata.gd_prvspace has changed.
|
#
69658 |
|
06-Dec-2000 |
peter |
This is kind of a nasty hack, but it appears to solve the Compaq DL360 SMP problem. Compaq, in their infinite wisdom, forgot to put the IO apic intpin #0 connection to the 8259 PIC into the mptable. This hack is to look and see if intpin #0 has *no* table entry and adds a fake ExtInt entry for the remap routines to use. isa/clock.c will still test the interrupts. This entry is only ever used on an already broken system.
|
#
69578 |
|
04-Dec-2000 |
peter |
Cleanup some leftover lint from the old interrupt system. Also, while here, run up to 32 interrupt sources on APIC systems. Normalize INTREN/INTRDIS so they are the same on both UP and SMP systems rather than sometimes a macro, and sometimes a function.
Reviewed by: jhb, jakeb
|
#
69334 |
|
28-Nov-2000 |
jhb |
Don't wait forever for CPUs to stop or restart. Instead, give up after a timeout. If DIAGNOSTIC is turned on, then display a message to the console with a map of which CPUs failed to stop or restart. This gives an SMP box at least a fighting chance of getting into DDB if one of the other CPUs has interrupts disabled.
|
#
67365 |
|
20-Oct-2000 |
jhb |
Catch up to moving headers: - machine/ipl.h -> sys/ipl.h - machine/mutex.h -> sys/mutex.h
|
#
66716 |
|
06-Oct-2000 |
jhb |
- Change fast interrupts on x86 to push a full interrupt frame and to return through doreti to handle ast's. This is necessary for the clock interrupts to work properly. - Change the clock interrupts on the x86 to be fast instead of threaded. This is needed because both hardclock() and statclock() need to run in the context of the current process, not in a separate thread context. - Kill the prevproc hack as it is no longer needed. - We really need Giant when we call psignal(), but we don't want to block during the clock interrupt. Instead, use two p_flag's in the proc struct to mark the current process as having a pending SIGVTALRM or a SIGPROF and let them be delivered during ast() when hardclock() has finished running. - Remove CLKF_BASEPRI, which was #ifdef'd out on the x86 anyways. It was broken on the x86 if it was turned on since cpl is gone. It's only use was to bogusly run softclock() directly during hardclock() rather than scheduling an SWI. - Remove the COM_LOCK simplelock and replace it with a clock_lock spin mutex. Since the spin mutex already handles disabling/restoring interrupts appropriately, this also lets us axe all the *_intr() fu. - Back out the hacks in the APIC_IO x86 cpu_initclocks() code to use temporary fast interrupts for the APIC trial. - Add two new process flags P_ALRMPEND and P_PROFPEND to mark the pending signals in hardclock() that are to be delivered in ast().
Submitted by: jakeb (making statclock safe in a fast interrupt) Submitted by: cp (concept of delaying signals until ast())
|
#
66277 |
|
22-Sep-2000 |
ps |
Remove the NCPU, NAPIC, NBUS, NINTR config options. Make NAPIC, NBUS, NINTR dynamic and set NCPU to a maximum of 16 under SMP.
Reviewed by: peter
|
#
65782 |
|
12-Sep-2000 |
jhb |
Clean up process accounting some more. Unfortunately, it is still not quite right on i386 as the CPU who runs statclock() doesn't have a valid clockframe to calculate statistics with.
|
#
65713 |
|
11-Sep-2000 |
jhb |
When doing statistics for statclock on other CPU's, use the other CPUs' idleproc pointers instead of our own for comparisons.
Submitted by: tegge
|
#
65620 |
|
08-Sep-2000 |
jhb |
Remove an unneeded extern declaration of cp_time.
|
#
65597 |
|
08-Sep-2000 |
jake |
Really fix USER_LDT. (Don't use currentldt as an L-value.)
|
#
65557 |
|
06-Sep-2000 |
jasone |
Major update to the way synchronization is done in the kernel. Highlights include:
* Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.)
* Per-CPU idle processes.
* Interrupts are run in their own separate kernel threads and can be preempted (i386 only).
Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
|
#
64529 |
|
11-Aug-2000 |
peter |
Clean up some low level bootstrap code:
- stop using the evil 'struct trapframe' argument for mi_startup() (formerly main()). There are much better ways of doing it. - do not use prepare_usermode() - setregs() in execve() will do it all for us as long as the p_md.md_regs pointer is set. (which is now done in machdep.c rather than init_main.c. The Alpha port did it this way all along and is much cleaner). - collect all the magic %cr0 etc register settings into one place and have the AP's call that instead of using magic numbers (!!) that keep changing over and over again. - Make it safe to call kthread_create() earlier, including during the device probe sequence. It doesn't need the callback mechanism that NetBSD's version uses. - kthreads created this way are root-less as they exist before the root filesystem is mounted. init(1) is set up so that it aquires the root pointers prior to running. If other kthreads want filesystem acccess we can make this code more generic. - set all threads start times once we have decided what time it is. - init uses a trampoline rather than the evil prepare_usermode() hack. - kern_descrip.c has a couple of tweaks to deal with forking when there is no rootdir or cwd etc. - adjust the early SYSINIT() sequence so that a few prereqisites are in place. eg: make sure the run queue is initialized before doing forks.
With this, the USB code can easily create a kthread to do the device tree discovery. (I have tested it, it works nicely).
There are still some open issues before this is truely useful. - tsleep() does not like working before the clock is running. It sort-of tries to spin wait, but it can do more useful things now. - stopping a kthread in kld code at unload time is "interesting" but we have a solution for that.
The Alpha code needs no changes for this. It already uses pretty much the same strategies, but a little cleaner.
|
#
64494 |
|
10-Aug-2000 |
tegge |
Don't skip IOAPIC id conflict detection when only one pci bus is present. PR: 20312 Reviewed by: Steve Roome <steve@sse0691.bri.hp.com>
|
#
64290 |
|
05-Aug-2000 |
tegge |
Be more verbose when changing APIC ID on an IO APIC.
Don't allow cpu entries in the MP table to contain APIC IDs out of range.
Don't write outside array boundaries if an IO APIC entry in the MP table contains an APIC ID out of range.
Assign APIC IDs for all IO APICs according to section 3.6.6 in the Intel MP spec:
- If the current APIC ID on an IO APIC doesn't conflict with other IO APICs or CPUs, that APIC ID should be used. The copy of the MP table must be updated if the corresponding APIC ID in the MP table is different.
- If the current APIC ID was in conflict with other units, the corresponding APIC ID specified in the MP table is checked for conflict.
- If a conflict is still found then fall back to using a new unique ID. The copy of the MP table must be updated.
- IDs out of range is considered to be in conflict.
During these operations, the IO_TO_ID array cannot be used, since any conflict would have caused information loss. The array is then corrected, since all APIC ID conflicts should have been resolved.
PR: 20312, 18919
|
#
61136 |
|
31-May-2000 |
msmith |
Further fixes for multiple-IO-APIC systems from Tor Egge:
Further experimentation showed that some Dell 2450 machines with the prevention kludge installed still got T_RESERVED traps. CPU interrupt vector 0x7A was observed to be triggered. This might have been the bitwise OR of two different vectors sent from each of the IOAPICs at the same time.
IOAPIC #0: 0x68 --> irq 8: RTC timer interrupt IOAPIC #1: 0x32 --> irq 18: scsi host adapter or network interface ---- 0x7a --> T_RESERVED
Both IOAPICs had ID 0.
Appendix B.3 in the MP spec indicates that the operating system is responsible for assigning unique IDs to the IOAPICs.
The enclosed patch programs the IOAPIC IDs according to the IOAPIC entries in the MP table.
Submitted by: tegge
|
#
58717 |
|
28-Mar-2000 |
dillon |
Commit major SMP cleanups and move the BGL (big giant lock) in the syscall path inward. A system call may select whether it needs the MP lock or not (the default being that it does need it).
A great deal of conditional SMP code for various deadended experiments has been removed. 'cil' and 'cml' have been removed entirely, and the locking around the cpl has been removed. The conditional separately-locked fast-interrupt code has been removed, meaning that interrupts must hold the CPL now (but they pretty much had to anyway). Another reason for doing this is that the original separate-lock for interrupts just doesn't apply to the interrupt thread mechanism being contemplated.
Modifications to the cpl may now ONLY occur while holding the MP lock. For example, if an otherwise MP safe syscall needs to mess with the cpl, it must hold the MP lock for the duration and must (as usual) save/restore the cpl in a nested fashion.
This is precursor work for the real meat coming later: avoiding having to hold the MP lock for common syscalls and I/O's and interrupt threads. It is expected that the spl mechanisms and new interrupt threading mechanisms will be able to run in tandem, allowing a slow piecemeal transition to occur.
This patch should result in a moderate performance improvement due to the considerable amount of code that has been removed from the critical path, especially the simplification of the spl*() calls. The real performance gains will come later.
Approved by: jkh Reviewed by: current, bde (exception.s) Some work taken from: luoqi's patch
|
#
55891 |
|
13-Jan-2000 |
mdodd |
Allow SMP systems with an MCA bus to work properly.
Reviewed by: peter
|
#
55540 |
|
07-Jan-2000 |
luoqi |
Allow SMP && NCPU == 1 to work. From now on, there's no restriction on the value of NCPU relative to the number of cpus physically present, the actual number of cpus utilized will be the smaller of the two.
|
#
55420 |
|
04-Jan-2000 |
tegge |
ISA device drivers use the ISA source interrupt number in locations where the low level interrupt handler number should be used. Change setup_apic_irq_mapping() to allocate low level interrupt handler X (Xintr${X}) for any ISA interrupt X mentioned in the MP table.
Remove an assumption in the driver for the system clock (clock.c) that interrupts mentioned in the MP table as delivered to IOAPIC #0 intpin Y is handled by low level interrupt handler Y (Xintr${Y}) but don't assume that low level interrupt handler 0 (Xintr0) is used.
Don't allocate two low level interrupt handlers for the system clock. Reviewed by: NOKUBI Hirotaka <hnokubi@yyy.or.jp>
|
#
53745 |
|
27-Nov-1999 |
bde |
Moved scheduling-related code to kern_synch.c so that it is easier to fix and extend. The new function containing the code is named schedclock() as in NetBSD, but it has slightly different semantics (it already handles incrementation of p->p_cpticks, and it should handle any calling frequency).
Agreed with in principle by: dufault
|
#
52271 |
|
15-Oct-1999 |
tegge |
Eliminate remaining part of incorrect PCI bus numbering sanity check on systems with more than one PCI bus.
|
#
52121 |
|
11-Oct-1999 |
peter |
Zap unneeded #includes
Submitted by: phk
|
#
50972 |
|
05-Sep-1999 |
peter |
Set up FPU state on the AP.
Tested by: phk
|
#
50477 |
|
27-Aug-1999 |
peter |
$Id$ -> $FreeBSD$
|
#
49558 |
|
09-Aug-1999 |
phk |
Merge the cons.c and cons.h to the best of my ability. alpha may or may not compile, I can't test it.
|
#
48924 |
|
20-Jul-1999 |
msmith |
Implement an all-CPU shootdown-style rendezvous facility. This allows the caller to specify a function to be guarded between an entry and exit barrier, as well as pre- and post-barrier functions.
The primary use for this function is synchronised update of per-cpu private data. The implementation is almost (but not quite) MI; with a better mechanism for masking per-CPU interrupts it could probably be hoisted.
Reviewed by: peter (partially)
|
#
48145 |
|
23-Jun-1999 |
msmith |
Changes in the way that the APs are started appears to have removed the problem with having more CPUs than NCPU.
PR: kern/4255 Submitted by: peter
|
#
48144 |
|
23-Jun-1999 |
luoqi |
Do not setup 4M pdir until all APs are up.
|
#
48119 |
|
22-Jun-1999 |
msmith |
Remove an unnecessary panic when sparse PCI bus numbering is encountered. This is found eg. on some Compaq Proliant systems.
Submitted by: peter
|
#
47678 |
|
01-Jun-1999 |
jlemon |
Unifdef VM86.
Reviewed by: silence on on -current
|
#
47081 |
|
12-May-1999 |
luoqi |
Unbreak VESA on SMP.
|
#
46703 |
|
08-May-1999 |
peter |
Make sure the mem_range_AP_init() prototype is seen where it's needed, and #ifdef SMP around it for fun.
|
#
46215 |
|
30-Apr-1999 |
msmith |
Add a hook that can be called to initialise a slave processor's memory range attributes after they have been extracted from the master.
Hook up the i686 MP code to do this for each AP.
Be more careful about printing the default memory type for the i686.
Suggestions from: luoqi
|
#
46129 |
|
27-Apr-1999 |
luoqi |
Enable vmspace sharing on SMP. Major changes are, - %fs register is added to trapframe and saved/restored upon kernel entry/exit. - Per-cpu pages are no longer mapped at the same virtual address. - Each cpu now has a separate gdt selector table. A new segment selector is added to point to per-cpu pages, per-cpu global variables are now accessed through this new selector (%fs). The selectors in gdt table are rearranged for cache line optimization. - fask_vfork is now on as default for both UP and SMP. - Some aio code cleanup.
Reviewed by: Alan Cox <alc@cs.rice.edu> John Dyson <dyson@iquest.net> Julian Elischer <julian@whistel.com> Bruce Evans <bde@zeta.org.au> David Greenman <dg@root.com>
|
#
45643 |
|
13-Apr-1999 |
tegge |
Backout early start of APs since it caused some machines to hang.
|
#
45566 |
|
10-Apr-1999 |
tegge |
Add prototype for wait_ap().
|
#
45562 |
|
10-Apr-1999 |
tegge |
Let BSP wait until all APs are initialized.
|
#
45436 |
|
07-Apr-1999 |
peter |
Disable the mtrr copy calls, it doesn't work with the i686_mem.c stuff. This should make it compile/link again.
|
#
44487 |
|
05-Mar-1999 |
bde |
The magic "no-cpu" cpu number is 0xff. Don't misrepresent cpu numbers as chars or use bogus casts in an attempt to unmisrepresnt them. In top, don't assume that 0xff is the only negative cpu number when cpu numbers are (mis)represented.
|
#
44289 |
|
26-Feb-1999 |
tegge |
Don't call assign_apic_irq with a value for irq that is out of range.
|
#
44157 |
|
19-Feb-1999 |
luoqi |
Introduce machine-dependent macro pgtok() to convert page count to number of kilobytes. Its definition for each architecture could be optimized to avoid potential numerical overflows.
|
#
44146 |
|
19-Feb-1999 |
luoqi |
Hide access to vmspace:vm_pmap with inline function vmspace_pmap(). This is the preparation step for moving pmap storage out of vmspace proper.
Reviewed by: Alan Cox <alc@cs.rice.edu> Matthew Dillion <dillon@apollo.backplane.com>
|
#
43314 |
|
27-Jan-1999 |
dillon |
Fix warnings in preparation for adding -Wall -Wcast-qual to the kernel compile
|
#
42880 |
|
20-Jan-1999 |
jkh |
Make more messages conditional on bootverbose
|
#
42543 |
|
11-Jan-1999 |
eivind |
Silence warnings.
|
#
41591 |
|
07-Dec-1998 |
archie |
The "easy" fixes for compiling the kernel -Wunused: remove unreferenced static and local variables, goto labels, and functions declared but not defined.
|
#
41367 |
|
26-Nov-1998 |
tegge |
Attempt to handle interrupts delivered to all IO APICs by using the first IO APIC with a sufficient number of pins.
|
#
41362 |
|
26-Nov-1998 |
eivind |
Staticize.
|
#
40179 |
|
10-Oct-1998 |
kato |
mp_machdep.c: Set a vector to boot code (PC-98). locore.s: Tell the bios to warmboot next time (PC-98).
|
#
40169 |
|
10-Oct-1998 |
kato |
PC-98 doesn't have CMOS ram.
|
#
40067 |
|
08-Oct-1998 |
kato |
BIOS ROM base address is 0xe8000 on PC-98.
|
#
38888 |
|
06-Sep-1998 |
tegge |
Maintain a mapping from irq number to (ioapic number, int pin) tuple, and use this when masking/unmasking interrupts.
Maintain a mapping from (iopaic number, int pin) tuple to irq number, and use this when configuring devices and programming the ioapics.
Previous code assumed that irq number was equal to int pin number, and that the ioapic number was 0.
Don't let an AP enter _cpu_switch before all local apics are initialized.
|
#
38505 |
|
24-Aug-1998 |
bde |
Fixed printf format errors. Only one left in LINT on i386's.
|
#
38422 |
|
18-Aug-1998 |
msmith |
Presently there is only one `currentldt' variable for all cpus in a SMP system. Unexpected things could happen if each cpu has a different ldt setting and one cpu tries to use value of currentldt set by another cpu.
The fix is to move currentldt to the per-cpu area. It includes patches I filed in PR i386/6219 which are also user ldt related.
PR: i386/7591, i386/6219 Submitted by: Luoqi Chen <luoqi@watermarkgroup.com>
|
#
38349 |
|
15-Aug-1998 |
bde |
pmap.c: Cast pointers to (vm_offset_t) instead of to (u_long) (as before) or to (uintptr_t)(void *) (as would be more correct). Don't cast vm_offset_t's to (u_long) just to do arithmetic on them.
mp_machdep.c: Cast pointers to (uintptr_t) instead of to (u_long). Don't forget to cast pointers to (void *) first or to recover from integral possible integral promotions, although this is too much work for machine-dependent code.
vm code generally avoids warnings for pointer vs long size mismatches by using vm_offset_t to represent pointers; pmap.c often uses plain `unsigned int' instead of vm_offset_t and didn't use u_long elsewhere, but this style was messed up by code apparently imported from mp_machdep.c.
|
#
36135 |
|
17-May-1998 |
tegge |
Add forwarding of roundrobin to other cpus. This gives a more regular update of cpu usage as shown by top when one process is cpu bound (no system calls) while the system is otherwise idle (except for top).
Don't attempt to switch to the BSP in boot(). If the system was idle when an interrupt caused a panic, this won't work. Instead, switch to the BSP in cpu_reset.
Remove some spurious forward_statclock/forward_hardclock warnings.
|
#
36125 |
|
17-May-1998 |
tegge |
For SMP, use prv_PPAGE1/prv_PMAP1 instead of PADDR1/PMAP1. get_ptbase and pmap_pte_quick no longer generates IPIs. This should reduce the number of IPIs during heavy paging.
|
#
35932 |
|
10-May-1998 |
dyson |
Attempt to set write combining mode for graphics devices.
|
#
35077 |
|
06-Apr-1998 |
peter |
Use real types for the SMP pages being allocated rather than arrays of ints. Remove some no longer needed casts. Initialize the per-cpu global data area using the structs rather than knowing too much about layout, alignment, etc.
|
#
35058 |
|
06-Apr-1998 |
phk |
Make a kernel version of the timer* functions called timerval* to be more consistent.
OK'ed by: bde
|
#
34990 |
|
01-Apr-1998 |
tegge |
Add two workarounds for broken MP tables:
- Attempt to handle PCI devices where the interrupt is an ISA/EISA interrupt according to the mp table.
- Attempt to handle multiple IO APIC pins connected to the same PCI or ISA/EISA interrupt source. Print a warning if this happens, since performance is suboptimal. This workaround is only used for PCI devices.
With these two workarounds, the -SMP kernel is capable of running on my Asus P/I-P65UP5 motherboard when version 1.4 of the MP table is disabled.
|
#
34197 |
|
07-Mar-1998 |
tegge |
The APs now reload the interrupt descriptor table pointer after f00f_hack has run.
Use the global r_idt descriptor in f00f_hack when in SMP mode, so the APs find the relocated interrupt descriptor table.
Submitted by: Partially from David A Adkins <adkin003@tc.umn.edu>
|
#
34021 |
|
03-Mar-1998 |
tegge |
When entering the apic version of slow interrupt handler, level interrupts are masked, and EOI is sent iff the corresponding ISR bit is set in the local apic. If the CPU cannot obtain the interrupt service lock (currently the global kernel lock) the interrupt is forwarded to the CPU holding that lock.
Clock interrupts now have higher priority than other slow interrupts.
|
#
34020 |
|
03-Mar-1998 |
tegge |
Forward the signal if the process runs on a different CPU. This reduces the signal handling latency for cpu-bound processes that performs very few system calls.
The IPI for forcing an additional software trap is no longer dependent upon BETTER_CLOCK being defined.
|
#
34019 |
|
03-Mar-1998 |
tegge |
Reduce timeout before assuming that forwarding of hardclock or softclock failed. Don't complain on forwarding failure, unless BETTER_CLOCK_DIAGNOSTIC is defined.
|
#
33936 |
|
01-Mar-1998 |
dyson |
1) Use a more consistent page wait methodology. 2) Do not unnecessarily force page blocking when paging pages out. 3) Further improve swap pager performance and correctness, including fixing the paging in progress deadlock (except in severe I/O error conditions.) 4) Enable vfs_ioopt=1 as a default. 5) Fix and enable the page prezeroing in SMP mode.
All in all, SMP systems especially should show a significant improvement in "snappyness."
|
#
33181 |
|
09-Feb-1998 |
eivind |
Staticize.
|
#
31723 |
|
15-Dec-1997 |
tegge |
Add support for low resolution SMP kernel profiling.
- A nonprofiling version of s_lock (called s_lock_np) is used by mcount.
- When profiling is active, more registers are clobbered in seemingly simple assembly routines. This means that some callers needed to save/restore extra registers.
- The stack pointer must have space for a 'fake' return address in idle, to avoid stack underflow.
|
#
31720 |
|
14-Dec-1997 |
tegge |
Don't forward hardclock or statclock to stopped cpus. Disable forwarding when a panic has occured.
|
#
31689 |
|
12-Dec-1997 |
tegge |
Add needed #include.
Problem found by: Bruce Evans <bde@zeta.org.au>
|
#
31639 |
|
08-Dec-1997 |
fsmp |
The improvements to clock statistics by Tor Egge Wrappered and enabled by the define BETTER_CLOCK (on by default in smpyests.h)
Reviewed by: smp@csn.net Submitted by: Tor Egge <Tor.Egge@idi.ntnu.no>
|
#
31030 |
|
07-Nov-1997 |
tegge |
Use UPAGES when setting up private pages for SMP (which includes idle stack).
|
#
30813 |
|
28-Oct-1997 |
bde |
Removed unused #includes.
|
#
30343 |
|
12-Oct-1997 |
peter |
Try and fix some style problems
|
#
30265 |
|
10-Oct-1997 |
peter |
Convert the VM86 option from a global option to an option only depended on by the files that use it. Changing the VM86 option now only causes a recompile of a dozen files or so rather than the entire kernel.
|
#
30136 |
|
06-Oct-1997 |
dyson |
It is possible that MB's with really broken bios's not set up more of the mtrr registers. This just fills in more of the registers.
|
#
30112 |
|
05-Oct-1997 |
dyson |
Make sure that the memory type registers are the same for each CPU in a P6 SMP system. Some MB bios'es don't set the registers up correctly for the AP's. Additionally, set the memory between 0xa0000 and 0xbffff as write combining.
|
#
29663 |
|
21-Sep-1997 |
peter |
Implement the parts needed for VM86 under SMP.
|
#
29655 |
|
21-Sep-1997 |
dyson |
Add support for more than 1 page of idle process stack on SMP systems.
|
#
29213 |
|
07-Sep-1997 |
fsmp |
General cleanup of the lock pushdown code. They are grouped and enabled from machine/smptests.h:
#define PUSHDOWN_LEVEL_1 #define PUSHDOWN_LEVEL_2 #define PUSHDOWN_LEVEL_3 #define PUSHDOWN_LEVEL_4_NOT
|
#
28999 |
|
01-Sep-1997 |
fsmp |
Cleanup.
|
#
28984 |
|
31-Aug-1997 |
bde |
Move closer to supporting VM86 under SMP.
LINT now compiles but doesn't link. Other link-time breakage for LINT is now visible (SMP is incompatible with SIMPLELOCK_DEBUG). Submitted by: jlemon
|
#
28951 |
|
31-Aug-1997 |
fsmp |
Debug version of simple_lock. This will store the CPU id of the holding CPU along with the lock. When a CPU fails to get the lock it compares its own id to the holder id. If they are the same it panic()s, as simple locks are binary, and this would cause a deadlock.
Controlled by smptests.h: SL_DEBUG, ON by default.
Some minor cleanup.
|
#
28921 |
|
30-Aug-1997 |
fsmp |
Another round of lock pushdown. Add a simplelock to deal with disable_intr()/enable_intr() as used in UP kernel. UP kernel expects that this is enough to guarantee exclusive access to regions of code bracketed by these 2 functions. Add a simplelock to bracket clock accesses in clock.c: clock_lock.
Help from: Bruce Evans <bde@zeta.org.au>
|
#
28809 |
|
26-Aug-1997 |
peter |
Correct some things I forgot about until it was too late with smp_active. smp_active = 1 used to indicate that the system had frozen previously started AP's, while smp_active = 0 was "AP's not yet started". I have split this into smp_started (which is set when the AP's come online), and smp_active is left for turning on/off AP scheduling.
|
#
28808 |
|
26-Aug-1997 |
peter |
Clean up the SMP AP bootstrap and eliminate the wretched idle procs.
- We now have enough per-cpu idle context, the real idle loop has been revived (cpu's halt now with nothing to do). - Some preliminary support for running some operations outside the global lock (eg: zeroing "free but not yet zeroed pages") is present but appears to cause problems. Off by default. - the smp_active sysctl now behaves differently. It's merely a 'true/false' option. Setting smp_active to zero causes the AP's to halt in the idle loop and stop scheduling processes. - bootstrap is a lot safer. Instead of sharing a statically compiled in stack a number of times (which has caused lots of problems) and then abandoning it, we use the idle context to boot the AP's directly. This should help >2 cpu support since the bootlock stuff was in doubt. - print physical apic id in traps.. helps identify private pages getting out of sync. (You don't want to know how much hair I tore out with this!)
More cleanup to follow, this is more of a checkpoint than a 'finished' thing.
|
#
28743 |
|
25-Aug-1997 |
bde |
Removed a bogus comment.
|
#
28669 |
|
24-Aug-1997 |
fsmp |
A clean fix for the spl "deadlock before smp_active" problem.
Added a new variable, 'bsp_apic_ready', which is set as soon as the bootstrap CPU has initialized its local APIC. Conditionalize the GENSPLR functions to call ss_lock ONLY after bsp_apic_ready is TRUE; This should prevent any problems with races between the time the 1st AP becomes ready and the time smp_active is set.
|
#
28487 |
|
21-Aug-1997 |
fsmp |
Made PEND_INTS default. Made NEW_STRATEGY default. Removed misc. old cruft.
Centralized simple locks into mp_machdep.c Centralized simple lock macros into param.h
More cleanup in the direction of making splxx()/cpl MP-safe.
|
#
28442 |
|
20-Aug-1997 |
fsmp |
Preperation for moving cpl into critical region access. Several new fine-grained locks. New FAST_INTR() methods: - separate simplelock for FAST_INTR, no more giant lock. - FAST_INTR()s no longer checks ipending on way out of ISR. sio made MP-safe (I hope).
|
#
28041 |
|
10-Aug-1997 |
fsmp |
Cheap fix for kern/4255. If the problem is seen this fix suggests a compile-time work-around then panics.
|
#
28027 |
|
09-Aug-1997 |
fsmp |
Some fixes towards making "default configs" work again. Still not fixed, no idea why.
Debug help from: "Thomas D. Dean" <tomdean@ix.netcom.com>
|
#
27780 |
|
31-Jul-1997 |
fsmp |
Converted the TEST_LOPRIO code to default. Created mplock functions that save/restore NO registers. Minor cleanup.
|
#
27728 |
|
28-Jul-1997 |
fsmp |
Modified the PEND_INTS algorithm to fix the ISA INT loss problem.
Noticed by: dave adkins <adkin003@gold.tc.umn.edu> and others.
|
#
27697 |
|
25-Jul-1997 |
fsmp |
mpapic.c & mp_machdep: - removed TEST_ALTTIMER. - removed APIC_PIN0_TIMER. - removed TIMER_ALL.
mplock.s: - minor update of try_mplock for new algorithm where a CPU uses try_mplock instead of get_mplock in the ISRs.
|
#
27634 |
|
23-Jul-1997 |
fsmp |
New simple_lock code in asm: - s_lock_init() - s_lock() - s_lock_try() - s_unlock()
Created lock for IO APIC and apic_imen (SMP version of imen) - imen_lock
Code to use imen_lock for access from apic_ipl.s and apic_vector.s. Moved this code *outside* of mp_lock.
It seems to work!!!
|
#
27612 |
|
22-Jul-1997 |
fsmp |
Major cleanup of APIC code around the imen variable. This is the first step towards making INTREN()/INTRDIS() MP-safe.
|
#
27563 |
|
20-Jul-1997 |
fsmp |
Developed a new strategy for handling the 8254/8259/APIC issue.
|
#
27561 |
|
20-Jul-1997 |
fsmp |
Minor cleanup. Pass string arg to apic_dump. Moved bootverbose printing of SMP enabled INTs from clock.c to autoconf.c
|
#
27535 |
|
20-Jul-1997 |
bde |
Removed unused #includes.
|
#
27523 |
|
19-Jul-1997 |
fsmp |
Added code to support #define APIC_PIN0_TIMER. This code ALWAYS runs the 8254 timer thru the 8259 ICU. It depricates the usage of "options SMP_TIMER_NC" in the config file.
|
#
27517 |
|
18-Jul-1997 |
fsmp |
Split TEST_CPUSTOP code into CPUSTOP_ON_DDBBREAK and mainline code.
|
#
27489 |
|
18-Jul-1997 |
fsmp |
printf cleanup.
|
#
27484 |
|
17-Jul-1997 |
dyson |
Hopefully fix a few problems that could cause hangs in SMP mode. 1) Make sure that the region mapped by a 4MB page is properly aligned. 2) Don't turn on the PG_G flag in locore for SMP. I plan to do that later in startup anyway. 3) Make sure the 2nd processor has PSE enabled, so that 4MB pages don't hose it.
We don't use PG_G yet on SMP -- there is work to be done to make that work correctly. It isn't that important anyway...
|
#
27353 |
|
12-Jul-1997 |
fsmp |
new code to control other CPUs: stop_cpus()/restart_cpus()/_Xstopcpu this code is controlled by smptests.h: TEST_CPUSTOP, OFF by default
new code for handling mixed-mode 8259/APIC programming without 'ExtInt' this code is controlled by smptests.h: TEST_ALTTIMER, ON by default
|
#
27289 |
|
08-Jul-1997 |
fsmp |
General cleanup of APIC code. stop_cpus()/restart_cpus() STILL not working!
|
#
27255 |
|
06-Jul-1997 |
fsmp |
stop_cpus(), currently BROKEN! (turned off in smptests.h by default). restart_cpus(), currently BROKEN! (turned off in smptests.h by default).
|
#
27005 |
|
27-Jun-1997 |
fsmp |
Added POST code output to various points of the startup code.
General cleanup.
New functions to stop/start CPUs via IPIs:
- int stop_cpus( u_int map ); - int restart_cpus( u_int map );
Turned off by default, enabled via smptests.h:TEST_CPUSTOP. Current version has a BUG, perhaps a deadlock?
|
#
27001 |
|
27-Jun-1997 |
fsmp |
Program lint1 to handle NMIs.
Till now NMIs would be ignored. Now an NMI is caught by the BSP. APs still ignore NMI, am working on code to allow a CPU to stop other CPUs via an IPI.
|
#
26950 |
|
25-Jun-1997 |
fsmp |
Merged/renamed functions:
- get_isa_apic_mask() -> isa_apic_mask() - get_isa_apic_irq() && get_eisa_apic_irq() -> isa_apic_pin() - get_pci_apic_irq() -> pci_apic_pin()
|
#
26896 |
|
24-Jun-1997 |
tegge |
Ensure that the boot CPU honours write protection in kernel mode. This fixes one of the problems noted in PR kern/3688.
|
#
26886 |
|
24-Jun-1997 |
fsmp |
Fix calculation of initial mplock value. We now use LOGICAL, not PHYSICAL, IDs to calculate the mplock.
|
#
26882 |
|
24-Jun-1997 |
fsmp |
Fixed breakage for "default" configurations in mptable_pass1().
|
#
26812 |
|
22-Jun-1997 |
peter |
Preliminary support for per-cpu data pages.
This eliminates a lot of #ifdef SMP type code. Things like _curproc reside in a data page that is unique on each cpu, eliminating the expensive macros like: #define curproc (SMPcurproc[cpunumber()])
There are some unresolved bootstrap and address space sharing issues at present, but Steve is waiting on this for other work. There is still some strictly temporary code present that isn't exactly pretty.
This is part of a larger change that has run into some bumps, this part is standalone so it should be safe. The temporary code goes away when the full idle cpu support is finished.
Reviewed by: fsmp, dyson
|
#
26379 |
|
02-Jun-1997 |
dfr |
Change isa_device.h to intr_machdep.h
|
#
26270 |
|
29-May-1997 |
fsmp |
Code such as apic_base[APIC_ID] converted to lapic__id
Changes to pmap.c for lapic_t lapic && ioapic_t ioapic pointers, currently equal to apic_base && io_apic_base, will stand alone with the private page mapping.
|
#
26266 |
|
29-May-1997 |
peter |
minor style police (recent divergence from KNF code)
|
#
26203 |
|
27-May-1997 |
fsmp |
Nuke the printing of the unredirect message unless bootverbose.
|
#
26155 |
|
26-May-1997 |
fsmp |
Added a test called 'LATE_START'.
This is now the default, it delays most of the MP startup to the function machdep.c:cpu_startup(). It should be possible to move the 2 functions found there (mp_start() & mp_announce()) even further down the path once we know exactly where that should be...
Help from: Peter Wemm <peter@spinner.dialix.com.au>
|
#
26108 |
|
25-May-1997 |
fsmp |
Broke up parse_mp_table() into 2 passes: - The 1st (preparse_mp_table()) counts the number of cpus, busses, etc. and records the LOCAL and IO APIC addresses. - The 2nd pass (parse_mp_table()) does the actual parsing of info and recording into the incore MP table.
This will allow us to defer the 2nd pass untill malloc() & private pages are available (but thats for another day!).
|
#
26101 |
|
24-May-1997 |
fsmp |
Now that panic() is properly printing messages for early SMP panics all the 'printf("..."); panic("\n")' sections are returned to 'panic("...")'.
|
#
26019 |
|
22-May-1997 |
fsmp |
Convert all: panic( "xxxxx\n" );
to: printf( "xxxxx\n" ); panic( "\n" );
For some as yet undetermined reason the argument to panic() is often NOT printed, and the system sometimes hangs before reaching the panic printout. So we hopefully at least print some useful info before the hang, as oppossed to leaving the user clueless as to what has happened.
|
#
25559 |
|
07-May-1997 |
fsmp |
fix bug in get_isa_apic_mask() where EISA bus was ignored.
Submitted by: Peter Wemm <peter@spinner.DIALix.COM>
|
#
25499 |
|
05-May-1997 |
fsmp |
Code to handle SMP/APIC_IO mapping of ISA INTs to APIC pins above IRQ15.
- doesn't break my system. - NOT yet verified on the affected motherboard.
Submitted by: "John S. Dyson" <toor@dyson.iquest.net>
|
#
25419 |
|
03-May-1997 |
fsmp |
new function to turn an APIC pin# into an INT mask. added missing APIC_IO define.
Submitted by: "John S. Dyson" <toor@dyson.iquest.net>
|
#
25361 |
|
01-May-1997 |
fsmp |
fixed spelling error.
Submitted by: Bruce Albrecht <bruce@zuhause.mn.org>
|
#
25292 |
|
29-Apr-1997 |
fsmp |
Enabled 'FIX_MP_TABLE_WORKS' code. This code re-numbers PCI busses in the MP table to match PCI semantics when the MP BIOS fails to do it properly.
Reviewed by: Peter Wemm <peter@spinner.DIALix.COM>
|
#
25216 |
|
27-Apr-1997 |
fsmp |
removed all the TEST_UPPERPRIO crud.
|
#
25215 |
|
27-Apr-1997 |
fsmp |
remove all the SMP_INVLTLB defines, making the code default for APIC_IO.
Reviewed by: informal discussion with Peter Wemm <peter@spinner.DIALix.COM>
|
#
25204 |
|
27-Apr-1997 |
fsmp |
informal discussion between Bruce Evans <bde@zeta.org.au>, Peter Wemm <peter@spinner.DIALix.COM>, Steve Passe <smp@csn.net>
removed all the IPI_INTS code. made the XFAST_IPI32 code default, renaming Xfastipi32 to Xinvltlb.
|
#
25164 |
|
26-Apr-1997 |
peter |
Man the liferafts! Here comes the long awaited SMP -> -current merge!
There are various options documented in i386/conf/LINT, there is more to come over the next few days.
The kernel should run pretty much "as before" without the options to activate SMP mode.
There are a handful of known "loose ends" that need to be fixed, but have been put off since the SMP kernel is in a moderately good condition at the moment.
This commit is the result of the tinkering and testing over the last 14 months by many people. A special thanks to Steve Passe for implementing the APIC code!
|