common/disp/cmt.c

4  * The contents of this file are subject to the terms of the
60  * relevant CMT sharing relationships existing between cpus to implement
63  * Load balancing policy seeks to improve performance by minimizing
65  * policies seek to improve cache and TLB utilization. Coalescence
80  * is willing to balance against all the top level PGs in the system.
82  * Groups of top level PGs are maintained to implement the above, one for each
89 cmt_lgrp_t		*cmt_root = NULL;	/* Reference to root cmt pg */
101  * Set this to non-zero to disable CMT scheduling
121  * One must be holding cpu_lock to change this.
127  * therefore may be subject to BIOS bugs.
132  * Macro to test if PG is managed by the CMT PG class
191  * Called to indicate a new CPU has started up so
203  * Return non-zero if thread can migrate between "from" and "to"
207 pg_cmt_can_migrate(cpu_t *from, cpu_t *to)
210 	    to->cpu_physid->cpu_cacheid)
238  * policies should be implemented to optimize performance and efficiency
246 	 * Give the platform a chance to override the default
303 	 * Stick with the default callbacks if there isn't going to be
328  * NULL, is a reference to the CPU's under-construction PG data.
347 		 * Nothing to do
356 	 * by the dispatcher. Pause CPUS to ensure exclusivity.
491  * being initialized). The argument "pgdata" is a reference to the CPU's PG
492  * data to be constructed.
494  * cp->cpu_pg is used by the dispatcher to access the CPU's PG data
496  * calls must be careful to operate only on the "pgdata" argument, and not
520 	 * Interrogate the platform to see if the CPU
535 		 * for which we know how to optimize.
556 		 * exists, or if one needs to be created.
586 		/* Add the CPU to the PG */
632 	 * This routine will also try to fix the lineage along with the
682 	 *	  it to it's parent and siblings.
744 		 * cpu_cacheid to that PG's logical id
761  * "pgdata" is a reference to the CPU's PG data to be deconstructed.
763  * cp->cpu_pg is used by the dispatcher to access the CPU's PG data
765  * pg_cmt_cpu_fini() and the routines it calls must be careful to operate only
801 		 * call to pg_cmt_cpu_fini() is part of that deconfiguration.
806 		 * cached for the boot CPU, since this is what needs to be
829 		 * If it's about to become empty, destroy it's children
831 		 * This is done here (rather than below) to avoid removing
899 	 * is large enough for all CMT PG's to which cp
942 		 * Add the PG to the bitset in the new partition.
996 		 * Move to the next generation since topology is changing
1066 		 * Move to the next generation since topology is changing
1251  * Interfaces to enable and disable power aware dispatching
1269 		 * Unable to find any instances of the specified type
1283 	 * PG and it's parent have the same CPUs to ensure it's policy
1289 		 * If the power domain is an only child to a parent
1291 		 * above the parent to activate the policy.
1318 		 * Unable to find any instances of the specified type of
1333 		 * above the power domain to ensure it's policy dominates.
1362  * Macro to test whether a thread is currently runnable on a CPU in a PG.
1456  * NULL, is a reference to the CPU's under-construction PG data.
1502 	 * to ensure the hierarchy's groups are properly sized.
1512 	 * We may only need to blacklist the illegal PGs, and other instances
1549 	 * We're operating on the PG hierarchy. Pause CPUs to ensure
1550 	 * exclusivity with respect to the dispatcher.
1580 		 * Move PG's children from it's children set to it's parent's
1586 		 * children to the parent's children set, and once we are done
1613 		 * Reset the callbacks to the defaults
1688  * This routine is invoked by pg_cmt_cpu_init() to validate the integrity
1692  * possible to integrate those groupings into the CMT PG hierarchy, if doing
1696  * pg_cmt_lineage_validate()'s purpose is to detect grouping definitions that
1698  * and the PG is of a grouping type who's definition is known to originate from
1699  * suspect sources (BIOS), then pg_cmt_prune() will be invoked to prune the
1703  * sources, then pg_cmt_disable() will be invoked to log an error, and disable
1706  * This routine is invoked after the CPU has been added to the PGs in which
1707  * it belongs, but before those PGs have been added to (or had their place
1715  * The addition of this new CPU to these pre-existing PGs means that those
1716  * PGs may need to be promoted up in the hierarchy to satisfy the subset
1717  * invariant. In additon to testing the subset invariant for the lineage,
1718  * this routine also verifies that the addition of the new CPU to the
1719  * existing PGs wouldn't cause the subset invariant to be violated in
1727  * was unable to recover from (and set cmt_lineage_status along the way).
1731  * "pgdata" is a reference to the CPU's under-construction PG data.
1732  * This routine must be careful to operate only on "pgdata", and not cp->cpu_pg.
1765 		 * The CPUs PG lineage was passed as the first argument to
1769 		 * the same. PG promotion will be used as the mechanism to
1770 		 * achieve this, but first we need to look for cases where
1776 		 * need to check whether the PG's existing parent (if any) is
1781 		 * will need to be promoted into place.
1783 		 * We need to verify all of this to defend against a buggy
1804 			 * need to be promoted above it. Verify the ancestor
1858 	 * We'll try to recover in such cases by pruning out the illegal
1870 		 * This isn't supported, as the dispatcher isn't allowed to
1871 		 * to do CMT thread placement across lgroups, as this would
1874 		 * If the PG is of a sharing relationship type known to
1879 		 * Otherwise, fall though to the case below for handling.
1901 		 * Something went wrong trying to prune out the bad level.
1913 		 * In this case, we examine the two PGs to see if either
1922 		 * or we would have to prune out level derived from what we
1944 		 * Something went wrong trying to identify and/or prune out
1952 		 * which we don't know how to recover. In this case, disable