kernel/scheduler: Remove "inline" attribute from PeekThread.

This will allow it to be invoked outside scheduler_cpu.cpp,
and GCC should automatically inline this function within that file
anyway.

No functional change intended.

kernel/scheduler: enable cpu load tracking after boot

when the cpufreq module is loaded, we let the scheduler update its policy.
Improve assert report
CoreEntry::GetLoad() could return more than kMaxLoad.

Change-Id: I127f9b3e8062b5996872aae30b4021b9904fa179
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3216
Reviewed-by: Jérôme Duval <jerome.duval@gmail.com>

scheduler: Fix load update on idle cores

To make sure that load statistics are accurate on idle cores each time
idle thread is scheduled a timer is set to update load when current
load measurement interval elapses. However, core load is defined as the
average load during last measurement interval and idle core may be still
considered busy if it was not idle during entire measurement interval.
Since, load update timer is a one shot timer that information will not be
updated until the core becomes active again.

To mitigate that issue load update timer is set to fire after two load
measurement intervals had elapsed.

scheduler: Always update core heaps after thread migration

The main purpose of this patch is to eliminate the delay between thread
migration and result of that migration being visible in load statistics.
Such delay, in certain circumstances, may cause some cores to become
overloaded because the scheduler migrates too many threads to them before
the effect of migration becomes apparent.

scheduler: Update load of idle cores

In order to keep the scheduler tickless core load is computed and updated
only during various scheduler events (i.e. thread enqueue, reschedule, etc).
The problem it creates is that if a core becomes idle its load may remain
outdated for an extended period of time thus resulting in suboptimal thread
migration decisions.

The solution to this problem is to add a timer each time an idle thread is
scheudled which, after kLoadMeasureInterval, would fire and force load
update.

scheduler: Relax penalty cancellation requirements

Priority penalties were made more strict in order to prevent situation
when two or more high priority threads uses up all available CPU time
in such manner that they do not receive a penalty but starve low priority
threads.

However, a significant change to thread priorites has been made since and
now priority of all non real time threads varies in a range from 1 to
static priority minus penalty. This means that the scheduler is able to
prevent thread starvation without any complex penalty policies.

scheduler: Provide more stable core load statistics

Originially, core load was a sum of eastimated loads of all currently
running or ready threads on a given core. Such value is changing very
rapidly preventing the thread migration logic from making any reasonable
decisions.

This patch changes the way core load is computed to make it more stable
thus improving the qualitiy of decisions made by the thread migration logic.
Currently core load is a sum of estimated loads of all threads that have been
ready during last load measurement interval and haven't been migrated or
killed.

scheduler: Do not update load of disabled cores

scheduler: Estimate the load thread is able to produce

Previous implementation based on the actual load of each core and share
each thread has in that load turned up to be very problematic when
balancing load on very heavily loaded systems (i.e. more threads
consuming all available CPU time than there is logical CPUs).

The new approach is to estimate how much load would a thread produce
if it had all CPU time only for itself. Summing such load estimations
of each thread assigned to a given core we get a rank that contains
much more information than just simple actual core load.

scheduler: Keep track of the number of the ready threads

scheduler: Improve recognition of CPU bound threads

scheduler: Remove unnecessary check against disabled CPU

scheduler: Improve thread creation performance

scheduler: Work around GCC2 limitations in function inlining

GCC2 won't inline a function if it is used before its definition.

scheduler: Disable load tracking when not needed

scheduler: Protect per CPU run queue with its own lock

scheduler: Fix divide error in _RequestPerformanceLevel

scheduler: Use single ended heap for CPU heap

scheduler: Remove CPUEntry::IncreaseActiveTime()

scheduler: Add scheduler profiler

A bit hackish implementation of a profiler for the scheduler.
SCHEDULER_ENTER_FUNCTION at the begining of each function aren't nice and
usage of __PRETTY_FUNCTION__ isn't any better (both gcc and clang support
it though), but it was quick to implement and doesn't lose information
on inlined functions. It's just a tool, not an integral part of the kernal
anyway.

scheduler: Encapsulate ThreadData fields

scheduler: Encapsulate CPUEntry fields

scheduler: Encapsulate CoreEntry fields

scheduler: Encapsulate PackageEntry fields

Apart from the refactoring this commit takes the opportunity and removes
unnecessary read locks when choosing a package and a core from idle lists.
The data structures are accessed in a thread safe way and it does not really
matter whether the obtained data becomes outdated just when we release the
lock or during our search for the appropriate package/core.

scheduler: Allow calling UpdatePriority() for disabled CPU

scheduler: Try to keep thread on the same logical CPU

Some SMT implementations (e.g. recent AMD microarchitectures) have
separate L1d cache for each SMT thread (which AMD decides to call "cores").
This means that we shouldn't move threads to another logical processor too
often even if it belongs to the same core. We aren't very strict about
this as it would complicate load balancing, but we try to reduce unnecessary
migrations.

scheduler: Use sequential locks instead of atomic 64 bit access

kernel: Protect cpu_ent::active_time with sequential lock

atomic_{get, set}64() are problematic on architectures without 64 bit
compare and swap.
Also, using sequential lock instead of atomic access ensures that
any reads from cpu_ent::active_time won't require any writes to shared
memory.

kernel: Propagate scheduler modes to cpu{freq, idle} modules

scheduler: Code refactoring

scheduler: Fix load update on idle cores

To make sure that load statistics are accurate on idle cores each time
idle thread is scheduled a timer is set to update load when current
load measurement interval elapses. However, core load is defined as the
average load during last measurement interval and idle core may be still
considered busy if it was not idle during entire measurement interval.
Since, load update timer is a one shot timer that information will not be
updated until the core becomes active again.

To mitigate that issue load update timer is set to fire after two load
measurement intervals had elapsed.

scheduler: Always update core heaps after thread migration

The main purpose of this patch is to eliminate the delay between thread
migration and result of that migration being visible in load statistics.
Such delay, in certain circumstances, may cause some cores to become
overloaded because the scheduler migrates too many threads to them before
the effect of migration becomes apparent.

scheduler: Update load of idle cores

In order to keep the scheduler tickless core load is computed and updated
only during various scheduler events (i.e. thread enqueue, reschedule, etc).
The problem it creates is that if a core becomes idle its load may remain
outdated for an extended period of time thus resulting in suboptimal thread
migration decisions.

The solution to this problem is to add a timer each time an idle thread is
scheudled which, after kLoadMeasureInterval, would fire and force load
update.

scheduler: Relax penalty cancellation requirements

Priority penalties were made more strict in order to prevent situation
when two or more high priority threads uses up all available CPU time
in such manner that they do not receive a penalty but starve low priority
threads.

However, a significant change to thread priorites has been made since and
now priority of all non real time threads varies in a range from 1 to
static priority minus penalty. This means that the scheduler is able to
prevent thread starvation without any complex penalty policies.

scheduler: Provide more stable core load statistics

Originially, core load was a sum of eastimated loads of all currently
running or ready threads on a given core. Such value is changing very
rapidly preventing the thread migration logic from making any reasonable
decisions.

This patch changes the way core load is computed to make it more stable
thus improving the qualitiy of decisions made by the thread migration logic.
Currently core load is a sum of estimated loads of all threads that have been
ready during last load measurement interval and haven't been migrated or
killed.

scheduler: Do not update load of disabled cores

scheduler: Estimate the load thread is able to produce

Previous implementation based on the actual load of each core and share
each thread has in that load turned up to be very problematic when
balancing load on very heavily loaded systems (i.e. more threads
consuming all available CPU time than there is logical CPUs).

The new approach is to estimate how much load would a thread produce
if it had all CPU time only for itself. Summing such load estimations
of each thread assigned to a given core we get a rank that contains
much more information than just simple actual core load.

scheduler: Keep track of the number of the ready threads

scheduler: Improve recognition of CPU bound threads

scheduler: Remove unnecessary check against disabled CPU

scheduler: Improve thread creation performance

scheduler: Work around GCC2 limitations in function inlining

GCC2 won't inline a function if it is used before its definition.

scheduler: Disable load tracking when not needed

scheduler: Protect per CPU run queue with its own lock

scheduler: Fix divide error in _RequestPerformanceLevel

scheduler: Use single ended heap for CPU heap

scheduler: Remove CPUEntry::IncreaseActiveTime()

scheduler: Add scheduler profiler

A bit hackish implementation of a profiler for the scheduler.
SCHEDULER_ENTER_FUNCTION at the begining of each function aren't nice and
usage of __PRETTY_FUNCTION__ isn't any better (both gcc and clang support
it though), but it was quick to implement and doesn't lose information
on inlined functions. It's just a tool, not an integral part of the kernal
anyway.

scheduler: Encapsulate ThreadData fields

scheduler: Encapsulate CPUEntry fields

scheduler: Encapsulate CoreEntry fields

scheduler: Encapsulate PackageEntry fields

Apart from the refactoring this commit takes the opportunity and removes
unnecessary read locks when choosing a package and a core from idle lists.
The data structures are accessed in a thread safe way and it does not really
matter whether the obtained data becomes outdated just when we release the
lock or during our search for the appropriate package/core.

scheduler: Allow calling UpdatePriority() for disabled CPU

scheduler: Try to keep thread on the same logical CPU

Some SMT implementations (e.g. recent AMD microarchitectures) have
separate L1d cache for each SMT thread (which AMD decides to call "cores").
This means that we shouldn't move threads to another logical processor too
often even if it belongs to the same core. We aren't very strict about
this as it would complicate load balancing, but we try to reduce unnecessary
migrations.

scheduler: Use sequential locks instead of atomic 64 bit access

kernel: Protect cpu_ent::active_time with sequential lock

atomic_{get, set}64() are problematic on architectures without 64 bit
compare and swap.
Also, using sequential lock instead of atomic access ensures that
any reads from cpu_ent::active_time won't require any writes to shared
memory.

kernel: Propagate scheduler modes to cpu{freq, idle} modules

scheduler: Code refactoring