kernel/scheduler: Remove "inline" attribute from PeekThread.

This will allow it to be invoked outside scheduler_cpu.cpp,
and GCC should automatically inline this function within that file
anyway.

No functional change intended.

kernel/scheduler: enable cpu load tracking after boot

when the cpufreq module is loaded, we let the scheduler update its policy.
Improve assert report
CoreEntry::GetLoad() could return more than kMaxLoad.

Change-Id: I127f9b3e8062b5996872aae30b4021b9904fa179
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3216
Reviewed-by: Jérôme Duval <jerome.duval@gmail.com>

More class/struct mixup fixes.

Spotted by Clang.

scheduler: Always update core heaps after thread migration

The main purpose of this patch is to eliminate the delay between thread
migration and result of that migration being visible in load statistics.
Such delay, in certain circumstances, may cause some cores to become
overloaded because the scheduler migrates too many threads to them before
the effect of migration becomes apparent.

scheduler: Update load of idle cores

In order to keep the scheduler tickless core load is computed and updated
only during various scheduler events (i.e. thread enqueue, reschedule, etc).
The problem it creates is that if a core becomes idle its load may remain
outdated for an extended period of time thus resulting in suboptimal thread
migration decisions.

The solution to this problem is to add a timer each time an idle thread is
scheudled which, after kLoadMeasureInterval, would fire and force load
update.

scheduler: Relax penalty cancellation requirements

Priority penalties were made more strict in order to prevent situation
when two or more high priority threads uses up all available CPU time
in such manner that they do not receive a penalty but starve low priority
threads.

However, a significant change to thread priorites has been made since and
now priority of all non real time threads varies in a range from 1 to
static priority minus penalty. This means that the scheduler is able to
prevent thread starvation without any complex penalty policies.

scheduler: Provide more stable core load statistics

Originially, core load was a sum of eastimated loads of all currently
running or ready threads on a given core. Such value is changing very
rapidly preventing the thread migration logic from making any reasonable
decisions.

This patch changes the way core load is computed to make it more stable
thus improving the qualitiy of decisions made by the thread migration logic.
Currently core load is a sum of estimated loads of all threads that have been
ready during last load measurement interval and haven't been migrated or
killed.

scheduler: Estimate the load thread is able to produce

Previous implementation based on the actual load of each core and share
each thread has in that load turned up to be very problematic when
balancing load on very heavily loaded systems (i.e. more threads
consuming all available CPU time than there is logical CPUs).

The new approach is to estimate how much load would a thread produce
if it had all CPU time only for itself. Summing such load estimations
of each thread assigned to a given core we get a rank that contains
much more information than just simple actual core load.

scheduler: Keep track of the number of the ready threads

scheduler: Improve recognition of CPU bound threads

scheduler: Improve thread creation performance

scheduler: Work around GCC2 limitations in function inlining

GCC2 won't inline a function if it is used before its definition.

scheduler: Fix order of the includes

Thanks Jérôme!

scheduler: Protect per CPU run queue with its own lock

scheduler: Strengthen CoreEntry::GetLoad() assertion

scheduler: Use single ended heap for CPU heap

scheduler: Remove CPUEntry::IncreaseActiveTime()

scheduler: Add scheduler profiler

A bit hackish implementation of a profiler for the scheduler.
SCHEDULER_ENTER_FUNCTION at the begining of each function aren't nice and
usage of __PRETTY_FUNCTION__ isn't any better (both gcc and clang support
it though), but it was quick to implement and doesn't lose information
on inlined functions. It's just a tool, not an integral part of the kernal
anyway.

scheduler: Encapsulate ThreadData fields

scheduler: Encapsulate CPUEntry fields

scheduler: Encapsulate CoreEntry fields

scheduler: Encapsulate PackageEntry fields

Apart from the refactoring this commit takes the opportunity and removes
unnecessary read locks when choosing a package and a core from idle lists.
The data structures are accessed in a thread safe way and it does not really
matter whether the obtained data becomes outdated just when we release the
lock or during our search for the appropriate package/core.

scheduler: Try to keep thread on the same logical CPU

Some SMT implementations (e.g. recent AMD microarchitectures) have
separate L1d cache for each SMT thread (which AMD decides to call "cores").
This means that we shouldn't move threads to another logical processor too
often even if it belongs to the same core. We aren't very strict about
this as it would complicate load balancing, but we try to reduce unnecessary
migrations.

scheduler: Use sequential locks instead of atomic 64 bit access

scheduler: Code refactoring

scheduler: Always update core heaps after thread migration

The main purpose of this patch is to eliminate the delay between thread
migration and result of that migration being visible in load statistics.
Such delay, in certain circumstances, may cause some cores to become
overloaded because the scheduler migrates too many threads to them before
the effect of migration becomes apparent.

scheduler: Update load of idle cores

In order to keep the scheduler tickless core load is computed and updated
only during various scheduler events (i.e. thread enqueue, reschedule, etc).
The problem it creates is that if a core becomes idle its load may remain
outdated for an extended period of time thus resulting in suboptimal thread
migration decisions.

The solution to this problem is to add a timer each time an idle thread is
scheudled which, after kLoadMeasureInterval, would fire and force load
update.

scheduler: Relax penalty cancellation requirements

Priority penalties were made more strict in order to prevent situation
when two or more high priority threads uses up all available CPU time
in such manner that they do not receive a penalty but starve low priority
threads.

However, a significant change to thread priorites has been made since and
now priority of all non real time threads varies in a range from 1 to
static priority minus penalty. This means that the scheduler is able to
prevent thread starvation without any complex penalty policies.

scheduler: Provide more stable core load statistics

Originially, core load was a sum of eastimated loads of all currently
running or ready threads on a given core. Such value is changing very
rapidly preventing the thread migration logic from making any reasonable
decisions.

This patch changes the way core load is computed to make it more stable
thus improving the qualitiy of decisions made by the thread migration logic.
Currently core load is a sum of estimated loads of all threads that have been
ready during last load measurement interval and haven't been migrated or
killed.

scheduler: Estimate the load thread is able to produce

Previous implementation based on the actual load of each core and share
each thread has in that load turned up to be very problematic when
balancing load on very heavily loaded systems (i.e. more threads
consuming all available CPU time than there is logical CPUs).

The new approach is to estimate how much load would a thread produce
if it had all CPU time only for itself. Summing such load estimations
of each thread assigned to a given core we get a rank that contains
much more information than just simple actual core load.

scheduler: Keep track of the number of the ready threads

scheduler: Improve recognition of CPU bound threads

scheduler: Improve thread creation performance

scheduler: Work around GCC2 limitations in function inlining

GCC2 won't inline a function if it is used before its definition.

scheduler: Fix order of the includes

Thanks Jérôme!

scheduler: Protect per CPU run queue with its own lock

scheduler: Strengthen CoreEntry::GetLoad() assertion

scheduler: Use single ended heap for CPU heap

scheduler: Remove CPUEntry::IncreaseActiveTime()

scheduler: Add scheduler profiler

A bit hackish implementation of a profiler for the scheduler.
SCHEDULER_ENTER_FUNCTION at the begining of each function aren't nice and
usage of __PRETTY_FUNCTION__ isn't any better (both gcc and clang support
it though), but it was quick to implement and doesn't lose information
on inlined functions. It's just a tool, not an integral part of the kernal
anyway.

scheduler: Encapsulate ThreadData fields

scheduler: Encapsulate CPUEntry fields

scheduler: Encapsulate CoreEntry fields

scheduler: Encapsulate PackageEntry fields

Apart from the refactoring this commit takes the opportunity and removes
unnecessary read locks when choosing a package and a core from idle lists.
The data structures are accessed in a thread safe way and it does not really
matter whether the obtained data becomes outdated just when we release the
lock or during our search for the appropriate package/core.

scheduler: Try to keep thread on the same logical CPU

Some SMT implementations (e.g. recent AMD microarchitectures) have
separate L1d cache for each SMT thread (which AMD decides to call "cores").
This means that we shouldn't move threads to another logical processor too
often even if it belongs to the same core. We aren't very strict about
this as it would complicate load balancing, but we try to reduce unnecessary
migrations.

scheduler: Use sequential locks instead of atomic 64 bit access

scheduler: Code refactoring