tools/virtio: Fix arm64 ringtest compilation error

Add cpu_relax() for arm64 instead of directly assert(), and add assert.h
header file. Also, add smp_wmb and smp_mb for arm64.

Compilation error as follows, avoid __always_inline undefined.

$ make
cc -Wall -pthread -O2 -ggdb -flto -fwhole-program -c -o ring.o ring.c
In file included from ring.c:10:
main.h: In function ‘busy_wait’:
main.h:99:21: warning: implicit declaration of function ‘assert’
[-Wimplicit-function-declaration]
99 | #define cpu_relax() assert(0)
| ^~~~~~
main.h:107:17: note: in expansion of macro ‘cpu_relax’
107 | cpu_relax();
| ^~~~~~~~~
main.h:12:1: note: ‘assert’ is defined in header ‘<assert.h>’; did you
forget to ‘#include <assert.h>’?
11 | #include <stdbool.h>
+++ |+#include <assert.h>
12 |
main.h: At top level:
main.h:143:23: error: expected ‘;’ before ‘void’
143 | static __always_inline
| ^
| ;
144 | void __read_once_size(const volatile void *p, void *res, int
size)
| ~~~~
main.h:158:23: error: expected ‘;’ before ‘void’
158 | static __always_inline void __write_once_size(volatile void *p,
void *res, int size)
| ^~~~~
| ;
make: *** [<builtin>: ring.o] Error 1

Signed-off-by: Rong Tao <rongtao@cestc.cn>
Message-Id: <tencent_F53E159DD7925174445D830DA19FACF44B07@qq.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

tools/virtio: remove smp_read_barrier_depends()

This gets rid of the last references to smp_read_barrier_depends()
which for the kernel side was removed in v5.9. The serialization
required for Alpha is done inside READ_ONCE() instead of having
users deal with it. Simply use a full barrier, the architecture
does not have rmb in the first place.

Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
Message-Id: <20221128034347.990-3-dave@stgolabs.net>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>

tools/virtio: remove stray characters

__read_once_size() is not a macro, remove those '/'s.

Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
Message-Id: <20221128034347.990-2-dave@stgolabs.net>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>

treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 482

Based on 1 normalized pattern(s):

this work is licensed under the terms of the gnu gpl version 2

extracted by the scancode license scanner the SPDX license identifier

GPL-2.0-only

has been chosen to replace the boilerplate/reference in 48 file(s).

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Allison Randal <allison@lohutok.net>
Reviewed-by: Enrico Weigelt <info@metux.net>
Cc: linux-spdx@vger.kernel.org
Link: https://lkml.kernel.org/r/20190604081204.624030236@linutronix.de
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

tools/virtio: fix smp_mb on x86

Offset 128 overlaps the last word of the redzone.
Use 132 which is always beyond that.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

tools/virtio: copy READ/WRITE_ONCE

This is to make ptr_ring test build again.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

locking/x86: Use LOCK ADD for smp_mb() instead of MFENCE

MFENCE appears to be way slower than a locked instruction - let's use
LOCK ADD unconditionally, as we always did on old 32-bit.

Performance testing results:

perf stat -r 10 -- ./virtio_ring_0_9 --sleep --host-affinity 0 --guest-affinity 0
Before:
0.922565990 seconds time elapsed ( +- 1.15% )
After:
0.578667024 seconds time elapsed ( +- 1.21% )

i.e. about ~60% faster.

Just poking at SP would be the most natural, but if we then read the
value from SP, we get a false dependency which will slow us down.

This was noted in this article:

http://shipilev.net/blog/2014/on-the-fence-with-dependencies/

And is easy to reproduce by sticking a barrier in a small non-inline
function.

So let's use a negative offset - which avoids this problem since we
build with the red zone disabled.

For userspace, use an address just below the redzone.

The one difference between LOCK ADD and MFENCE is that LOCK ADD does
not affect CLFLUSH, previous patches converted all uses of CLFLUSH to
call mb(), such that changes to smp_mb() won't affect it.

Update mb/rmb/wmb() on 32-bit to use the negative offset, too, for
consistency.

As a follow-up, it might be worth considering switching users
of CLFLUSH to another API (e.g. clflush_mb()?) - we will
then be able to convert mb() to smp_mb() again.

Also arguably, GCC should switch to use LOCK ADD for __sync_synchronize().
This might be worth pursuing separately.

Suggested-by: Andy Lutomirski <luto@amacapital.net>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Acked-by: Peter Zijlstra <peterz@infradead.org>
Acked-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: qemu-devel@nongnu.org
Cc: virtualization@lists.linux-foundation.org
Link: http://lkml.kernel.org/r/1509118355-4890-1-git-send-email-mst@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>

ringtest: support test specific parameters

Add a new flag for passing test-specific parameters.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

tools/virtio/ringtest: tweaks for s390

Make ringtest work on s390 too.

Signed-off-by: Halil Pasic <pasic@linux.vnet.ibm.com>
Acked-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
Signed-off-by: Cornelia Huck <cornelia.huck@de.ibm.com>

ringtest: commonize implementation of poll_avail/poll_used

Provide new primitives used_empty/avail_empty and
build poll_avail/poll_used on top of it.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

tools/virtio: add ringtest utilities

This adds micro-benchmarks useful for tuning virtio ring layouts.
Three layouts are currently implemented:

- virtio 0.9 compatible one
- an experimental extension bypassing the ring index, polling ring
itself instead
- an experimental extension bypassing avail and used ring completely

Typical use:

sh run-on-all.sh perf stat -r 10 --log-fd 1 -- ./ring

It doesn't depend on the kernel directly, but it's handy
to have as much virtio stuff as possible in one tree.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>