tools: ynl: fix setting presence bits in simple nests

When we set members of simple nested structures in requests
we need to set "presence" bits for all the nesting layers
below. This has nothing to do with the presence type of
the last layer.

Fixes: be5bea1cc0bf ("net: add basic C code generators for Netlink")
Reviewed-by: Breno Leitao <leitao@debian.org>
Link: https://lore.kernel.org/r/20240321020214.1250202-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

netlink: specs: support generating code for genl socket priv

The family struct is auto-generated for new families, support
use of the sock_priv_* mechanism added in commit a731132424ad
("genetlink: introduce per-sock family private storage").

For example if the family wants to use struct sk_buff as its
private struct (unrealistic but just for illustration), it would
add to its spec:

kernel-family:
headers: [ "linux/skbuff.h" ]
sock-priv: struct sk_buff

ynl-gen-c will declare the appropriate priv size and hook
in function prototypes to be implemented by the family.

Link: https://lore.kernel.org/r/20240308190319.2523704-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: support using pre-defined values in attr checks

Support using pre-defined values in checks so we don't need to use hard
code number for the string, binary length. e.g. we have a definition like

#define TEAM_STRING_MAX_LEN 32

Which defined in yaml like:

definitions:
-
name: string-max-len
type: const
value: 32

It can be used in the attribute-sets like

attribute-sets:
-
name: attr-option
name-prefix: team-attr-option-
attributes:
-
name: name
type: string
checks:
len: string-max-len

With this patch it will be converted to

[TEAM_ATTR_OPTION_NAME] = { .type = NLA_STRING, .len = TEAM_STRING_MAX_LEN, }

Signed-off-by: Hangbin Liu <liuhangbin@gmail.com>
Link: https://lore.kernel.org/r/20240311140727.109562-1-liuhangbin@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools/net/ynl: Fix c codegen for array-nest

ynl-gen-c generates e.g. 'calloc(mcast_groups, sizeof(*dst->mcast_groups))'
for array-nest attrs when it should be 'n_mcast_groups'.

Add a 'n_' prefix in the generated code for array-nests.

Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://lore.kernel.org/r/20240306231046.97158-4-donald.hunter@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: remove the libmnl dependency

We don't use libmnl any more.

Acked-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
Link: https://lore.kernel.org/r/20240227223032.1835527-15-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: switch away from MNL_CB_*

Create a local version of the MNL_CB_* parser control values.

Acked-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
Link: https://lore.kernel.org/r/20240227223032.1835527-13-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: switch away from mnl_cb_t

All YNL parsing callbacks take struct ynl_parse_arg as the argument.
Make that official by using a local callback type instead of mnl_cb_t.

Acked-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
Link: https://lore.kernel.org/r/20240227223032.1835527-12-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: remove unused parse code

Commit f2ba1e5e2208 ("tools: ynl-gen: stop generating common notification handlers")
removed the last caller of the parse_cb_run() helper.
We no longer need to export ynl_cb_array.

Acked-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
Link: https://lore.kernel.org/r/20240227223032.1835527-8-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: make yarg the first member of struct ynl_dump_state

All YNL parsing code expects a pointer to struct ynl_parse_arg AKA yarg.
For dump was pass in struct ynl_dump_state, which works fine, because
struct ynl_dump_state and struct ynl_parse_arg have identical layout
for the members that matter.. but it's a bit hacky.

Acked-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
Link: https://lore.kernel.org/r/20240227223032.1835527-7-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: create local ARRAY_SIZE() helper

libc doesn't have an ARRAY_SIZE() create one locally.

Acked-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
Link: https://lore.kernel.org/r/20240227223032.1835527-6-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: create local nlmsg access helpers

Create helpers for accessing payloads of struct nlmsg.
Use them instead of the libmnl ones.

Acked-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
Link: https://lore.kernel.org/r/20240227223032.1835527-5-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: create local for_each helpers

Create ynl_attr_for_each*() iteration helpers.
Use them instead of the mnl ones.

Acked-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
Link: https://lore.kernel.org/r/20240227223032.1835527-4-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: create local attribute helpers

Don't use mnl attr helpers, we're trying to remove the libmnl
dependency. Create both signed and unsigned helpers, libmnl
had unsigned helpers, so code generator no longer needs
the mnl_type() hack.

The new helpers are written from first principles, but are
hopefully not too buggy.

Acked-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
Link: https://lore.kernel.org/r/20240227223032.1835527-3-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: print prototypes for recursive stuff

We avoid printing forward declarations and prototypes for most
types by sorting things topologically. But if structs nest we
do need the forward declarations, there's no other way.

Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://lore.kernel.org/r/20231213231432.2944749-9-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: store recursive nests by a pointer

To avoid infinite nesting store recursive structs by pointer.
If recursive struct is placed in the op directly - the first
instance can be stored by value. That makes the code much
less of a pain for majority of practical uses.

Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://lore.kernel.org/r/20231213231432.2944749-8-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: re-sort ignoring recursive nests

We try to keep the structures and helpers "topologically sorted",
to avoid forward declarations. When recursive nests are at play
we need to sort twice, because structs which end up being marked
as recursive will get a full set of forward declarations, so we
should ignore them for the purpose of sorting.

Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://lore.kernel.org/r/20231213231432.2944749-7-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: record information about recursive nests

Track which nests are recursive. Non-recursive nesting gets
rendered in C as directly nested structs. For recursive
ones we need to put a pointer in, rather than full struct.

Track this information, no change to generated code, yet.

Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://lore.kernel.org/r/20231213231432.2944749-6-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: fill in implementations for TypeUnused

Fill in more empty handlers for TypeUnused. When 'unused'
attr gets specified in a nested set we have to cleanly
skip it during code generation.

Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://lore.kernel.org/r/20231213231432.2944749-5-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: support fixed headers in genetlink

Support genetlink families using simple fixed headers.
Assume fixed header is identical for all ops of the family for now.

Fixed headers are added to the request and reply structs as a _hdr
member, and copied to/from netlink messages appropriately.

Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://lore.kernel.org/r/20231213231432.2944749-4-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: use enum user type for members and args

Commit 30c902001534 ("tools: ynl-gen: use enum name from the spec")
added pre-cooked user type for enums. Use it to fix ignoring
enum-name provided in the spec.

This changes a type in struct ethtool_tunnel_udp_entry but is
generally inconsequential for current families.

Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://lore.kernel.org/r/20231213231432.2944749-3-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: add missing request free helpers for dumps

The code gen generates a prototype for dump request free
in the header, but no implementation in the source.

Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://lore.kernel.org/r/20231213231432.2944749-2-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: don't skip regeneration from make targets

Commit 2b7ac0c87d98 ("tools: ynl-gen: don't touch the output file if
content is the same") is working too well. It was added so that
ynl-regen -f doesn't make us rebuild half of the kernel, if there
are no actual changes in any generated code.

When ynl-gen-c is called by make, however, we're better off trusting
make's tracking and overwrite the file. Otherwise if output is identical
we won't update file timestamps and make will retry code gen on every
invocation.

Link: https://lore.kernel.org/r/20231129193622.2912353-5-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: use enum name from the spec

The enum name used for id-to-str table does not handle
the enum-name override in the spec correctly.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>

tools: ynl-get: use family c-name

If a new family is ever added with a dash in the name
the C codegen will break. Make sure we use the "safe"
form of the name consistently.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>

tools: ynl-gen: always append ULL/LL to range types

32bit builds generate the following warning when we use a u32-max
in range validation:

warning: decimal constant 4294967295 is between LONG_MAX and ULONG_MAX. For C99 that means long long, C90 compilers are very likely to produce unsigned long (and a warning) here

The range values are u64, slap ULL/LL on all of them just
to avoid such noise.

There's currently no code using full range validation, but
it will matter in the upcoming page-pool introspection.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>

tools: ynl-gen: always construct struct ynl_req_state

struct ynl_req_state carries reply-related info from generated code
into generic YNL code. While we don't need reply info to execute
a request without a reply, we still need to pass in the struct, because
it's also where we get the pointer to struct ynl_sock from. Passing NULL
results in crashes if kernel returns an error or an unexpected reply.

Fixes: dc0956c98f11 ("tools: ynl-gen: move the response reading logic into YNL")
Link: https://lore.kernel.org/r/20231126225858.2144136-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: fix duplicate op name in devlink

We don't support CRUD-inspired message types in YNL too well.
One aspect that currently trips us up is the fact that single
message ID can be used in multiple commands (as the response).
This leads to duplicate entries in the id-to-string tables:

devlink-user.c:19:34: warning: initialized field overwritten [-Woverride-init]
19 | [DEVLINK_CMD_PORT_NEW] = "port-new",
| ^~~~~~~~~~
devlink-user.c:19:34: note: (near initialization for ‘devlink_op_strmap[7]’)

Fixes tag points at where the code was generated, the "real" problem
is that the code generator does not support CRUD.

Fixes: f2f9dd164db0 ("netlink: specs: devlink: add the remaining command to generate complete split_ops")
Link: https://lore.kernel.org/r/20231123030558.1611831-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: don't touch the output file if content is the same

I often regenerate all YNL files in the tree to make sure they
are in sync with the codegen and specs. Generator rewrites
the files unconditionally, so since make looks at file modification
time to decide what to rebuild - my next build takes longer.

We already generate the code to a tempfile most of the time,
only overwrite the target when we have to.

Before:

$ stat include/uapi/linux/netdev.h
File: include/uapi/linux/netdev.h
Size: 2307 Blocks: 8 IO Block: 4096 regular file
Access: 2023-10-27 15:19:56.347071940 -0700
Modify: 2023-10-27 15:19:45.089000900 -0700
Change: 2023-10-27 15:19:45.089000900 -0700
Birth: 2023-10-27 15:19:45.088000894 -0700

$ ./tools/net/ynl/ynl-regen.sh -f
[...]

$ stat include/uapi/linux/netdev.h
File: include/uapi/linux/netdev.h
Size: 2307 Blocks: 8 IO Block: 4096 regular file
Access: 2023-10-27 15:19:56.347071940 -0700
Modify: 2023-10-27 15:22:18.417968446 -0700
Change: 2023-10-27 15:22:18.417968446 -0700
Birth: 2023-10-27 15:19:45.088000894 -0700

After:

$ stat include/uapi/linux/netdev.h
File: include/uapi/linux/netdev.h
Size: 2307 Blocks: 8 IO Block: 4096 regular file
Access: 2023-10-27 15:22:41.520114221 -0700
Modify: 2023-10-27 15:22:18.417968446 -0700
Change: 2023-10-27 15:22:18.417968446 -0700
Birth: 2023-10-27 15:19:45.088000894 -0700

$ ./tools/net/ynl/ynl-regen.sh -f
[...]

$ stat include/uapi/linux/netdev.h
File: include/uapi/linux/netdev.h
Size: 2307 Blocks: 8 IO Block: 4096 regular file
Access: 2023-10-27 15:22:41.520114221 -0700
Modify: 2023-10-27 15:22:18.417968446 -0700
Change: 2023-10-27 15:22:18.417968446 -0700
Birth: 2023-10-27 15:19:45.088000894 -0700

Reviewed-by: Jiri Pirko <jiri@nvidia.com>
Link: https://lore.kernel.org/r/20231027223408.1865704-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: respect attr-cnt-name at the attr set level

Davide reports that we look for the attr-cnt-name in the wrong
object. We try to read it from the family, but the schema only
allows for it to exist at attr-set level.

Reported-by: Davide Caratti <dcaratti@redhat.com>
Link: https://lore.kernel.org/all/CAKa-r6vCj+gPEUKpv7AsXqM77N6pB0evuh7myHq=585RA3oD5g@mail.gmail.com/
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
Link: https://lore.kernel.org/r/20231025182739.184706-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

netlink: specs: support conditional operations

Page pool code is compiled conditionally, but the operations
are part of the shared netlink family. We can handle this
by reporting empty list of pools or -EOPNOTSUPP / -ENOSYS
but the cleanest way seems to be removing the ops completely
at compilation time. That way user can see that the page
pool ops are not present using genetlink introspection.
Same way they'd check if the kernel is "new enough" to
support the ops.

Extend the specs with the ability to specify the config
condition under which op (and its policies, etc.) should
be hidden.

Reviewed-by: Jiri Pirko <jiri@nvidia.com>
Link: https://lore.kernel.org/r/20231025162253.133159-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

netlink: make range pointers in policies const

struct nla_policy is usually constant itself, but unless
we make the ranges inside constant we won't be able to
make range structs const. The ranges are not modified
by the core.

Reviewed-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: David Ahern <dsahern@kernel.org>
Reviewed-by: Nikolay Aleksandrov <razor@blackwall.org>
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
Link: https://lore.kernel.org/r/20231025162204.132528-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: add support for exact-len validation

add support for 'exact-len' validation on netlink attributes.

Link: https://github.com/multipath-tcp/mptcp_net-next/issues/340
Acked-by: Matthieu Baerts <matttbe@kernel.org>
Signed-off-by: Davide Caratti <dcaratti@redhat.com>
Signed-off-by: Mat Martineau <martineau@kernel.org>
Link: https://lore.kernel.org/r/20231023-send-net-next-20231023-1-v2-2-16b1f701f900@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: render rsp_parse() helpers if cmd has only dump op

Due to the check in RenderInfo class constructor, type_consistent
flag is set to False to avoid rendering the same response parsing
helper for do and dump ops. However, in case there is no do, the helper
needs to be rendered for dump op. So split check to achieve that.

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
Link: https://lore.kernel.org/r/20231021112711.660606-4-jiri@resnulli.us
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: introduce support for bitfield32 attribute type

Introduce support for attribute type bitfield32.
Note that since the generated code works with struct nla_bitfield32,
the generator adds netlink.h to the list of includes for userspace
headers in case any bitfield32 is present.

Note that this is added only to genetlink-legacy scheme as requested
by Jakub Kicinski.

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
Link: https://lore.kernel.org/r/20231021112711.660606-3-jiri@resnulli.us
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: change spacing around __attribute__

checkpatch gets confused and treats __attribute__ as a function call.
It complains about white space before "(":

WARNING:SPACING: space prohibited between function name and open parenthesis '('
+ struct netdev_queue_get_rsp obj __attribute__ ((aligned (8)));

No spaces wins in the kernel:

$ git grep 'attribute__((.*aligned(' | wc -l
480
$ git grep 'attribute__ ((.*aligned (' | wc -l
110
$ git grep 'attribute__ ((.*aligned(' | wc -l
94
$ git grep 'attribute__((.*aligned (' | wc -l
63

So, whatever, change the codegen.

Note that checkpatch also thinks we should use __aligned(),
but this is user space code.

Link: https://lore.kernel.org/all/202310190900.9Dzgkbev-lkp@intel.com/
Acked-by: Stanislav Fomichev <sdf@google.com>
Reviewed-by: Amritha Nambiar <amritha.nambiar@intel.com>
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
Link: https://lore.kernel.org/r/20231020221827.3436697-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

netlink: specs: add support for auto-sized scalars

Support uint / sint types in specs and YNL.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Acked-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

tools: ynl-gen: make the mnl_type() method public

uint/sint support will add more logic to mnl_type(),
deduplicate it and make it more accessible.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>

tools: ynl-gen: support limit names

Support the use of symbolic names like s8-min or u32-max in checks
to make writing specs less painful.

Link: https://lore.kernel.org/r/20231018163917.2514503-4-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: support full range of min/max checks for integer values

Extend the support to full range of min/max checks.
None of the existing YNL families required complex integer validation.

The support is less than trivial, because we try to keep struct nla_policy
tiny the min/max members it holds in place are s16. Meaning we can only
express checks in range of s16. For larger ranges we need to define
a structure and link it in the policy.

Link: https://lore.kernel.org/r/20231018163917.2514503-3-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: track attribute use

For range validation we'll need to know if any individual
attribute is used on input (i.e. whether we will generate
a policy for it). Track this information.

Link: https://lore.kernel.org/r/20231018163917.2514503-2-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: handle do ops with no input attrs

The code supports dumps with no input attributes currently
thru a combination of special-casing and luck.
Clean up the handling of ops with no inputs. Create empty
Structs, and skip printing of empty types.
This makes dos with no inputs work.

Tested-by: Lorenzo Bianconi <lorenzo.bianconi@redhat.com>
Link: https://lore.kernel.org/r/20231006135032.3328523-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: use uapi header name for the header guard

Chuck points out that we should use the uapi-header property
when generating the guard. Otherwise we may generate the same
guard as another file in the tree.

Tested-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Reviewed-by: Przemek Kitszel <przemyslaw.kitszel@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

tools: ynl-gen: fix uAPI generation after tempfile changes

We use a tempfile for code generation, to avoid wiping the target
file out if the code generator crashes. File contents are copied
from tempfile to actual destination at the end of main().

uAPI generation is relatively simple so when generating the uAPI
header we return from main() early, and never reach the "copy code
over" stage. Since commit under Fixes uAPI headers are not updated
by ynl-gen.

Move the copy/commit of the code into CodeWriter, to make it
easier to call at any point in time. Hook it into the destructor
to make sure we don't miss calling it.

Fixes: f65f305ae008 ("tools: ynl-gen: use temporary file for rendering")
Link: https://lore.kernel.org/r/20230824212431.1683612-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: support empty attribute lists

Differentiate between empty list and None for member lists.
New families may want to create request responses with no attribute.
If we treat those the same as None we end up rendering
a full parsing policy in user space, instead of an empty one.

Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://lore.kernel.org/r/20230824003056.1436637-5-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: fix collecting global policy attrs

We look for attributes inside do.request, but there's another
layer of nesting in the spec, look inside do.request.attributes.

This bug had no effect as all global policies we generate (fou)
seem to be full, anyway, and we treat full and empty the same.

Next patch will change the treatment of empty policies.

Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://lore.kernel.org/r/20230824003056.1436637-4-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: set length of binary fields

Remember to set the length field in the request setters.

Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://lore.kernel.org/r/20230824003056.1436637-3-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: use temporary file for rendering

Currently any error during render leads to output an empty file.
That is quite annoying when using tools/net/ynl/ynl-regen.sh
which git greps files with content of "YNL-GEN.." and therefore ignores
empty files. So once you fail to regen, you have to checkout the file.

Avoid that by rendering to a temporary file first, only at the end
copy the content to the actual destination.

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Reviewed-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>

tools: ynl-gen: add missing empty line between policies

We're missing empty line between policies.
DPLL will need this.

Tested-by: Vadim Fedorenko <vadim.fedorenko@linux.dev>
Link: https://lore.kernel.org/r/20230808200907.1290647-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: avoid rendering empty validate field

When dont-validate flags are filtered out for do/dump op, the list may
be empty. In that case, avoid rendering the validate field.

Fixes: fa8ba3502ade ("ynl-gen-c.py: render netlink policies static for split ops")
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Reviewed-by: Jakub Kicinski <kuba@kernel.org>
Link: https://lore.kernel.org/r/20230808090344.1368874-1-jiri@resnulli.us
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

ynl-gen-c.py: render netlink policies static for split ops

When policies are rendered for split ops, they are consumed in the same
file. No need to expose them for user outside, make them static.

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Reviewed-by: Jakub Kicinski <kuba@kernel.org>
Link: https://lore.kernel.org/r/20230803111340.1074067-5-jiri@resnulli.us
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

ynl-gen-c.py: allow directional model for kernel mode

Directional model limitation is only applicable for uapi mode.
For kernel mode, the code is generated correctly using right cmd values
for do/dump requests. Lift the limitation.

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Reviewed-by: Jakub Kicinski <kuba@kernel.org>
Link: https://lore.kernel.org/r/20230803111340.1074067-4-jiri@resnulli.us
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

ynl-gen-c.py: filter rendering of validate field values for split ops

For split ops, do and dump has different meaningful values in
validate field.

Fix the rendering to allow the values per op type as follows:
do: strict
dump: dump, strict-dump

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Reviewed-by: Jakub Kicinski <kuba@kernel.org>
Link: https://lore.kernel.org/r/20230803111340.1074067-3-jiri@resnulli.us
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

ynl: mark max/mask as private for kdoc

Simon mentioned in another thread that it makes kdoc happy
and Jakub confirms that commit e27cb89a22ad ("scripts: kernel-doc: support
private / public marking for enums") actually added the needed
support.

Signed-off-by: Stanislav Fomichev <sdf@google.com>
Reviewed-by: Jakub Kicinski <kuba@kernel.org>
Link: https://lore.kernel.org/r/20230727163001.3952878-3-sdf@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: inherit policy in multi-attr

Instead of reimplementing policies in MutliAttr for every
underlying type forward the calls to the base type.
This will be needed for DPLL which uses a multi-attr nest,
and currently gets an invalid NLA_NEST policy generated.

Reviewed-by: Jacob Keller <Jacob.e.keller@intel.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: correct enum policies

Scalar range validation assumes enums start at 0.
Teach it to properly calculate the value range.

Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: resolve enum vs struct name conflicts

Ethtool has an attribute set called stringset, from which
we'll generate struct ethtool_stringset. Unfortunately,
the old ethtool header declares enum ethtool_stringset
(the same name), to which compilers object.

This seems unavoidable. Check struct names against known
constants and append an underscore if conflict is detected.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>

tools: ynl-gen: don't generate enum types if unnamed

If attr set or enum has empty enum name we need to use u32 or int
as function arguments and struct members.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>

netlink: specs: support setting prefix-name per attribute

Ethtool's PSE PoDL has a attr nest with different prefixes:

/* Power Sourcing Equipment */
enum {
ETHTOOL_A_PSE_UNSPEC,
ETHTOOL_A_PSE_HEADER, /* nest - _A_HEADER_* */
ETHTOOL_A_PODL_PSE_ADMIN_STATE, /* u32 */
ETHTOOL_A_PODL_PSE_ADMIN_CONTROL, /* u32 */
ETHTOOL_A_PODL_PSE_PW_D_STATUS, /* u32 */

Header has a prefix of ETHTOOL_A_PSE_ and other attrs prefix of
ETHTOOL_A_PODL_PSE_ we can't cover them uniformly.
If PODL was after PSE life would be easy.

Now we either need to add prefixes to attr names which is yucky
or support setting prefix name per attr.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>

tools: ynl-gen: record extra args for regen

ynl-regen needs to know the arguments used to generate a file.
Record excluded ops and, while at it, user headers.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>

tools: ynl-gen: support excluding tricky ops

The ethtool family has a small handful of quite tricky ops
and a lot of simple very useful ops. Teach ynl-gen to skip
ops so that we can bypass the tricky ones.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>

tools: ynl-gen: support / skip pads on the way to kernel

Kernel does not have padding requirements for 64b attrs.
We can ignore pad attrs.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: don't pass op_name to RenderInfo

The op_name argument is barely used and identical to op.name
in all cases.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: support code gen for events

Netlink specs support both events and notifications (former can
define their own message contents). Plug in missing code to
generate types, parsers and include events into notification
tables.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: sanitize notification tracking

Don't modify the raw dicts (as loaded from YAML) to pretend
that the notify attributes also exist on the ops. This makes
the code easier to follow.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: stop generating common notification handlers

Common notification handler was supposed to be a way for the user
to parse the notifications from a socket synchronously.
I don't think we'll end up using it, ynl_ntf_check() works for
all known use cases.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: get attr type outside of if()

Reading attr type with mnl_attr_get_type() for each condition
leads to most conditions being longer than 80 chars.
Avoid this by reading the type to a variable on the stack.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: combine else with closing bracket

Code gen currently prints:

}
else if (...

This is really ugly. Fix it by delaying printing of closing
brackets in anticipation of else coming along.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: complete the C keyword list

C keywords need to be avoided when naming things.
Complete the list (ethtool has at least one thing called "auto").

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: cleanup user space header includes

Bots started screaming that we're including stdlib.h twice.
While at it move string.h into a common spot and drop stdio.h
which we don't need.

Reported-by: Abaci Robot <abaci@linux.alibaba.com>
Closes: https://bugzilla.openanolis.cn/show_bug.cgi?id=5464
Closes: https://bugzilla.openanolis.cn/show_bug.cgi?id=5466
Closes: https://bugzilla.openanolis.cn/show_bug.cgi?id=5467
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: don't generate forward declarations for policies

Now that all nested types have structs and are sorted topologically
there should be no need to generate forward declarations for policies.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: walk nested types in depth

So far we had only created structures for nested types nested
directly in messages (second level of attrs so to speak).
Walk types in depth to support deeper nesting.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: inherit struct use info

We only render parse and netlink generation helpers as needed,
to avoid generating dead code. Propagate the information from
first- and second-layer attribute sets onto all children.
Otherwise devlink won't work, it has a lot more levels of nesting.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: try to sort the types more intelligently

We need to sort the structures to avoid the need for forward
declarations. While at it remove the sort of structs when
rendering, it doesn't do anything.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: enable code gen for directional specs

I think that user space code gen for directional specs
works after recent changes. Let them through.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: refactor strmap helper generation

Move generating strmap lookup function to a helper.
No functional changes.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: use enum names in op strmap more carefully

In preparation for supporting families which use different msg
ids to and from the kernel - make sure the ids in op strmap
are correct. The map is expected to be used mostly for notifications,
don't generate a separate map for the "to kernel" direction.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: improve unwind on parsing errors

When parsing multi-attr we count the objects and then allocate
an array to hold the parsed objects. If an attr space has multiple
multi-attr objects, however, if parsing the first array fails
we'll leave the object count for the second even tho the second
array was never allocated.

This may cause crashes when freeing objects on error.

Count attributes to a variable on the stack and only set the count
in the object once the memory was allocated.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: fill in support for MultiAttr scalars

The handshake family needs support for MultiAttr scalars.
Right now we only support code gen for MultiAttr nested
types.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: clean up stray new lines at the end of reply-less requests

Do not print empty lines before closing brackets.

Reviewed-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: generate static descriptions of notifications

Notifications may come in at any time. The family must be always
ready to parse a random incoming notification. Generate notification
table for parsing and tell YNL which request we're processing
to distinguish responses from notifications.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: switch to family struct

We'll want to store static info about the family soon.
Generate a struct. This changes creation from, e.g.:

ys = ynl_sock_create("netdev", &yerr);
to:
ys = ynl_sock_create(&ynl_netdev_family, &yerr);

on user's side.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: generate alloc and free helpers for req

We expect user to allocate requests with calloc(),
make things a bit more consistent and provide helpers.
Generate free calls, too.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: move the response reading logic into YNL

We generate send() and recv() calls and all msg handling for
each operation. It's a lot of repeated code and will only grow
with notification handling. Call back to a helper YNL lib instead.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: generate enum-to-string helpers

It's sometimes useful to print the name of an enum value,
flag or name of the op. Python can do it, add C helper
code gen for getting names of things.

Example:

static const char * const netdev_xdp_act_strmap[] = {
[0] = "basic",
[1] = "redirect",
[2] = "ndo-xmit",
[3] = "xsk-zerocopy",
[4] = "hw-offload",
[5] = "rx-sg",
[6] = "ndo-xmit-sg",
};

const char *netdev_xdp_act_str(enum netdev_xdp_act value)
{
value = ffs(value) - 1;
if (value < 0 || value >= (int)MNL_ARRAY_SIZE(netdev_xdp_act_strmap))
return NULL;
return netdev_xdp_act_strmap[value];
}

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: add error checking for nested structs

Parsing nested types may return an error, propagate it.
Not marking as a fix, because nothing uses YNL upstream.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: loosen type consistency check for events

Both event and notify types are always consistent. Rewrite
the condition checking if we can reuse reply types to be
less picky and let notify thru.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: don't override pure nested struct

For pure structs (parsed nested attributes) we track what
forms of the struct exist in request and reply directions.
Make sure we don't overwrite the recorded struct each time,
otherwise the information is lost.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: fix unused / pad attribute handling

Unused and Pad attributes don't carry information.
Unused should never exist, and be rejected.
Pad should be silently skipped.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: add extra headers for user space

Make sure all relevant headers are included, we allocate memory,
use memcpy() and Linux types without including the headers.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

net: ynl: prefix uAPI header include with uapi/

To keep things simple we used to include the uAPI header
in the kernel in the #include <linux/$family.h> format.
This works well enough, most of the genl families should
have headers in include/net/ so linux/$family.h ends up
referring to the uAPI header, anyway. And if it doesn't
no big deal, we'll just include more info than we need.

Unless that is there is a naming conflict. Someone recently
created include/linux/psp.h which will be a problem when
supporting the PSP protocol. (I'm talking about
work-in-progress patches, but it's just a proof that assuming
lack of name conflicts was overly optimistic.)

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Reviewed-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

tools: ynl: default to treating enums as flags for mask generation

I was a bit too optimistic in commit bf51d27704c9 ("tools: ynl: fix
get_mask utility routine"), not every mask we use is necessarily
coming from an enum of type "flags". We also allow flipping an
enum into flags on per-attribute basis. That's done by
the 'enum-as-flags' property of an attribute.

Restore this functionality, it's not currently used by any in-tree
family.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>

tools: ynl: skip the explicit op array size when not needed

Jiri suggests it reads more naturally to skip the explicit
array size when possible. When we export the symbol we want
to make sure that the size is right but for statics its
not needed.

Link: https://lore.kernel.org/r/20230321044159.1031040-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

ynl: make the tooling check the license

The (only recently documented) expectation is that all specs
are under a certain license, but we don't actually enforce it.
What's worse we then go ahead and assume the license was right,
outputting the expected license into generated files.

Fixes: 37d9df224d1e ("ynl: re-license uniformly under GPL-2.0 OR BSD-3-Clause")
Reviewed-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

ynl: broaden the license even more

I relicensed Netlink spec code to GPL-2.0 OR BSD-3-Clause but
we still put a slightly different license on the uAPI header
than the rest of the code. Use the Linux-syscall-note on all
the specs and all generated code. It's moot for kernel code,
but should not hurt. This way the licenses match everywhere.

Cc: Chuck Lever <chuck.lever@oracle.com>
Fixes: 37d9df224d1e ("ynl: re-license uniformly under GPL-2.0 OR BSD-3-Clause")
Reviewed-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: fix render-max for flags definition

Properly manage render-max property for flags definition type
introducing mask value and setting it to (last_element << 1) - 1
instead of adding max value set to last_element + 1

Fixes: be5bea1cc0bf ("net: add basic C code generators for Netlink")
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: move the enum classes to shared code

Move bulk of the EnumSet and EnumEntry code to shared
code for reuse by cli.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

ynl: re-license uniformly under GPL-2.0 OR BSD-3-Clause

I was intending to make all the Netlink Spec code BSD-3-Clause
to ease the adoption but it appears that:
- I fumbled the uAPI and used "GPL WITH uAPI note" there
- it gives people pause as they expect GPL in the kernel
As suggested by Chuck re-license under dual. This gives us benefit
of full BSD freedom while fulfilling the broad "kernel is under GPL"
expectations.

Link: https://lore.kernel.org/all/20230304120108.05dd44c5@kernel.org/
Link: https://lore.kernel.org/r/20230306200457.3903854-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: use 1 as the default for first entry in attrs/ops

Pretty much all families use value: 1 or reserve as unspec
the first entry in attribute set and the first operation.
Make this the default. Update documentation (the doc for
values of operations just refers back to doc for attrs
so updating only attrs).

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>

tools: ynl-gen: fix single attribute structs with attr 0 only

Chuck run into an issue with a single-element attr-set which
only has an attr with value of 0. The search for max attr in
a struct records attrs with value larger than 0 only (max_val
is set to 0 at the start). Adjust the comparison, alternatively
max_val could be init'ed to -1. Somehow picking the last attr
of a value seems like a good idea in general.

Reported-by: Chuck Lever III <chuck.lever@oracle.com>
Fixes: be5bea1cc0bf ("net: add basic C code generators for Netlink")
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: net: use python3 explicitly

The scripts require Python 3 and some distros are dropping
Python 2 support.

Reported-by: Stanislav Fomichev <sdf@google.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: use the common YAML loading and validation code

Adapt the common object hierarchy in code gen and CLI.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl-gen: prevent do / dump reordering

An earlier fix tried to address generated code jumping around
one code-gen run to another. Turns out dict()s are already
ordered since Python 3.7, the problem is that we iterate over
operation modes using a set(). Sets are unordered in Python.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: store ops in ordered dict to avoid random ordering

When rendering code we should walk the ops in the order in which
they are declared in the spec. This is both more intuitive and
prevents code from jumping around when hashing in the dict changes.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: rename ops_list -> msg_list

ops_list contains all the operations, but the main iteration use
case is to walk only ops which define attrs. Rename ops_list to
msg_list, because now it looks like the contents are the same,
just the format is different. While at it convert from tuple
to just keys, none of the users care about the name of the op.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>

tools: ynl: support kdocs for flags in code generation

Lorenzo reports that after switching from enum to flags netdev
family lost ability to render kdoc (and the enum contents got
generally garbled).

Combine the flags and enum handling in uAPI handling.

Reported-by: Lorenzo Bianconi <lorenzo@kernel.org>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

net: add basic C code generators for Netlink

Code generators to turn Netlink specs into C code.
I'm definitely not proud of it.

The main generator is in Python, there's a bash script
to regen all code-gen'ed files in tree after making
spec changes.

Acked-by: Stanislav Fomichev <sdf@google.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Paolo Abeni <pabeni@redhat.com>