11590Srgrimes=============== 21590SrgrimesBPF ring buffer 31590Srgrimes=============== 41590Srgrimes 51590SrgrimesThis document describes BPF ring buffer design, API, and implementation details. 61590Srgrimes 71590Srgrimes.. contents:: 81590Srgrimes :local: 91590Srgrimes :depth: 2 101590Srgrimes 111590SrgrimesMotivation 121590Srgrimes---------- 131590Srgrimes 141590SrgrimesThere are two distinctive motivators for this work, which are not satisfied by 151590Srgrimesexisting perf buffer, which prompted creation of a new ring buffer 161590Srgrimesimplementation. 171590Srgrimes 181590Srgrimes- more efficient memory utilization by sharing ring buffer across CPUs; 191590Srgrimes- preserving ordering of events that happen sequentially in time, even across 201590Srgrimes multiple CPUs (e.g., fork/exec/exit events for a task). 211590Srgrimes 221590SrgrimesThese two problems are independent, but perf buffer fails to satisfy both. 231590SrgrimesBoth are a result of a choice to have per-CPU perf ring buffer. Both can be 241590Srgrimesalso solved by having an MPSC implementation of ring buffer. The ordering 251590Srgrimesproblem could technically be solved for perf buffer with some in-kernel 261590Srgrimescounting, but given the first one requires an MPSC buffer, the same solution 271590Srgrimeswould solve the second problem automatically. 281590Srgrimes 291590SrgrimesSemantics and APIs 301590Srgrimes------------------ 311590Srgrimes 321590SrgrimesSingle ring buffer is presented to BPF programs as an instance of BPF map of 331590Srgrimestype ``BPF_MAP_TYPE_RINGBUF``. Two other alternatives considered, but 341590Srgrimesultimately rejected. 3574769Smikeh 361590SrgrimesOne way would be to, similar to ``BPF_MAP_TYPE_PERF_EVENT_ARRAY``, make 3774769Smikeh``BPF_MAP_TYPE_RINGBUF`` could represent an array of ring buffers, but not 3874769Smikehenforce "same CPU only" rule. This would be more familiar interface compatible 3974769Smikehwith existing perf buffer use in BPF, but would fail if application needed more 401590Srgrimesadvanced logic to lookup ring buffer by arbitrary key. 411590Srgrimes``BPF_MAP_TYPE_HASH_OF_MAPS`` addresses this with current approach. 421590SrgrimesAdditionally, given the performance of BPF ringbuf, many use cases would just 431590Srgrimesopt into a simple single ring buffer shared among all CPUs, for which current 441590Srgrimesapproach would be an overkill. 451590Srgrimes 461590SrgrimesAnother approach could introduce a new concept, alongside BPF map, to represent 471590Srgrimesgeneric "container" object, which doesn't necessarily have key/value interface 481590Srgrimeswith lookup/update/delete operations. This approach would add a lot of extra 491590Srgrimesinfrastructure that has to be built for observability and verifier support. It 501590Srgrimeswould also add another concept that BPF developers would have to familiarize 5177274Smikehthemselves with, new syntax in libbpf, etc. But then would really provide no 5277274Smikehadditional benefits over the approach of using a map. ``BPF_MAP_TYPE_RINGBUF`` 5377274Smikehdoesn't support lookup/update/delete operations, but so doesn't few other map 5477274Smikehtypes (e.g., queue and stack; array doesn't support delete, etc). 551590Srgrimes 561590SrgrimesThe approach chosen has an advantage of re-using existing BPF map 571590Srgrimesinfrastructure (introspection APIs in kernel, libbpf support, etc), being 581590Srgrimesfamiliar concept (no need to teach users a new type of object in BPF program), 5977274Smikehand utilizing existing tooling (bpftool). For common scenario of using a single 6077274Smikehring buffer for all CPUs, it's as simple and straightforward, as would be with 6177274Smikeha dedicated "container" object. On the other hand, by being a map, it can be 6277274Smikehcombined with ``ARRAY_OF_MAPS`` and ``HASH_OF_MAPS`` map-in-maps to implement 6377274Smikeha wide variety of topologies, from one ring buffer for each CPU (e.g., as 641590Srgrimesa replacement for perf buffer use cases), to a complicated application 651590Srgrimeshashing/sharding of ring buffers (e.g., having a small pool of ring buffers 661590Srgrimeswith hashed task's tgid being a look up key to preserve order, but reduce 671590Srgrimescontention). 6874769Smikeh 691590SrgrimesKey and value sizes are enforced to be zero. ``max_entries`` is used to specify 701590Srgrimesthe size of ring buffer and has to be a power of 2 value. 7177274Smikeh 7277274SmikehThere are a bunch of similarities between perf buffer 731590Srgrimes(``BPF_MAP_TYPE_PERF_EVENT_ARRAY``) and new BPF ring buffer semantics: 741590Srgrimes 751590Srgrimes- variable-length records; 7677274Smikeh- if there is no more space left in ring buffer, reservation fails, no 771590Srgrimes blocking; 7877274Smikeh- memory-mappable data area for user-space applications for ease of 791590Srgrimes consumption and high performance; 8077274Smikeh- epoll notifications for new incoming data; 811590Srgrimes- but still the ability to do busy polling for new data to achieve the 821590Srgrimes lowest latency, if necessary. 831590Srgrimes 841590SrgrimesBPF ringbuf provides two sets of APIs to BPF programs: 851590Srgrimes 8677274Smikeh- ``bpf_ringbuf_output()`` allows to *copy* data from one place to a ring 871590Srgrimes buffer, similarly to ``bpf_perf_event_output()``; 881590Srgrimes- ``bpf_ringbuf_reserve()``/``bpf_ringbuf_commit()``/``bpf_ringbuf_discard()`` 891590Srgrimes APIs split the whole process into two steps. First, a fixed amount of space 901590Srgrimes is reserved. If successful, a pointer to a data inside ring buffer data 911590Srgrimes area is returned, which BPF programs can use similarly to a data inside 9277274Smikeh array/hash maps. Once ready, this piece of memory is either committed or 931590Srgrimes discarded. Discard is similar to commit, but makes consumer ignore the 9477274Smikeh record. 951590Srgrimes 961590Srgrimes``bpf_ringbuf_output()`` has disadvantage of incurring extra memory copy, 971590Srgrimesbecause record has to be prepared in some other place first. But it allows to 981590Srgrimessubmit records of the length that's not known to verifier beforehand. It also 991590Srgrimesclosely matches ``bpf_perf_event_output()``, so will simplify migration 1001590Srgrimessignificantly. 1011590Srgrimes 10277274Smikeh``bpf_ringbuf_reserve()`` avoids the extra copy of memory by providing a memory 1031590Srgrimespointer directly to ring buffer memory. In a lot of cases records are larger 1041590Srgrimesthan BPF stack space allows, so many programs have use extra per-CPU array as 1051590Srgrimesa temporary heap for preparing sample. bpf_ringbuf_reserve() avoid this needs 1061590Srgrimescompletely. But in exchange, it only allows a known constant size of memory to 1071590Srgrimesbe reserved, such that verifier can verify that BPF program can't access memory 10877274Smikehoutside its reserved record space. bpf_ringbuf_output(), while slightly slower 1091590Srgrimesdue to extra memory copy, covers some use cases that are not suitable for 1101590Srgrimes``bpf_ringbuf_reserve()``. 1111590Srgrimes 1121590SrgrimesThe difference between commit and discard is very small. Discard just marks 11388150Smikeha record as discarded, and such records are supposed to be ignored by consumer 1141590Srgrimescode. Discard is useful for some advanced use-cases, such as ensuring 1151590Srgrimesall-or-nothing multi-record submission, or emulating temporary 1161590Srgrimes``malloc()``/``free()`` within single BPF program invocation. 11777274Smikeh 11877274SmikehEach reserved record is tracked by verifier through existing 11977274Smikehreference-tracking logic, similar to socket ref-tracking. It is thus 1201590Srgrimesimpossible to reserve a record, but forget to submit (or discard) it. 1211590Srgrimes 1221590Srgrimes``bpf_ringbuf_query()`` helper allows to query various properties of ring 1231590Srgrimesbuffer. Currently 4 are supported: 1241590Srgrimes 1251590Srgrimes- ``BPF_RB_AVAIL_DATA`` returns amount of unconsumed data in ring buffer; 1261590Srgrimes- ``BPF_RB_RING_SIZE`` returns the size of ring buffer; 1271590Srgrimes- ``BPF_RB_CONS_POS``/``BPF_RB_PROD_POS`` returns current logical position 1281590Srgrimes of consumer/producer, respectively. 12988150Smikeh 13088150SmikehReturned values are momentarily snapshots of ring buffer state and could be 13177274Smikehoff by the time helper returns, so this should be used only for 13277274Smikehdebugging/reporting reasons or for implementing various heuristics, that take 13377274Smikehinto account highly-changeable nature of some of those characteristics. 1341590Srgrimes 13577274SmikehOne such heuristic might involve more fine-grained control over poll/epoll 1361590Srgrimesnotifications about new data availability in ring buffer. Together with 1371590Srgrimes``BPF_RB_NO_WAKEUP``/``BPF_RB_FORCE_WAKEUP`` flags for output/commit/discard 13877274Smikehhelpers, it allows BPF program a high degree of control and, e.g., more 1391590Srgrimesefficient batched notifications. Default self-balancing strategy, though, 1401590Srgrimesshould be adequate for most applications and will work reliable and efficiently 1411590Srgrimesalready. 1421590Srgrimes 1431590SrgrimesDesign and Implementation 1441590Srgrimes------------------------- 1451590Srgrimes 14688150SmikehThis reserve/commit schema allows a natural way for multiple producers, either 1471590Srgrimeson different CPUs or even on the same CPU/in the same BPF program, to reserve 1481590Srgrimesindependent records and work with them without blocking other producers. This 1491590Srgrimesmeans that if BPF program was interrupted by another BPF program sharing the 15077274Smikehsame ring buffer, they will both get a record reserved (provided there is 15188150Smikehenough space left) and can work with it and submit it independently. This 15288150Smikehapplies to NMI context as well, except that due to using a spinlock during 15388150Smikehreservation, in NMI context, ``bpf_ringbuf_reserve()`` might fail to get 15488150Smikeha lock, in which case reservation will fail even if ring buffer is not full. 1551590Srgrimes 15688150SmikehThe ring buffer itself internally is implemented as a power-of-2 sized 15777274Smikehcircular buffer, with two logical and ever-increasing counters (which might 1581590Srgrimeswrap around on 32-bit architectures, that's not a problem): 1591590Srgrimes 1601590Srgrimes- consumer counter shows up to which logical position consumer consumed the 1611590Srgrimes data; 1621590Srgrimes- producer counter denotes amount of data reserved by all producers. 1631590Srgrimes 16477274SmikehEach time a record is reserved, producer that "owns" the record will 1651590Srgrimessuccessfully advance producer counter. At that point, data is still not yet 16677274Smikehready to be consumed, though. Each record has 8 byte header, which contains the 1671590Srgrimeslength of reserved record, as well as two extra bits: busy bit to denote that 16877274Smikehrecord is still being worked on, and discard bit, which might be set at commit 1691590Srgrimestime if record is discarded. In the latter case, consumer is supposed to skip 1701590Srgrimesthe record and move on to the next one. Record header also encodes record's 1711590Srgrimesrelative offset from the beginning of ring buffer data area (in pages). This 1721590Srgrimesallows ``bpf_ringbuf_commit()``/``bpf_ringbuf_discard()`` to accept only the 1731590Srgrimespointer to the record itself, without requiring also the pointer to ring buffer 1741590Srgrimesitself. Ring buffer memory location will be restored from record metadata 1751590Srgrimesheader. This significantly simplifies verifier, as well as improving API 1761590Srgrimesusability. 1771590Srgrimes 17877274SmikehProducer counter increments are serialized under spinlock, so there is 17974769Smikeha strict ordering between reservations. Commits, on the other hand, are 1801590Srgrimescompletely lockless and independent. All records become available to consumer 1811590Srgrimesin the order of reservations, but only after all previous records where 1821590Srgrimesalready committed. It is thus possible for slow producers to temporarily hold 1831590Srgrimesoff submitted records, that were reserved later. 1841590Srgrimes 1851590SrgrimesOne interesting implementation bit, that significantly simplifies (and thus 1861590Srgrimesspeeds up as well) implementation of both producers and consumers is how data 1871590Srgrimesarea is mapped twice contiguously back-to-back in the virtual memory. This 1881590Srgrimesallows to not take any special measures for samples that have to wrap around 1891590Srgrimesat the end of the circular buffer data area, because the next page after the 1901590Srgrimeslast data page would be first data page again, and thus the sample will still 1911590Srgrimesappear completely contiguous in virtual memory. See comment and a simple ASCII 1921590Srgrimesdiagram showing this visually in ``bpf_ringbuf_area_alloc()``. 19377274Smikeh 1941590SrgrimesAnother feature that distinguishes BPF ringbuf from perf ring buffer is 1951590Srgrimesa self-pacing notifications of new data being availability. 19677274Smikeh``bpf_ringbuf_commit()`` implementation will send a notification of new record 1971590Srgrimesbeing available after commit only if consumer has already caught up right up to 1981590Srgrimesthe record being committed. If not, consumer still has to catch up and thus 19974769Smikehwill see new data anyways without needing an extra poll notification. 20074769SmikehBenchmarks (see tools/testing/selftests/bpf/benchs/bench_ringbufs.c) show that 2011590Srgrimesthis allows to achieve a very high throughput without having to resort to 2021590Srgrimestricks like "notify only every Nth sample", which are necessary with perf 20374769Smikehbuffer. For extreme cases, when BPF program wants more manual control of 2041590Srgrimesnotifications, commit/discard/output helpers accept ``BPF_RB_NO_WAKEUP`` and 2051590Srgrimes``BPF_RB_FORCE_WAKEUP`` flags, which give full control over notifications of 2061590Srgrimesdata availability, but require extra caution and diligence in using this API. 2071590Srgrimes